(Some) Anchors Do Not Work - Raises Invalid YAML #14
Comments
|
@acederberg there can be subtle differences with YAML parsing across platforms. IIRC if you compare w/ python, consider to add the PHP YAML extension if possible. There is a fallback mechanism and if one parser fails, the other is tried. With PHP ext-yaml, there is a second chance. For the concrete problem if you can add a (reduced is fine, too) bitbucket-pipelines.yml that triggers this issue would be good so I can tell more. |
|
@ktomk Thanks! I (think I) figured it out. I'm using some YAML anchors and that is causing the issue. I rendered my YAML with python and dumped it into a new pipeline YAML and it works ( it uses all anchors besides the For example image : python:3.10
options : { docker : true }
definitions :
services :
mongodb :
environment :
MONGO_INITDB_ROOT_USERNAME : adrian
MONGO_INITDB_ROOT_PASSWORD : somepassword
MONGO_INITDB_DATABASE : tests
image : mongo
steps :
- step : &create_config
name : "Create '.env'."
script :
- export ENV_PATH="$PWD/.env"
- touch $ENV_PATH
- echo "mongodb_host='brain_rtus_db'" >> $ENV_PATH
- echo "mongodb_port=27017" >> $ENV_PATH
- echo "mongodb_user='adrian'" >> $ENV_PATH
- echo "mongodb_password='somepassword'" >> $ENV_PATH
- echo "mongodb_use_atlas='False'" >> $ENV_PATH
- echo "uvicorn_port=8001" >> $ENV_PATH
- echo "uvicorn_host='localhost'" >> $ENV_PATH
- echo "uvicorn_reload='True'" >> $ENV_PATH
- realpath $ENV_PATH
artifacts :
- .env
pipelines :
default :
- step :
<<: *create_configis invalid. But when I render it with python to get different anchors : it is valid. |
acederberg
changed the title
acederberg
changed the title
|
thanks for looking further into it @acederberg
Okay, I can see that. And you got me for a moment because this YAML merge keys can be problematic (in general). I've written a summary of these in YAML Anchor, Aliases and Merge Keys, see especially the section Caveat: Support of Merge Keys. However, and thanks to your example YAML, I had no problems to use it with pipelines (pipelines: version 0.0.62+6-g3dc9e6a+dirty w/ php 7.4.28 (libyaml: 2.1.0)) and it is not because I have the PHP yaml extension as I have disabled that YAML parser internally for this test. Toggle me!So merge keys might be related but also are supported (in pipelines(1)). This is interesting, as it does not work on your end. Perhaps there is something different in the file triggering the issue? /E: Test changes here: 497d505 |
|
Hi again @ktomk. Looking into it still and it appears that I am entirely wrong about anchors. I completely deanchored another yaml and it does not work, I am free to share this yaml. here it is : definitions:
services:
mysql:
image: mysql
variables:
MYSQL_DATABASE: mve_brain_sqlalchemy_tests
MYSQL_PASSWORD: somepassword
MYSQL_RANDOM_ROOT_PASSWORD: 'yes'
MYSQL_USER: adrian
steps:
- step:
artifacts:
- .env.test
- fetchers/.weather.env
name: Create .env.test for testing since this file should never be commited.
script:
- export ENV_TEST_PATH=".env.test"
- touch $ENV_TEST_PATH
- echo 'mysql_database="mve_brain_sqlalchemy_tests"' >> $ENV_TEST_PATH
- echo 'mysql_host=127.0.0.1' >> $ENV_TEST_PATH
- echo 'mysql_port=3306' >> $ENV_TEST_PATH
- echo 'mysql_username="adrian"' >> $ENV_TEST_PATH
- echo 'mysql_password="somepassword"' >> $ENV_TEST_PATH
- echo 'mysql_drivername="mysql+asyncmy"' >> $ENV_TEST_PATH
- echo 'uvicorn_host="0.0.0.0"' >> $ENV_TEST_PATH
- echo 'uvicorn_port=8000' >> $ENV_TEST_PATH
- export ENV_TEST_FETCHERS=./api/fetchers/.fetchers.env
- touch $ENV_TEST_FETCHERS
- echo "weather_key=$WEATHER_KEY" >> $ENV_TEST_FETCHERS
image: python:3.10
options:
docker: true
pipelines:
custom:
build_pipeline_service:
- step:
artifacts:
- .env.test
- fetchers/.weather.env
name: Create .env.test for testing since this file should never be commited.
script:
- export ENV_TEST_PATH=".env.test"
- touch $ENV_TEST_PATH
- echo 'mysql_database="mve_brain_sqlalchemy_tests"' >> $ENV_TEST_PATH
- echo 'mysql_host=127.0.0.1' >> $ENV_TEST_PATH
- echo 'mysql_port=3306' >> $ENV_TEST_PATH
- echo 'mysql_username="adrian"' >> $ENV_TEST_PATH
- echo 'mysql_password="somepassword"' >> $ENV_TEST_PATH
- echo 'mysql_drivername="mysql+asyncmy"' >> $ENV_TEST_PATH
- echo 'uvicorn_host="0.0.0.0"' >> $ENV_TEST_PATH
- echo 'uvicorn_port=8000' >> $ENV_TEST_PATH
- export ENV_TEST_FETCHERS=./api/fetchers/.fetchers.env
- touch $ENV_TEST_FETCHERS
- echo "weather_key=$WEATHER_KEY" >> $ENV_TEST_FETCHERS
- step:
name: "Building an api image to be used in other pipelines using development\
\ container multistage build. \nThis includes the .env file constructed\
\ here.\n"
script:
- cp .env.test .env
- export IMAGE_NAME="$DOCKERHUB_USERNAME/$DOCKERHUB_BRAIN_REPOSITORY"
- export IMAGE_PIPELINE_SERVICE="$IMAGE_NAME:api_pipeline_service_$BITBUCKET_COMMIT"
- docker login --username $DOCKERHUB_USERNAME --password $DOCKERHUB_PASSWORD
- docker build -t "$IMAGE_PIPELINE_SERVICE" -f "Dockerfile" .
- docker run --name test2 --detach "$IMAGE_PIPELINE_SERVICE"
- sleep 15
- docker stop test2
- docker push $IMAGE_NAME
default:
- step:
caches:
- pip
name: Install all dependencies.
script:
- pip install -r requirements.txt
- pip install -r requirements.dev.txt
- step:
artifacts:
- .env.test
- fetchers/.weather.env
name: Create .env.test for testing since this file should never be commited.
script:
- export ENV_TEST_PATH=".env.test"
- touch $ENV_TEST_PATH
- echo 'mysql_database="mve_brain_sqlalchemy_tests"' >> $ENV_TEST_PATH
- echo 'mysql_host=127.0.0.1' >> $ENV_TEST_PATH
- echo 'mysql_port=3306' >> $ENV_TEST_PATH
- echo 'mysql_username="adrian"' >> $ENV_TEST_PATH
- echo 'mysql_password="somepassword"' >> $ENV_TEST_PATH
- echo 'mysql_drivername="mysql+asyncmy"' >> $ENV_TEST_PATH
- echo 'uvicorn_host="0.0.0.0"' >> $ENV_TEST_PATH
- echo 'uvicorn_port=8000' >> $ENV_TEST_PATH
- export ENV_TEST_FETCHERS=./api/fetchers/.fetchers.env
- touch $ENV_TEST_FETCHERS
- echo "weather_key=$WEATHER_KEY" >> $ENV_TEST_FETCHERS
- step:
caches:
- pip
name: Test
script:
- pip install -r requirements.txt
- pip install -r requirements.dev.txt
- python -m pytest .
services:
- mysql
- step:
caches:
- pip
name: Building prod and fetchers docker images (without an enironment files)
script:
- export IMAGE_NAME="$DOCKERHUB_USERNAME/$DOCKERHUB_BRAIN_REPOSITORY"
- export IMAGE_API="$IMAGE_NAME:api_$BITBUCKET_COMMIT"
- export IMAGE_FETCHERS="$IMAGE_NAME:fetchers_$BITBUCKET_COMMIT"
- docker login --username $DOCKERHUB_USERNAME --password $DOCKERHUB_PASSWORD
- docker build -t "$IMAGE_API" -f "Dockerfile.prod" --target "prod" .
- docker build -t "$IMAGE_FETCHERS" -f "Dockerfile.prod" --target "fetcher_runner"
.
- docker run --name test1 --detach "$IMAGE_API"
- sleep 30
- docker stop test1
- docker push $IMAGE_NAME |
|
With the Sf2Yaml it works on my end. With the LibYaml it does not and I could find two "culprits":
If I remove both, it works with the libyaml parser, too. Wondering a bit why it's not working on your end as IIRC the Sf2Yaml parser is taken on your end as you don't have LibYaml. @acederberg |
|
Here's the verbose output prior to those changes mentioned above : I got it to run after fixing those, so it looks like some whacky stuff is happening with pyyaml causing it to dump the YAML with those errors. I am still ( and now to a greater degree than before ) confused as there is still something in my other yaml that will make it fail after it's rendered. IDK what I am doing wrong elsewhere, but it must be an error in my yaml and not the parser itself. Is there a way to get better feedback on the YAML errors from pipelines? e.g. "bad syntax on line n column k" |
|
Well the YAML parser in pipelines most likely isn't golden. I had issues earlier and from where its coming from it was always with some fallback. I'll tweak the handling based on the feedback here probably. Thinking about putting the Sf2Yaml parser upfront and use the LibYaml one as fallback (currently the other way round, but what I learned this is good for a change).
Yes, that would be good to at least have some pointers. I'll patch something in, it should provide more info at least in some cases. It's not entirely structural like line and column AFAIK. |
|
Okay, I made a mistake in testing as I now see:
This is from the Sf2Yaml parser. The LibYaml parser is fine. Looks like I mixed it up. That should explain why I was puzzled earlier. I'll put error messages in. It will be a bit text-walling but better than current. |
previously when parsing a pipelines YAML file that could not be parsed, for example b/c of unsupported syntax in the YAML parser, pipelines did only notify which file and that it was invalid for pipelines. addition is to capture error messages so that when YAML parsing fails, this error message is extended. report: #14
|
Version 0.0.63 has been released showing more detailed error messages on YAML file parse errors, Link to full change-log.. |
|
@ktomk Thank you! For the reference of anybody who needs help in the future who is unfamiliar with php package managers, update by editing the version number in |
|
@acederberg: Yes, if you install with Takes the latest 0.0.x version (as require also updates). It is using require here so it works with however you have set it up originally. As it also works from the source version, it should be possible as well to install non-tagged revisions with composer: How did you learn about installing this utility? Just curious and I may think about improving the docs about it. |
Thanks for the awesome tool, it has/will save me a lot of time. I was running this on a new pipeline like
and it says
But I can parse the file with python
and it passes in the bitbucket validator. When I run the pipeline on bitbucket it works. I can provide my yaml, but it will require some amendments as this is internal to my organization and thus I am hesitant. If I figure out what is making the parser upset I will add that to my post here.
It also might be worth considering that I am using the windows subsystem for linux.
Thank you again for any time, it is really appreciated.
EDIT : Debug output
Running
pipelines --debuggetsEDIT
It turns out that the parser does not like anchors and this is the root of the problem.
The text was updated successfully, but these errors were encountered: