Build conda on merge to main #1247

ryan-s · 2020-09-30T15:47:02Z

Pull Request Description

Add in a conda package build as part of the ci when a PR is merged into main. This PR adds in a small python script to change the recipe to use the local build (conda_config.py). Once we have that, we clone the feedstock and start to replicate what the python and shell scripts were doing do to issues on circleCI. The end result is a build that passes or fails and produces no artifacts.

After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of docs/source/release_notes.rst to include this pull request by adding :pr:123.

freddyaboulton

@ryan-s this is awesome! I left some comments to explain to the team what's going on (feel free to correct me where I'm mistaken hehe).

We ran this check on circle ci with main and a purposely broken branch and the check is working as expected!

dev-requirements.txt

freddyaboulton · 2020-09-30T16:05:20Z

.circleci/config.yml

+            cp -r `ls -A | grep -v "evalml-core-feedstock"` ./evalml-core-feedstock/evalml/
+            python conda_config.py "$(python setup.py --version)"
+            cd evalml-core-feedstock
+            echo "$DOCKER_HUB_PASS" | docker login -u psalter --password-stdin


We have manually check out this container and run docker run ourselves as opposed to running run_docker_build.sh in the feedstock repo because circle-ci doesn't let you mount volumes to the docker image. I think this is ok because we're not straying far from the contents of that script. @ryan-s and I discussed ways of tightening this up in the future too!

@ryan-s @freddyaboulton is this env var DOCKER_HUB_PASS set by circleci? Did you have to enter it?

@dsherry I entered it in circleci. Not sure if we have a vault somewhere to share the credentials?

freddyaboulton · 2020-09-30T16:05:47Z

.circleci/config.yml

+            - shellcheck/check
+          filters:
+            branches:
+              only: main


This means that this will run after a PR gets merged to main.

We created this check to catch breaking changes to our conda recipe (like new dependencies or dependency updates) before release day and having it run after a merge accomplishes that without having to slow down development or consume too many ci resources.

codecov · 2020-09-30T18:23:39Z

Codecov Report

Merging #1247 into main will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1247   +/-   ##
=======================================
  Coverage   99.93%   99.93%           
=======================================
  Files         207      207           
  Lines       13031    13031           
=======================================
  Hits        13022    13022           
  Misses          9        9

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 95e05e4...8cdcce6. Read the comment docs.

dsherry

@ryan-s this rocks!!! I'm so excited you're adding this! It's gonna make our release process a lot simpler and less likely to break conda. Woohoo!!

So, I take it we'll still need to do the actual conda release manually, right?

@freddyaboulton once this is in, could you update release.md to reflect this?

@ryan-s I left a bunch of questions, and two comments to address before merge:

Please move that new file conda_config.py to somewhere other than the top-level directory -- I think it should go in .circleci/. Hmm, on reflection, since its python code, perhaps it should go in a new folder evalml/tests/circleci/. Your call there.
I left a comment about using Pathlib--let's resolve that, either by making the change or by you telling me its unnecessary lol

🚢

dsherry · 2020-10-01T13:25:37Z

.circleci/config.yml

+            - shellcheck/check
+          filters:
+            branches:
+              only: main


dsherry · 2020-10-01T13:26:30Z

.circleci/config.yml

+            cp -r `ls -A | grep -v "evalml-core-feedstock"` ./evalml-core-feedstock/evalml/
+            python conda_config.py "$(python setup.py --version)"
+            cd evalml-core-feedstock
+            echo "$DOCKER_HUB_PASS" | docker login -u psalter --password-stdin


@ryan-s @freddyaboulton is this env var DOCKER_HUB_PASS set by circleci? Did you have to enter it?

dsherry · 2020-10-01T13:27:50Z

.circleci/config.yml

+            export UPLOAD_PACKAGES=False
+            export HOST_USER_ID=$(id -u)
+            export FEEDSTOCK_NAME=evalml-core-feedstock
+            docker run -t -e CONFIG -e HOST_USER_ID -e UPLOAD_PACKAGES -e GIT_BRANCH -e UPLOAD_ON_BRANCH -e CI -e FEEDSTOCK_NAME -e CPU_COUNT -e BINSTAR_TOKEN -e FEEDSTOCK_TOKEN -e STAGING_BINSTAR_TOKEN psalter/build:latest bash /home/conda/feedstock_root/.scripts/build_steps.sh


If I'm following right, this command will have a non-zero return code if there's an error? Meaning the CI job will get flagged as failed?

Will the error output appear in the CI job, or do we have to do something else to get the error output? It would be amazing to have the output appear in the CI job, because otherwise its gonna be hard to debug.

The error appears in the CI job!

dsherry · 2020-10-01T13:28:45Z

.circleci/config.yml

+      - checkout
+      - install_dependencies_test
+      - setup_remote_docker:
+          version: 19.03.12


What is this version of? Is this the version of docker itself? And is setup_remote_docker a circleci predefined macro, or something else?

Correct, this is the latest version of docker that circleci supports and it is predefined by them.

dsherry · 2020-10-01T13:29:14Z

.circleci/config.yml

+    working_directory: ~/evalml/
+    executor:
+      name: python
+      python_version: "3.6"


@ryan-s @freddyaboulton why python 3.6? Should we do this for all our supported python versions, 3.6, 3.7 and 3.8? Or just the latest version, 3.8?

I think this sets the version of python in the circleci worker. The docker run command will actually build the package in python 3.7.8 - this is the same as what happens in the CI of our feedstock repo. Unfortunately, conda doesn't provide an out-of-the-box way to build with different python versions but I will add this to the future work section of our design doc for this issue.

Ah got it. So we'll only build and run our tests on 3.7.8 for now. No problem, this is a great starting point. Thanks!

Correct, I just needed something to install pyyaml into so that we can update the recipe.

I think the v2 of this would be to create our own container(s) based off of condaforge/linux-anvil-comp7 where we can move the existing functionality and any new improvements into.

dsherry · 2020-10-01T13:30:54Z

conda_config.py

+    parser = argparse.ArgumentParser(description="Configure conda for local build. Run from the feedstock root")
+    parser.add_argument('version', help='The version of EvalML being built')
+    args = parser.parse_args()
+    write_conda_recipe(args.version)


@ryan-s this is super cool!!

Please move this file to a different location instead of the top-level directory. Should it go in .circleci/? That would be my first inclination.

dsherry · 2020-10-01T13:31:16Z

conda_config.py

+        version: The version of EvalML we are building with this feedstock
+
+    Returns:
+        None: Side effect of overwriting the existing meta.yaml in the feedstock


Amazing, thank you for documenting.

dsherry · 2020-10-01T13:31:45Z

conda_config.py

+class CondaDumper(yaml.Dumper):
+
+    def increase_indent(self, flow=False, indentless=False):
+        return super(CondaDumper, self).increase_indent(flow, False)


Why is this necessary? Is this just a style thing, or is it a functional thing?

Unfortunately it is functional. The existing recipe is actually a Jinja2 template masquerading as a yaml file. Whatever they are using to parse that template requires that we have the increased indents even though the YAML spec does not.

dsherry · 2020-10-01T13:32:03Z

test-requirements.txt

@@ -7,3 +7,4 @@ codecov==2.1.0
 category_encoders>=2.0.0
 featuretools
 nlp_primitives>=1.0.0
+PyYAML==5.3.1


dsherry · 2020-10-01T13:34:27Z

conda_config.py

+        # Toss out the first line that declares the version since its not supported YAML syntax
+        next(config_file)
+        config = yaml.safe_load(config_file)
+        recipe_path = f'../feedstock_root/evalml'


@ryan-s could you please use pathlib.Path here instead? For hypothetical windows support haha

recipe_path = str(pathlib.Path('..', 'feedstock_root', 'evalml'))

Also what is this path? Like, this is intended to be the path on the docker container? A comment explaining this would be super helpful to future readers.

freddyaboulton · 2020-10-01T13:55:24Z

@ryan-s this rocks!!! I'm so excited you're adding this! It's gonna make our release process a lot simpler and less likely to break conda. Woohoo!!

So, I take it we'll still need to do the actual conda release manually, right?

@freddyaboulton once this is in, could you update release.md to reflect this?

@ryan-s I left a bunch of questions, and two comments to address before merge:

Please move that new file conda_config.py to somewhere other than the top-level directory -- I think it should go in .circleci/. Hmm, on reflection, since its python code, perhaps it should go in a new folder evalml/tests/circleci/. Your call there.

I left a comment about using Pathlib--let's resolve that, either by making the change or by you telling me its unnecessary lol

🚢

Yes, we still need to manually edit the recipe file in the feedstock if there are dependency changes in a release but now we can skip the "Test conda version before releasing on PyPI" section of the release process! I'll update once this is in.

ryan-s · 2020-10-02T16:06:49Z

@ryan-s this rocks!!! I'm so excited you're adding this! It's gonna make our release process a lot simpler and less likely to break conda. Woohoo!!

So, I take it we'll still need to do the actual conda release manually, right?

@freddyaboulton once this is in, could you update release.md to reflect this?

@ryan-s I left a bunch of questions, and two comments to address before merge:

Please move that new file conda_config.py to somewhere other than the top-level directory -- I think it should go in .circleci/. Hmm, on reflection, since its python code, perhaps it should go in a new folder evalml/tests/circleci/. Your call there.

I left a comment about using Pathlib--let's resolve that, either by making the change or by you telling me its unnecessary lol

🚢

As of right now yes, but in theory it wouldn't be too hard to copy the artifact out of the container and push it. You would just need to do the inverse of what we do when we copy our feedstock into the container.

freddyaboulton approved these changes Sep 30, 2020

View reviewed changes

freddyaboulton requested review from dsherry, angela97lin, jeremyliweishih, christopherbunn, bchen1116 and eccabay September 30, 2020 16:17

dsherry changed the title ~~Config changes to build on merge to main.~~ Build conda on merge to main Oct 1, 2020

dsherry approved these changes Oct 1, 2020

View reviewed changes

dsherry assigned ryan-s Oct 2, 2020

Config changes to build on merge to main.

8cdcce6

ryan-s force-pushed the ci-build-conda branch from 3c91eb0 to 8cdcce6 Compare October 2, 2020 18:28

freddyaboulton merged commit 3d72dff into main Oct 2, 2020

freddyaboulton mentioned this pull request Oct 8, 2020

Removing conda pre-release check from the release process documents. #1282

Merged

dsherry mentioned this pull request Oct 29, 2020

Release v0.15.0 #1370

Merged

freddyaboulton deleted the ci-build-conda branch May 13, 2022 14:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build conda on merge to main #1247

Build conda on merge to main #1247

ryan-s commented Sep 30, 2020

freddyaboulton left a comment

freddyaboulton Sep 30, 2020

dsherry Oct 1, 2020

ryan-s Oct 2, 2020

freddyaboulton Sep 30, 2020

angela97lin Sep 30, 2020

dsherry Oct 1, 2020

codecov bot commented Sep 30, 2020 •

edited

Loading

dsherry left a comment

dsherry Oct 1, 2020

dsherry Oct 1, 2020

dsherry Oct 1, 2020

freddyaboulton Oct 1, 2020

dsherry Oct 1, 2020

ryan-s Oct 2, 2020

dsherry Oct 1, 2020

freddyaboulton Oct 1, 2020 •

edited

Loading

dsherry Oct 1, 2020

ryan-s Oct 2, 2020

dsherry Oct 1, 2020

dsherry Oct 1, 2020

dsherry Oct 1, 2020

ryan-s Oct 2, 2020

dsherry Oct 1, 2020

dsherry Oct 1, 2020

freddyaboulton commented Oct 1, 2020

ryan-s commented Oct 2, 2020

Build conda on merge to main #1247

Build conda on merge to main #1247

Conversation

ryan-s commented Sep 30, 2020

Pull Request Description

freddyaboulton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 30, 2020 • edited Loading

Codecov Report

dsherry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton Oct 1, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton commented Oct 1, 2020

ryan-s commented Oct 2, 2020

codecov bot commented Sep 30, 2020 •

edited

Loading

freddyaboulton Oct 1, 2020 •

edited

Loading