Feature/circleci fv3net build push #192

frodre · 2020-03-19T21:41:56Z

This PR adds automated version bumping and CircleCI integration for building/pushing images related to fv3net.

Changes:

CircleCI will now build and push fv3net images to GCR for all releases, or an update to "latest" whenever master is updated
Added files for versioning documentation and tracking changes over time
CircleCI now cache's the fv3net dependencies of the conda environment for the pytest_default job. This significantly shortens the testing process.
Updated build_environment.sh to additionally output a list of all installed packages under .circleci/.installed_build_env_deps. This is used for caching python dependencies on CircleCI and for reproducibility down the line.
EDIT: No longer the case. Updated environment.yml to install fewer packages using conda and more using pip. This also shortens the dependency install time by reducing the conda dependency graph search.

Builds a docker image with fv3net, vcm, and fv3gfs-python installed.

This reverts commit ebb0d4c.

This reverts commit 956936f.

…fv3net

frodre · 2020-03-24T16:43:07Z

Sorry about all the commits! Quite a bit of trial and error involved with CircleCI on this.

Edit: all seems to be good on the home front

nbren12

The bumpversion configuration looks fine to me.

nbren12 · 2020-03-24T16:58:23Z

setup.cfg

+commit = True
+tag = False
+
+[bumpversion:file:setup.py]


Looks Like you got all the ones I added.

nbren12 · 2020-03-24T16:59:16Z

setup.cfg

+search = {current_version}
+replace = {new_version}
+
+[bumpversion:file:Makefile]


The wheels built by the makefile have version strings, but I stopped using the wheels in #193 anyway, so let's not worry about.

Co-authored-by: W. Andre Perkins <frodre@gmail.com>

nbren12

Thanks for putting this together @frodre. I think this will be a big improvement for all of our workflows. I do have some concerns that the changes to the dependency management system will make it harder to debug and find errors due to upstream version changes. Perhaps we should talk on zoom.

nbren12 · 2020-03-24T20:25:58Z

.circleci/config.yml

+          command: cat environment.yml .circleci/.installed_build_env_deps > combined_deps.txt
+      - restore_cache:
+          keys:
+            - v2-fv3net-env-{{ checksum "combined_deps.txt" }}


This is pretty cool. But I worry that this will make it tricky to see if we get regressions when the resolved dependencies in environment.yml change. For instance, when working on #185 the same environment.yml started installing pandas 2.0, which broke xarray. With this caching strategy we would not catch such errors, so the CI might still pass even though people's local environments and manually pushed images fail.

Yeah, I had similar concerns. I feel like the answer could be having some type of more rigorous detailing of versions to install than just the environment.yml. I tried caching the conda pkgs directory and pip directory, but that wasn't any faster than the old mechanism. Perhaps the builds would depend on some usage of the conda list --explicit file and whatever we need to add the pip packages back in could work.

nbren12 · 2020-03-24T20:31:02Z

.circleci/config.yml

 jobs:
-  build:    
+  pytest_default:    


It seems somewhat redundant to build both the docker image with the environment, and the environment here. Maybe we should run these tests within the docker images built below. I think that is what we do the Circle CI fv3gfs builds.

The pytest_default runs as our normal testing framework for branches. Build only calls on merge to master and releases. I think we want to leave the pytest step rather than building images on each branch? We can clarify this in chat.

nbren12 · 2020-03-24T20:31:32Z

.circleci/config.yml

+                use_version=$CIRCLE_TAG
+            fi
+            make build_images VERSION=$use_version
+      - run:


Consider moving the tests here, and deleting the pytest_default job.

nbren12 · 2020-03-24T20:34:43Z

Makefile

-wheels:
-	pip wheel --no-deps .
-	pip wheel --no-deps external/vcm
+# wheels:


Remove this commented code (I know I did this).

nbren12 · 2020-03-24T20:36:47Z

RELEASE.rst

+Release naming guidelines
+-------------------------
+
+If `x.y.z` is the version, bump `y` (minor) on new features or breaking changes, and `z` on smaller changes.


I think this convention should only hold while we are 0.y.z. Once we reach 1.0, we should start using semver, which means we bump x for breaking versions, y for new features, and z for patches.

Yeah, totally makes sense. I used this format for now more for the explanation below to use 'x' rather than '0' for the usage description of bumpversion

nbren12 · 2020-03-24T20:37:46Z

docker/prognostic_run/Dockerfile

-COPY fv3net-0.1.0-py3-none-any.whl /wheels/fv3net-0.1.0-py3-none-any.whl
-COPY vcm-0.1.0-py3-none-any.whl /wheels/vcm-0.1.0-py3-none-any.whl
-RUN pip3 install --no-deps /wheels/fv3net-0.1.0-py3-none-any.whl  && pip3 install /wheels/vcm-0.1.0-py3-none-any.whl
+RUN pip3 install wheel


I think this line can be removed actually.

nbren12 · 2020-03-24T20:42:55Z

environment.yml

+  - pip
+  - h5netcdf>=0.8
+  - h5py>=2.10
+  - netcdf4>=1.4
  - pip:


What is the motivation for moving things to pip? I realize it is faster, but conda is more robust and actually does proper dependency resolution. OTOH, the pip install will change based on the order of the packages here, with the final options taking precedence. For instance, pip install numpy==1.16 pandas==1.17 does not guarantee that the installed numpy verison is 1.16 if pandas requires a higher version. In this case, conda will fail with an dependency resolution behavior. I think this change could make debugging and fixing such errors a lot harder, especially since so few of these packages actually have constrained versions.

I reset it to the old style where most dependencies are resolved through conda.

frodre · 2020-03-25T15:14:47Z

Ready for re-review @nbren12.

Adjustments:

Removed no-longer used code from dockerfiles and makefile
returned to the original environment.yml
Build/push operation on CircleCI now runs tests in the image and only pushes on success. This alleviates some of the issue that the image dependencies might be different from the cached fv3net deps, but as we spoke about, we'll need some more explicit versioning if we want to keep local/builds consistent.
e.g., https://circleci.com/gh/VulcanClimateModeling/fv3net/3068#tests/containers/0

Thanks for the comments/suggestions!

nbren12

Great! LGTM @frodre.

* Fix init race See fsspec/s3fs#257 Maybe Fixes #180 Fixes #167 * add test (oops, rerecord) * add new recording file * restrict pip upgrades * more deps * try that again

frodre and others added 30 commits March 12, 2020 20:25

Add .editorconfig

ca7acdf

Add HISTORY.rst

2af3a50

Add RELEASE.rst

7ad73ea

Update manifest, probably incomplete

c99c8f4

Add version to fv3net init

740f040

Add bumpversion to environment.yml

74f1430

Fix bumpversion path to __init__

65a2d17

Get important image making files from noah

ebb0d4c

Add noahs environment scripts

956936f

Temporary testing with different env setups

17d2867

Refactore online_modules to fv3net

8ca8819

black

5909c73

Create us.gcr.io/vcm-ml/prognostic_run:v0.1.0

cee28fd

Builds a docker image with fv3net, vcm, and fv3gfs-python installed.

Refactor us.gcr.io/vcm-ml/fv3net image build code

305629f

Add build_images makefile target

ff9f938

Add __version__ to fv3net init

b31a099

update prognostic_run_diags configuration

8f9cb3c

black

31930d9

update readme

f994766

Fix table in README

871be05

fix yaml bug

db16a39

pin pandas version to 1.0.1

47d69e3

Revert "Get important image making files from noah"

5b508e4

This reverts commit ebb0d4c.

Revert "Add noahs environment scripts"

c4c6c05

This reverts commit 956936f.

Merge branch 'master' into feature/versioned-fv3net

0993ecd

Fix environment.yml from merge

fc46839

Add conda update to CircleCI file

2132dfc

Merge branch 'refactor/fv3net-online-modules' into feature/versioned-…

8126de2

…fv3net

Add caching to the CircleCI for fv3net deps

c2855bc

Add cache saving

48543f4

frodre added 6 commits March 24, 2020 15:09

Switch to miniconda3-3.7.0

035e60a

Update python using conda

8fcc312

Try with unpulled fv3gfs version

81637a8

Undo testing changes to makefile and dockerfile for prognostic

a78b7fb

Remove testing jobs from CircleCI

51af0a8

Allow wheel dir specification and add pyenv switch for docker

2b2422b

frodre marked this pull request as ready for review March 24, 2020 16:41

Add pip update to make wheel work in python 2.7

eb49d0f

nbren12 mentioned this pull request Mar 24, 2020

Use fv3atm for end-to-end workflow #196

Merged

nbren12 reviewed Mar 24, 2020

View reviewed changes

frodre and others added 8 commits March 24, 2020 17:31

Merge branch 'master' into feature/circleci-fv3net-build-push

fc22e7a

Add in dockerfile features from another branch

b70b0fb

Co-authored-by: W. Andre Perkins <frodre@gmail.com>

Update CircleCI after dockerfile build updates

405c44d

Use push_images

9aa6c70

Add submodule init/update

db17dea

Fix indent on submodule ci step

bd1c4e3

Fix config.yml for image push version->VERSION

ce082df

Final clean up of leftover testing changes

ee03ad2

nbren12 suggested changes Mar 24, 2020

View reviewed changes

frodre added 3 commits March 25, 2020 05:34

Remove unused code to address review comments

f928211

add script to run tests in docker image after build

8fa3e84

Reset environment.yml

f5c9a3f

frodre requested a review from nbren12 March 25, 2020 15:15

nbren12 approved these changes Mar 25, 2020

View reviewed changes

Merge branch 'master' into feature/circleci-fv3net-build-push

5b3a5b5

frodre merged commit 530ee87 into master Mar 25, 2020

frodre deleted the feature/circleci-fv3net-build-push branch March 25, 2020 16:47

spencerkclark pushed a commit that referenced this pull request May 7, 2021

Race (#192)

a8ad012

* Fix init race See fsspec/s3fs#257 Maybe Fixes #180 Fixes #167 * add test (oops, rerecord) * add new recording file * restrict pip upgrades * more deps * try that again

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/circleci fv3net build push #192

Feature/circleci fv3net build push #192

frodre commented Mar 19, 2020 •

edited

Loading

frodre commented Mar 24, 2020 •

edited

Loading

nbren12 left a comment

nbren12 Mar 24, 2020

nbren12 Mar 24, 2020

nbren12 left a comment

nbren12 Mar 24, 2020

frodre Mar 24, 2020

nbren12 Mar 24, 2020

frodre Mar 24, 2020

nbren12 Mar 24, 2020

nbren12 Mar 24, 2020

nbren12 Mar 24, 2020

frodre Mar 24, 2020

nbren12 Mar 24, 2020

nbren12 Mar 24, 2020

frodre Mar 25, 2020

frodre commented Mar 25, 2020 •

edited

Loading

nbren12 left a comment

Feature/circleci fv3net build push #192

Feature/circleci fv3net build push #192

Conversation

frodre commented Mar 19, 2020 • edited Loading

frodre commented Mar 24, 2020 • edited Loading

nbren12 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nbren12 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frodre commented Mar 25, 2020 • edited Loading

nbren12 left a comment

Choose a reason for hiding this comment

frodre commented Mar 19, 2020 •

edited

Loading

frodre commented Mar 24, 2020 •

edited

Loading

frodre commented Mar 25, 2020 •

edited

Loading