Automatically build and push a docker image

[jhkennedy]

This PR:

1. adds an GitHub Actions workflow for docker which:

   • Builds a docker image for any PR to main
   • Builds a docker image for any push to main and then pushes the images to the GitHub Container Registry
     • the image will be tagged with the short commit SHA
     • the image will be tagged with latest
   • Builds a docker image for any pushed v* tag (e.g., v1.3.1) and then pushes the images to the GitHub Container Registry
     • the image will be tagged with the short commit SHA
     • the image will be tagged with the version number (e.g., 1.3.1 for a v1.3.1 tag)

2. updates the Dockerfile (again)

   • moved to the top level (repository root)
   • copies in the current repository instead of cloning or installing via git+https
   • includes jupyterlab and ipympl in the environment by defafult

3. updates the environment/requirements files to use the conda-forge version of pyaps3 and pysolid instead of cloning or installing via git+https

Reminders

• Fix regularly updated docker image #123
• Pass Codacy code review (green)
• Pass Circle CI test (green)
• Make sure that your code follows our style. Use the other functions/files as a basis.
• If modifying functionality, describe changes to function behavior and arguments in a comment below the function declaration.
• If adding new functionality, add a detailed description to the documentation and/or an example.

[jhkennedy]

@yunjunz @forrestfwilliams I've added some review comments with more details about some of the choices I've made here.

In particular, I'm interested in if you thinking this is the right approach for tagging the images.

Right now, I'm tagging any push to main latest so if you docker pull ghcr.io/insarlab/mintpy:latest you'll get whatever is on main.

However, latest in the docker world is typically used as "the image I want users to pull" which is generally "the last stable release", so I could also see instead tagging the last release latest and any merge to main as test (or develop or similar)

[jhkennedy]

You can get a container build off this branch like:

docker pull jhkennedy/mintpy:test

You can test the docker container by running:

docker run --rm -it jhkennedy/mintpy:test python tools/MintPy/tests/test_smallbaselineApp.py

Or drop into the container to run the tests:

docker run -it jhkennedy/mintpy:test
cd tools/MintPy/tests
python test_smallbaselineApp.py

You can also run the jupyterlab instance by running:

docker run --rm -p 8888:8888 -it jhkennedy/mintpy:test jupyter lab --no-browser  --port=8888 --ip=0.0.0.0 --NotebookApp.token=mintpy

(Might be worth making a jupyter launch script...)

[yunjunz]

Thank you for the so long desired PR @jhkennedy!!!

I feel that we are better off if splitting the "requirements + circle CI" related changes to another PR. For each of them, I would like to update the related documentation and links as well during the PR review to make sure that 1) code and doc are consistent and 2) we are all on the same page for the related instruction. What do you say?

I have updated pyaps3 in the following aspects, hoping to help this PR:

• moved the repo to the insarlab org (https://github.com/insarlab/pyaps)
• re-structurer the code following the standard practice (together with necessary changes in the code)
• made a new release of version 0.3.0 and upload it to PyPI (https://pypi.org/project/pyaps3/)

We should update the version of pyaps3 on the conda-forge channel accordingly, maybe you could do it?

I will take a closer look at the docker file and workflow related changes later this week.

[yunjunz]

In particular, I'm interested in if you thinking this is the right approach for tagging the images.

Having latest to the last stable release and develop to main sounds accurate and clear to me.

[jhkennedy]

I feel that we are better off if splitting the "requirements + circle CI" related changes to another PR.

@yunjunz Agreed! I'll split those out into a different PR. Changes kinda cascaded on me.

For each of them, I would like to update the related documentation and links as well during the PR review to make sure that 1) code and doc are consistent and 2) we are all on the same page for the related instruction. What do you say?

Yes! I wanted to make sure the tagging pattern matched what you and @forrestfwilliams thought before investing in the docs.

I have updated pyaps3 in the following aspects, hoping to help this PR:

• moved the repo to the insarlab org (https://github.com/insarlab/pyaps)
• re-structurer the code following the standard practice (together with necessary changes in the code)
• made a new release of version 0.3.0 and upload it to PyPI (https://pypi.org/project/pyaps3/)

We should update the version of pyaps3 on the conda-forge channel accordingly, maybe you could do it?

That sounds great! I'll work on the conda-forge PR first, then split out the requirements into a seperate PR that can go before this one, and then we can all circle back to this one, giving @forrestfwilliams a little time to look it over.

[jhkennedy]

In particular, I'm interested in if you thinking this is the right approach for tagging the images.

Having latest to the last stable release and develop to main sounds accurate and clear to me.

👍, @forrestfwilliams does that match your feeling as well?

[forrestfwilliams]

Hi @jhkennedy, yes I think it makes sense to split this into two separate PRs as @yunjunz suggested. As for the image tags, I like the idea of having the latest tag correspond to the last stable release and the develop tag correspond to main. I think people not involved in this discussion will find this to be more intuitive.

[jhkennedy]

Okay, conda-forge/pyaps3-feedstock#2 is open, so the PyAPS v0.3.0 should be available soon.

[jhkennedy]

@yunjunz I'll have time to pick this up again tomorrow -- I've been slammed the last couple months

[yunjunz]

Sounds great @jhkennedy, please take your time.

I have been using pip-install for the development version of mintpy as you suggested, instead of path setup, and it feels great! With a few minor fixes that will be included in the next release, I believe the conda-installed version of mintpy will be functioning in its full capacity.

[yunjunz]

I rebase this PR against the latest insarlab/main branch. I tried the jhkennedy/mintpy:test which has version 1.3.1, and it works well. It would be great if we could finish this PR @jhkennedy @forrestfwilliams.

Regarding the documentation, I could think of the following:

1. add a link to the mintpy docker image registrary?
2. in docs/installation.md: how to pull from the latest or stable version; explain the two tags; and how to update them
3. since we are updating the docker image for every commit, there is no need to write a note on how to build the image from scratch, right?
4. the various styles of running the docker image, as in the current docs/docker.md

A question I have been wondering regarding docker is: for users who want to use isce2 and mintpy together via docker, what would be your recommended way to proceed?

[jhkennedy]

Thanks @yunjunz I think I'll be able to work on this again next week some -- I'll work on addressing 1-4.

A question I have been wondering regarding docker is: for users who want to use isce2 and mintpy together via docker, what would be your recommended way to proceed?

There's a lot of options -- easiest is create a new conda environment for isce2 when you jump into the container. It's also possible to build a docker image that combines an ISCE2 image and this image though I'd have to look at how the ISCE2 one is laid out. We could also build an ISCE2 env into this container...

[yunjunz]

A question I have been wondering regarding docker is: for users who want to use isce2 and mintpy together via docker, what would be your recommended way to proceed?

There's a lot of options -- easiest is create a new conda environment for isce2 when you jump into the container. It's also possible to build a docker image that combines an ISCE2 image and this image though I'd have to look at how the ISCE2 one is laid out. We could also build an ISCE2 env into this container...

I like the idea of adding isce2 as another dependency to this container, so that the users could import isce as a library and run mintpy on the output of isce2 stack processors. It's simple on our side (don't have to deal with the isce2 setup for topsStack/stripmapStack). Although one won't be able to run isce2 data processing here, he/she could do that in another dedicated isce2 container.

[jhkennedy]

@yunjunz finally getting back to this -- I just forced push rebasing onto insarlab/MintPy@main so I can see where we're at and will work on the outstanding stuff now

[jhkennedy]

@yunjunz and @forrestfwilliams alright, I updated the documentation now as well, so I think this is ready for another review.

I did push a couple container images to insarlab/mintpy in the GitHub Container Registry, so you can:

• Pull a container image built with v1.3.3 like: docker pull ghcr.io/insarlab/mintpy:v1.3.3 or
  docker pull ghcr.io/insarlab/mintpy:latest
• Pull a container image built from this branch like: docker pull ghcr.io/insarlab/mintpy:v1.3.3-38-g6690f9a1
  

Note: By default the MintPy container images repository is set to private, so any MintPy maintainer can pull these as long as they log into ghcr.io with docker, but general users won't be able to until the repository is made public.

@yunjunz if you want to make the container repository public, you (or another owner) will need to go to:
https://github.com/orgs/insarlab/packages/container/mintpy/settings
and scroll to the "Danger Zone" at the bottom and change the visibility to public:

You may also notice the container "packages" show up on the side of the GitHub repository now (if you have access to them):

[jhkennedy]

@yunjunz @forrestfwilliams I pushed some changes to make the Jupyter integration a little easier, and updates the docs on how to start the Jupyter server. As such, I pushed an updated image based on this PR:

docker pull ghcr.io/insarlab/mintpy:v1.3.3-39-g5f718b2b

I'll rebuild and push the v1.3.3/latest images again once this PR is merged so they match all the documentation.

Also, Codacy is complaining about two lines in the scripts/jupyter_notebook_config.py file, but those are written and distributed by the Jupyter Team so I think they can safely be ignored.

[yunjunz]

Thank you @jhkennedy for finishing this exciting PR!!! Both the implementation and documentations look great. I have changed the container image (package) visibility to public, following your suggestion.

Testings, including tools/MintPy/tests/test_smallbaselineApp.py, on my laptop work very well, except for one issue: when I started an interactive shell session as below, view.py seems to work well (normal print-out messages), except that the plotting window is not popping up. Any idea?

docker run -it -v ~/data:/home/mambauser/data --name mintpy ghcr.io/insarlab/mintpy:latest
cd data/test/FernandinaSenDT128/mintpy
view.py velocity.h5

FYI, view.py velocity.h5 --nodisplay also works (successfully plotted and saved the figure into a file). So this seems to be an interactive plotting window issue.

[jhkennedy]

except for one issue: when I started an interactive shell session as below, view.py seems to work well (normal print-out messages), except that the plotting window is not popping up.

@yunjunz I'm not sure -- we're using a pretty minimal base image so it might be a missing x11 or the like; I'll see if I can get it going

[jhkennedy]

@yunjunz the issue does come down to X.Org windows system issue, that is a bit annoying to work around and can induce some security risks. Further reading:

• https://stackoverflow.com/questions/46018102/how-can-i-use-matplotlib-pyplot-in-a-docker-container
• https://mybyways.com/blog/running-linux-gui-applications-in-a-docker-container-using-xpra

There's a couple tools that solve a lot of the issues that are more-or-less straightforward to use that set up X.Org windows system between the host and the container:

• x11docker on Linux
• x11docker plus XQuartz on Mac

But since we include Jupyter, that's the easiest/best path to plotting figures in the container.

[yunjunz]

Thank you very much for the useful information regarding GUI @jhkennedy. I agree that Jupyter is a good and easy path for us right now. More work on the GUI in a Docker can be done laterward in another PR, I have created an issue for it (#801).

As for this PR, it looks all great to me. Cheers!

[yunjunz]

The Github workflow failed with the following error, which seems to be related to some permission issue. @jhkennedy I have added you as a maintainer role. Any idea how we could resolve this?

https://github.com/insarlab/MintPy/runs/7099087138?check_suite_focus=true#step:6:1603

#12 ERROR: unexpected status: 403 Forbidden
------
 > exporting to image:
------
error: failed to solve: unexpected status: 403 Forbidden
Error: buildx failed with: error: failed to solve: unexpected status: 403 Forbidden

[jhkennedy]

Hmm, can you confirm that the insarlab/MintPy actions setting -> workflow settings:
https://github.com/insarlab/MintPy/settings/actions

Looks like this:

I don't think I can look at the org/repo setting without admin or owner level permissions.

[jhkennedy]

@yunjunz I think I found the setting needed:
https://github.com/orgs/insarlab/packages/container/mintpy/settings

Re-running the action now to check

[jhkennedy]

@yunjunz that did it!
https://github.com/insarlab/MintPy/runs/7100056427?check_suite_focus=true

[yunjunz]

Great!