cruft removal attempt

[melange396]

An attempt to remove a bunch of old cruft from our dependencies, tests, and builds...

Summary:

• Removes download/gitclone and usage of 5 unnecessary repos (undefx/py3tester, undefx/undef-analysis, cmu-delphi/github-deploy-repo, cmu-delphi/flu-contest, and cmu-delphi/nowcast) from the github actions for CI & performance tests, the delphi_web_python Dockerfile definition, and the dev env installation script.
• Adds local directory mounts to the test runner commands in the github CI action (which now better matches the dev env Makefile commands).
• Removes testing directive from the github-deploy-repo config (aka ./deploy.json) so we can axe py3tester junk. We run tests in many other places, so it isnt really necessary for g-d-r to do it too.
• Incorporates some file move commands into the Dockerfile for the delphi_web_python image, removing need for its old separate bash script (dev/docker/python/setup.sh)
• Passes tests using its own modified CI definitions!

Results:

• (Hopefully) speeds up building and testing, and reduces filesize/disk footprints of images and installations.
• Local dev envs can probably delete their copies of the 5 extraneous repos.

Caveats:

• Some local dev envs may have some small problems after this, if they use nonstandard procedures (particularly because of changes to the building and structure of the delphi_web_python docker image)

Precursor to:

• get rid of "__test_target__" et al #1129
• Move acquisition deployment off of github-deploy-repo #1042
  • Which then aids in:
    • Infra refactor: drop 'delphi' level of directory indirection from driver/repos #938
    • Infra refactor: convert acquisition to an installable package #939

Replaces:

• Remove unused undefx depends from CI #935

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[melange396]

removing this will prevent github-deploy-repo from running local tests during its deployment process

[melange396]

our local wizard, @korlaxxalrok , just successfully tested this on the staging server, so it should be good to go!

[dshemetov]

Have a few tangential comments, some just me thinking through consequences and making sure we don't miss anything in the untangling and some ideas for refactors. If you agree with any of my comments, let me know and I can make separate issues.

But overall, looks good and the fact that it runs in staging means 👍

[dshemetov]

Comment (non-blocking, probably be separate issue): we should move to having CI use the Makefile commands. It would help keep local testing identical with CI and would remove code duplication.

[dshemetov]

Question (non-blocking, separate issue): any reason why unit tests and integrations are separated? In the Makefile both are run together, using that command here would remove this Docker command duplication.

[dshemetov]

Comment (non-blocking): this COPY command is later overwritten by a runtime mount between the same folders we run tests. Consider getting rid of this.

Thinking out loud (non-blocking): looks like we're dropping the top-level repos dir within the Dockerfile. Want to make sure we think through the consequences of this. AFAICT, previously we copied the entire local repos folder over into /usr/src/app/repos and then moved pieces over to /usr/src/app/delphi/ and /usr/src/app/undefx/. So it looks like we're skipping the intermediate step. So is there anything that still relies on that repos folder? The only thing I can find are the Docker testing commands that end with e.g. python -m pytest --import-mode importlib repos/delphi/delphi-epidata/integrations. But that is handled by the first mount command in those Docker commands:

--mount type=bind,source=./repos/delphi/delphi-epidata,target=/usr/src/app/repos/delphi/delphi-epidata,readonly \

So the edit to my first sentence from this sequence of thoughts is: we're not dropping the repos dir, it just gets added by a mount at runtime.

Comment (non-blocking, separate issue): it's tough to mentally imagine the directory structure of Dockerfile when runtime mounts are involved (due to non-locality between Docker run command and the Dockerfile). Maybe we should add a comment at the end of this Dockerfile saying something like "at runtime, these additional directories are present".

Comment (non-blocking, separate issue): a future future solution to this non-locality is to make a unified docker-compose.yml file.

[dshemetov]

Most of my review suggestions are addressed by #1315