Custom base image support in calibanconfig

[sritchie]

This PR gets us started with an approach for custom base images; this is inspired by @sagravat's work in #20, but I had changed so much about the project organization in the meantime that it felt appropriate to whip up this example.

@sagravat, would love your thoughts on how this feels to use!

The user can supply a "base_image" entry into .calibanconfig.json that can either be:

• a string, or
• a dict with "cpu" and "gpu" keys. Optional is fine, of course.

Any of the DLVM images will automatically expand. I took slight liberty by forcing the job mode into the name and adding a prefix. So we now have these DLVM options:

dlvm:pytorch-cpu
dlvm:pytorch-cpu-1.0
dlvm:pytorch-cpu-1.1
dlvm:pytorch-cpu-1.2
dlvm:pytorch-cpu-1.3
dlvm:pytorch-cpu-1.4
dlvm:tf-cpu
dlvm:tf-cpu-1.0
dlvm:tf-cpu-1.13
dlvm:tf-cpu-1.14
dlvm:tf-cpu-1.15
dlvm:tf2-cpu
dlvm:tf2-cpu-2.0
dlvm:tf2-cpu-2.1
dlvm:tf2-cpu-2.2
dlvm:pytorch-gpu
dlvm:pytorch-gpu-1.0
dlvm:pytorch-gpu-1.1
dlvm:pytorch-gpu-1.2
dlvm:pytorch-gpu-1.3
dlvm:pytorch-gpu-1.4
dlvm:tf-gpu
dlvm:tf-gpu-1.0
dlvm:tf-gpu-1.13
dlvm:tf-gpu-1.14
dlvm:tf-gpu-1.15
dlvm:tf2-gpu
dlvm:tf2-gpu-2.0
dlvm:tf2-gpu-2.1
dlvm:tf2-gpu-2.2

Let me know what you think!

[sritchie]

@ajslone @sagravat one thought... we COULD allow the user to provide a format string, like

"dlvm:tf-{}-1.14"

And then we could splice in the job mode at runtime. Is this good? Then it's easier to specify this from the command line.

{"base_image": "dlvm:tf-{}-1.14"}

or

--base_image "dlvm:tf-{}-1.14"

Thoughts?

[ajslone]

LGTM, this will enable many use cases that have been painful or impossible before.

[ajslone]

much better using config

[sagravat]

LGTM, +1

[codecov]

Codecov Report

Merging #39 into master will increase coverage by 1.00%.
The diff coverage is 96.66%.

@@            Coverage Diff             @@
##           master      #39      +/-   ##
==========================================
+ Coverage   49.51%   50.51%   +1.00%     
==========================================
  Files          28       28              
  Lines        2872     2894      +22     
==========================================
+ Hits         1422     1462      +40     
+ Misses       1450     1432      -18     

Impacted Files	Coverage Δ
caliban/platform/notebook.py	45.00% <ø> (ø)
caliban/docker/build.py	31.84% <50.00%> (+0.35%)	⬆️
caliban/config/__init__.py	100.00% <100.00%> (+28.07%)	⬆️
caliban/platform/cloud/types.py	89.47% <0.00%> (-0.66%)	⬇️
caliban/platform/gke/util.py	71.92% <0.00%> (+0.38%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f641c9d...4d3ed9e. Read the comment docs.