Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG] Define an interface for Container engines #848

Open
wants to merge 19 commits into
base: master
from

Conversation

@manics
Copy link
Contributor

manics commented Feb 11, 2020

Summary

Closes #682

  • Abstract the interface that builds a Dockerfile (defined in repo2docker/engine.py). Engines are LoggingConfigurable- in future this could allow advanced configuration though the repo2docker config file. This is the file that needs the most review since it defines the specification for others.
  • Implement docker (via docker-py) as the default implementation (see repo2docker/docker.py)

This is based on my attempt at a podman backend in #806

In general most of the Docker Client methods could be easily implemented in a Podman wrapper.

Details

The main exceptions are build() and push_image():

  • The Docker client can take a tar file-like-object as the Docker context. To implement it in Podman you have to untar it into a temporary directory. This is inconvenient, though not a blocker since Python makes it easy to untar.
  • Build logs: The output format used by Docker is unclear, and the content of each line is specific to Docker. In #806 I worked around it by taking whatever podman spits out and wrapping it as {"stream": line} which is enough to convince repo2docker:
    • if "stream" in l:
      self.log.info(l["stream"], extra=dict(phase="building"))
      .
    • In this PR I've decided it's easier to just return strings, and throw an exception to indicate an error. Engine classes have a flag string_output to indicate whether they follow the Docker API in returning objects or if they return plain strings.
  • Push image logs: The docker client outputs logs and progress information as events in a way that isn't portable to podman:
    • if "progressDetail" in progress and progress["progressDetail"]:
      progress_layers[progress["id"]] = progress["progressDetail"]
      else:
      progress_layers[progress["id"]] = progress["status"]
      # include full progress data for each layer in 'layers' data
      layers[progress["id"]] = progress
    • as with build() I've switched to plain strings with the option of keeping the docker format.

Remaining todos:

  • Fix unit tests
  • Remove commented lines/in-progress notes
  • Update podman backend #806 to implement this interface as verification of the design, either in this repo or as an external plugin with a repo2docker.engine entrypoint

Other engines

I've switched to using an entrypoint for ContainerEngine implementations. The default docker engine remains in this project. I've moved #806 (WIP: Podman backend) into a new project: https://github.com/manics/repo2docker-podman. Instructions are included in that project's README.

@@ -11,15 +11,15 @@
import sys
import logging
import os
import entrypoints

This comment has been minimized.

Copy link
@manics

manics Feb 13, 2020

Author Contributor

This is a new dependency, but it's already used in other Jupyter projects such as jupyterhub for managing entrypoints:
https://github.com/jupyterhub/jupyterhub/blob/bc7bb5076ff2deabc7702c75dce208166d14568e/requirements.txt#L4

@@ -3,3 +3,5 @@ repos:
rev: 18.9b0
hooks:
- id: black
# Doesn't work:
# args: [--target-version py35]

This comment has been minimized.

Copy link
@manics

manics Feb 14, 2020

Author Contributor

pre-commit/black formats the files as Python 3.6 but this is incompatible with Python 3.5 and the above config doesn't work.
See the # fmt: off comments in repo2docker/docker.py for the alternative workaround.

This comment has been minimized.

Copy link
@manics

manics Feb 14, 2020

Author Contributor

Hopefully #849 will fix this

tag : str
Tag to add to the image
custom_context : bool

This comment has been minimized.

Copy link
@manics

manics Feb 14, 2020

Author Contributor

custom_context and fileobj and very biased towards the Docker API. However it's fairly easy in Python to untar this tarfile-like-object into a temporary directory so it can be made to work.

@manics manics changed the title WIP: Define an interface for Container engines [MRG] Define an interface for Container engines Feb 14, 2020
@manics manics marked this pull request as ready for review Feb 14, 2020
manics added a commit to manics/repo2docker-podman that referenced this pull request Feb 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

1 participant
You can’t perform that action at this time.