Allow building with thin module Docker containers

[mikegerber]

In #68 @bertsky :

But the real problem is that TF2 dependencies are lurking everywhere, so we will very soon have the unacceptable state that no catch-all venv (satisfying both TF1 and TF2 modules) is possible anymore. By then, a new solution needs to be in place, which (at least partially) isolates venvs from each other again.

[mikegerber]

I think OCR-D has to decide if you want a "have it all" environment. Then either separate venvs per processor or some kind of Docker setup could be the solution.

I'd go for a Docker solution because that also solves dependency issues outside of the Python world. And it is actually possible to do without over-engineering too hard. I.e. I use a run one-liner that just calls a container with the right UID and working directory in place.

[mikegerber]

docker run --rm -t  --user `id -u`:`id -g` --mount type=bind,src="$(pwd)",target=/data ocrd_example_processor

[bertsky]

I think OCR-D has to decide if you want a "have it all" environment. Then either separate venvs per processor or some kind of Docker setup could be the solution.

As long as we don't have REST integration, all-in-one is all we have. Isolation within that approach will always be up to demand and interdependencies. Whether we then delegate to local venvs or thin Docker containers is indeed another freedom of choice (which we ideally should also make a user choice here).

I'd go for a Docker solution because that also solves dependency issues outside of the Python world.

Yes, but it may also increase space and time cost unnecessarily under circumstances (depending on which modules are enabled and what platform the host is). So I'd really like that to be for the user to decide ultimately.

And it is actually possible to do without over-engineering too hard. I.e. I use a run one-liner that just calls a container with the right UID and working directory in place.

Yes of course, once we wrap the processor CLIs in another shell script layer, we can make that local venv or Docker run again. And for the latter we only need to ensure we pass on all the arguments. (Perhaps we could even avoid spinning up a new container with each invocation by using --attach somehow.)

The only thing that troubles me with delegating to thin module Docker containers is that we more or less surrender version control. It's really difficult to do that with Docker images solely based on digest numbers. But we could of course petition module providers to use the docker tag mechanism in a certain way.

[kba]

Unfortunately, we have now a situation where both tensorflow (tf2) and tensorflow-gpu (tf1) can be installed side-by-side, so scripts won't fail at startup anymore but when doing the customary import tensorflow as tf, this will lead to runtime errors if the script (or its dependencies!) expect tf1 API but actually get tf2 API.

After discussing with @bertsky I see no alternative to isolated venvs. Implementing this in the Makefile is tedious but doable. However for our Docker builds, we need to decide on a mechanism to create an entry-point "venv broker" script that activates the right environment or similar. I'm stumped on how we can sensibly support ocrd process in this scenario.

But the situation right now, with runtime errors instead of startup failures, is unacceptable, I see no alternative to package isolation. If anyone else does see an alternative, I'd be happy to hear.

[stweil]

Are there plans to upgrade models and software to TF 2? I think it would help to get an overview of all processors which still use TF 1 with an estimation whether and when they will run with TF 2 and who is responsable for that. And we should have all TF 1 based processors in their own group TF1_EXECUTABLES in the ocrd_all Makefile.

As soon as there is a separate group of TF 1 executables, the Makefile implementation could be straight forward by calling make recursively. We'd have the normal venv with all other processors. And we'd have a 2nd virtual environment, maybe simply a subdirectory of the other one, with the TF 1 processors. So all TF 1 processors would be in venv/tf1/bin. We'd have a generic executablevenv/bin/tf1 which can be linked to venv/bin/processor_x and which simply runs venv/tf1/bin/processor_x, passing all arguments and nearly all environment variables unmodified. Only PATH and VIRTUAL_ENV would be fixed.

[kba]

From the direct requirements:

project	tf1	tf2
cor-asv-ann	✗
ocrd_anybaseocr		✗
ocrd_calamari	✗
ocrd_keraslm	✗
sbb_textline_detector	✗

But we'll also have to check transitive dependencies.

[stweil]

ocrd_pc_segmentation depends indirectly on tensorflow<2.1.0,>=2.0.0. Again I have the problem that there is no prebuild tf2 for Python 3.8. That is not available for Python 3.8 and conflicts with tensorflow 2.2.0 which is available.

[stweil]

Because of the close relationship between Python version(s) and available prebuild Tensorflow versions we must be prepared that TF1 might require a different (= older) Python version than TF2. I already have that situation when I want to build with a recent Linux distribution, and because of a bug in Debian / Ubuntu it is currently not possible to create a Python 3.7 venv when Python 3.8 is installed, too. This is of course not relevant for the old Ubuntu which is currently our reference platform.

[stweil]

I'm stumped on how we can sensibly support ocrd process in this scenario.

I did not look into the details of the code. Does it spawn processes for the single steps (then it should work), or do all steps run in the same Python process?

[bertsky]

@stweil

Are there plans to upgrade models and software to TF 2? I think it would help to get an overview of all processors which still use TF 1 with an estimation whether and when they will run with TF 2 and who is responsable for that

The README is up-to-date w.r.t. that.

TF2 migration can be less or more effort, depending on what part of the API the module (or its dependent) relies. TF offers an upgrade script to rewrite the code to use tf.compat.v1 where necessary, but that is not always enough (automatic). True (native) TF2 migration also involves model retraining.

Plus TF 2.2 brings even more breaking changes (as you observed for ocrd_pc_segmentation already).

Since we have modules like ocrd_calamari (which depends on calamari_ocr which depends on TF1 with no visible migration activity) and ocrd_segment (which has a tool that depends on maskrcnn which depends on TF1 and is not maintained at the moment), this cannot be forced. (And forcing it would probably mean a lot of work for cor-asv-ann and ocrd_keraslm, too.)

As soon as there is a separate group of TF 1 executables, the Makefile implementation could be straight forward by calling make recursively. We'd have the normal venv with all other processors. And we'd have a 2nd virtual environment, maybe simply a subdirectory of the other one, with the TF 1 processors. So all TF 1 processors would be in venv/tf1/bin. We'd have a generic executablevenv/bin/tf1 which can be linked to venv/bin/processor_x and which simply runs venv/tf1/bin/processor_x, passing all arguments and nearly all environment variables unmodified. Only PATH and VIRTUAL_ENV would be fixed.

This would work, but why make an exception for TF? We have seen other conflicting dependencies already, and we know that pip is as bad as it gets in resolving these. See above for broader recipes, including delegation to thin Docker containers (possibly even behind REST in the future).

@kba

Implementing this in the Makefile is tedious but doable. However for our Docker builds, we need to decide on a mechanism to create an entry-point "venv broker" script that activates the right environment or similar. I'm stumped on how we can sensibly support ocrd process in this scenario.

The recipe is simple (and has already been discussed above): the top-level PATH directory (which will be a tiny venv merely for ocrd CLI, or /usr/local/bin in a Docker image) contains only shell scripts (created automatically at install-time) which then delegate to the right venv or Docker container. (Which modules are grouped together can be organised centrally in the Makefile.) Why shouldn't we allow venvs inside Docker images?

[kba]

Why shouldn't we allow venvs inside Docker images?

I was not clear: That can also be done with a mechanism like the one you describe. No reason not to use venv in a Docker image, on the contrary.

I'm stumped on how we can sensibly support ocrd process in this scenario.

After sleeping on it, I am indeed unstumped. I misremembered ocrd process to use run_processor but it obviously uses run_cli, so the processors are run with their own python instance (or bash or compiled code) which can be set at install time as we're discussing.

[stweil]

O what a mess! I now tried an installation of ocrd_all with a 2nd venv for sbb_textline_detector as an example of a TF1 processor. The build worked, no obvious conflicts, tensorflow-gpu 1.15.3 correctly installed for sbb_textline_detector. But it also installed Keras 2.4.3. And here is the result when running sbb_textline_detector --help:

ImportError: Keras requires TensorFlow 2.2 or higher. Install TensorFlow via `pip install tensorflow`

@mikegerber, I am afraid that requirements.txt should be more precise regarding the required Keras version.

[bertsky]

But it also installed Keras 2.4.3. And here is the result when running sbb_textline_detector --help:

ImportError: Keras requires TensorFlow 2.2 or higher. Install TensorFlow via `pip install tensorflow`

@mikegerber, I am afraid that requirements.txt should be more precise regarding the required Keras version.

@stweil please report to the respective repo (yes, probably keras < 2.4, maybe even keras < 2.3)!

[stweil]

please report to the respective repo

See qurator-spk/sbb_textline_detection#34

[stweil]

@bertsky, ocrd_keraslm must also limit the Keras version. It currently uses keras >= 2.3.1.

[mikegerber]

The only thing that troubles me with delegating to thin module Docker containers is that we more or less surrender version control. It's really difficult to do that with Docker images solely based on digest numbers. But we could of course petition module providers to use the docker tag mechanism in a certain way.

(Sorry for the late reply to this, I'm reviewing open issues.) I'm trying to understand this and I think you're saying that going from, for example, ocrd_calamari==1.2.3 to the Docker container ocrd_calamari b32098d882d6 would surrender version control? If so, then yes, tagging that container with version 1.2.3 is absolutely required and would have to be done by CI/CD. The same goes for a hypothetical AppImage (my Q&D example just sets the version from pip output).

[bertsky]

The only thing that troubles me with delegating to thin module Docker containers is that we more or less surrender version control. It's really difficult to do that with Docker images solely based on digest numbers. But we could of course petition module providers to use the docker tag mechanism in a certain way.

I'm trying to understand this and I think you're saying that going from, for example, ocrd_calamari==1.2.3 to the Docker container ocrd_calamari b32098d882d6 would surrender version control?

This is not about containers, but images. And digest numbers are the only reliable identification that Docker images get unconditionally (without extra steps at build time). But then digest numbers would have to be mapped to the git submodule commits that ocrd_all already uses, which seems unmanagable to me. So practically I guess everyone would just try to get the most recent image and pray they never have to go backwards.

If so, then yes, tagging that container with version 1.2.3 is absolutely required and would have to be done by CI/CD.

ocrd_all is more fine-grained than version numbers / release tags – it manages the submodules' commits. So if you replace version with commit then yes, that's what I mean. All Docker builds need to automatically include their git revisions.

With that in place, and with some script foo, we could selectively exchange native installations with thin Docker containers per module as needed – to the point where all modules are Dockerized, so the top level (be it native or Docker itself) becomes thin itself.

[bertsky]

Anyway, with #118 ff. the original issue (different TF requirements) has been solved. The topic has since moved on to how do we integrate/compose thin module Docker images (where available) as an alternative, without giving up version control.

I therefore suggest to rename the issue to Allow building with thin module Docker containers.

[bertsky]

And it is actually possible to do without over-engineering too hard. I.e. I use a run one-liner that just calls a container with the right UID and working directory in place.

Yes of course, once we wrap the processor CLIs in another shell script layer, we can make that local venv or Docker run again. And for the latter we only need to ensure we pass on all the arguments.

Since we now have a script mechanism in place delegating to sub-venvs, we could start delegating to thin Docker containers. But we have to consider that we would be calling Docker containers from Docker containers. It's doable, but needs to be accounted for. Especially that the existing mountpoints and bind-mounts need to be passed on. (The situation is different for @mikegerber's solution IIUC, because its outer layer is native, not Docker.)

ocrd_all is more fine-grained than version numbers / release tags – it manages the submodules' commits. So if you replace version with commit then yes, that's what I mean. All Docker builds need to automatically include their git revisions.

With that in place, and with some script foo, we could selectively exchange native installations with thin Docker containers per module as needed – to the point where all modules are Dockerized, so the top level (be it native or Docker itself) becomes thin itself.

So maybe we should start by devising a scheme for including the git version number into all thin/module images. ocrd/all already uses these labels:

• org.label-schema.vcs-ref
• org.label-schema.vcs-url
• org.label-schema.build-date

Let's extend that to all existing submodule images, i.e.

• ocrd/core
• ocrd/olena,
• ocrd/im6convert,
• ocrd/pagetopdf
• ocrd/tesserocr,
• ocrd/fileformat
• ocrd/anybaseocr,
• ocrd/cis (currently flobar/ocrd_cis),
• ocrd/pc_segmentation (currently ls6uniwue/ocrd_pixelclassifier_segmentation),
• ocrd/sbb_textline_detector (currently not on Dockerhub, as this is built differently than in the module itself)

Then we can follow up with a PR here that inserts the docker (pull and) run for the revision of the respective submodule into the CLI delegator script.

[mikegerber]

(The situation is different for @mikegerber's solution IIUC, because its outer layer is native, not Docker.)

Yes, the containers are intended to provide dependency-isolated processors to the native/host-side workflow script ("the outer layer").

[bertsky]

Besides spinning up multiple CLI-only containers (somehow) sharing volumes, we could also integrate containers as true network services, but merely by installing a thin OpenSSH server layer on top of each module's CLI offerings. This was done for ocrd/all in https://github.com/bertsky/ocrd_controller, but the same recipe could be used for each module.

[bertsky]

Besides spinning up multiple CLI-only containers (somehow) sharing volumes, we could also integrate containers as true network services, but merely by installing a thin OpenSSH server layer on top of each module's CLI offerings. This was done for ocrd/all in https://github.com/bertsky/ocrd_controller, but the same recipe could be used for each module.

Not necessary anymore: we now have the possibility to build network services for module processors by either

• starting them as a Processing Worker (connecting to some Processing Server's queues), or
• starting them as a Processor Server

both options could be predefined in a docker-compose.yml, each using the same (module) image but differing entry points (i.e. command attribute):

• ocrd network processing-worker EXECUTABLE-NAME --database MONGO-URL --queue RABBITMQ-URL, or
• ocrd network processor-server EXECUTABLE-NAME --database MONGO-URL --address HOST:PORT

Thus, in ocrd_all, make all could then aggregate and generate two global docker-compose.yml files (one for each mode, i.e. docker-compose.processing-server.yml and docker-compose.processor-servers.yml) which simply delegate to all the services in the modules, e.g.

...
  ocrd-tesserocr-recognize:
    extends:
      file: ocrd_tesserocr/docker-compose.yml
      service: ocrd-tesserocr-recognize
    command: ocrd network processing-worker ocrd-tesserocr-recognize --database $MONGO_URL --queue $RABBITMQ_URL
    depends_on:
      - ocrd-processing-server
      - ocrd-mongo-db
      - ocrd-rabbit-mq
...

...
  ocrd-tesserocr-recognize:
    extends:
      file: ocrd_tesserocr/docker-compose.yml
      service: ocrd-tesserocr-recognize
    command: ocrd network processor-server ocrd-tesserocr-recognize --database $MONGO_URL --address ocrd-tesserocr-recognize:80
    depends_on:
      - ocrd-mongo-db
...

where configuration (i.e. setting environment variables) can happen via .env mechanism or shell.

Now, what's left is generating CLI entry points that delegate to each respective REST endpoint:

ifneq ($(findstring ocrd_tesserocr, $(OCRD_MODULES)),)
OCRD_TESSEROCR := $(BIN)/ocrd-tesserocr-binarize
OCRD_TESSEROCR += $(BIN)/ocrd-tesserocr-crop
OCRD_TESSEROCR += $(BIN)/ocrd-tesserocr-deskew
OCRD_TESSEROCR += $(BIN)/ocrd-tesserocr-recognize
OCRD_TESSEROCR += $(BIN)/ocrd-tesserocr-segment-line
OCRD_TESSEROCR += $(BIN)/ocrd-tesserocr-segment-region
OCRD_TESSEROCR += $(BIN)/ocrd-tesserocr-segment-word
OCRD_EXECUTABLES += $(OCRD_TESSEROCR)
$(OCRD_TESSEROCR): ocrd_tesserocr
	$(file >$@,$(call delegator,$(@F)))
	chmod +x $@
endif

...

define delegator
#!/bin/bash
ocrd network client processing process $(1) "$$@"
endef

So this would create executable files like venv/bin/ocrd-tesserocr-recognize behaving like the true CLI but really just passing arguments to the server (that must have been started before via docker compose start).

This whole approach could replace both the sub-venvs and pip install rules. Of course, it requires that each module provides a Dockerfile (and docker-compose.yml, perhaps even prebuilt images on the registry).

I am not sure I have the full picture of what we should do, though. Thoughts @kba ?

[kba]

I am not sure I have the full picture of what we should do, though. Thoughts @kba ?

This looks fairly complete to me, I'm currently trying out the updated ocrd_network in core and will post a proof-of-concept based on docker-compose like you propose.

[joschrew]

How would you address the problem regarding getting the workspace to be processed to the processing-worker? Currently when running and using ocrd in docker the current directory is volume-mounted into /data of the container.
But when installing core to a venv and delegating the processor calls to the processing-workers, the container is already running. Because of that you cannot invoke the processors on workspaces from anywhere you want (as you could do with ocrd installed to a venv) because the volume mounting must have already been done.

[bertsky]

@joschrew
I don't understand your logic here. You could easily bind-mount a NFS or other distributed storage, both on the Processing Server and on the Processing Worker and on the client side, so IMO accessibility is not the issue.

But we do have to talk about efficiency, in fact we already did – last time was when conceptualising the METS Server. There in particular, I laid out the existing (in the sense of currently available) implicit transfer model (backed by ad-hoc Workspace.download_file calls of the image URLs) vs. some future (in the sense of still unimplemented) explicit transfer model (prior, queued download to fast local storage). In my understanding, the latter is what is assumed in the /workspace part of https://github.com/OCR-D/ocrd-webapi-implementation, i.e. data ingest at workspace definition (but I could be wrong and IIRC there was also a discussion whether it should be allowed to just clone from a METS instead of uploading a complete ocrd-zip).

So these questions unfortunately hinge on a lot of unfinished business:

• METS Server implementation – @kba started a PR
• FLocat URL bookkeeping – @kba said he started implementing this locally
• how to do data pipelining (page by page or workspace by workspace) in the Processing Server and Workflow Server