change API to get page-level parallelization everywhere

[bertsky]

In #278 we already had the case were the METS got broken due to ctrl+C in a single process. We discussed about parallelization but agreed to defer that topic. Now this is about providing a locking mechanism that allows parallel processing.

Currently it is impossible to run in parallel over a workspace, simply because it is too dangerous: the result of one actor could be lost due to another without any indication of it. This hampers makefilization and similar efforts.

IIUC, getting FS locking in a portable way is tricky, if you want to get it right.

Does anyone have experience or solutions?

[wrznr]

No experience let alone solutions. Sry.

[kba]

Portability isn't of the essence though because I don't foresee the need to support non-POSIX filesystems.

No experience doing this with Python but the principle is simple and https://github.com/dmfrey/FileLock/blob/master/filelock/filelock.py (the accepted answer you linked to) looks like a reasonable solution.

[bertsky]

I'd like to bring in another idea: Maybe file locking is the wrong way to think of page-level parallelization altogether. We have dependencies between successive processors in a workflow, so it does not make much sense to think of running these in parallel (without going into elaborate synchronization mechanisms like semaphores for all pages), except for a very narrow set of circumstances where workflows contain parallel branches. On the other hand, what really can be run independently (at least for most processors and workflows) are the pages within one processing step. Even now, a processor implementation can do that, e.g. by using a multiprocessing.Pool.map() on the workspace.input_files.

And there's the rub really: In the current design, Processor.process() implementations must iterate over input files themselves. The responsibility for parallelization therefore is in the module provider (instead of the base class in core).

So for 3.0, what we should aim at is the following:

1. deprecate Processor.process()
2. create Processor.process_page() as an abstract method, taking a single OcrdFile as input – and change all implementations in the modules accordingly (which is a lot of work, but trivial).
3. create Processor.process_workspace(), iterating over the input files and delegating to process_page() respectively. Here we have the freedom to use a CPU-parallel implementation under user control. (Of course, as in the workspace-parallel scenario, we then have to make sure GPU-enabled processors do not over-allocate the GPU resources and cause memory races; either using a CPU-fallback or retreating to a safe GPU "bottleneck".)

As a side note: One could of course always argue that workspace-parallel execution is much less complicated and ensures maximum throughput without extra design/implementation effort and synchronization overhead. But this does not cover the use-case of a single (large) workspace, or the use-case of minimum latency.

I guess this proposal should go through the spec first?

[kba]

It's a good proposal, while the Processor.process approach with a thin wrapper around the METS was a neat idea at the time, it has run its course and breaking this up will allow us more flexibility. I also agree that iterating over the files from within the processor is not a scalable approach and also leads to unnecessary and error-prone copy-paste boilerplate in the processors. We should also make sure that the process_* methods are implemented in a way that there are ways to augment them at runtime with hooks or a method convention like "processor implements a method _process_page" while the BaseProcessor offers flexible process_page method wrappers.

I guess this proposal should go through the spec first?

Not really, the workspace-processor approach in core is just one implementation and while it will be some work to refactor the processors, they will still follow the same CLI, just that the code of the processors can get rid of a lot of loops and inconsistent implementations across processors. From what I see this change should not result in any changed behavior for users? I will think about it some more though, quite possible I'm missing something here.

[bertsky]

One thing we have to take into account are processors that run on multiple input file groups. For those, process_page will have to take a tuple of OcrdFiles – for the same pageId. Our base implementation process_workspace will have to provide for that. (And if no full matches can be found for some pages, then either skip with an error or provide the processor with a None input.) This is what ocrd-cor-asv-ann-evaluate or ocrd-cis-align do on their own now. But there might be additional concerns in the processor, like MIME type filtering (if there is more than 1 file per page in some fileGrp).

There might even be processors which read multi-page files (like COCO segmentation, TIFF or PDF). For example, https://github.com/OCR-D/ocrd_segment/blob/master/ocrd_segment/import_coco_segmentation.py. See OCR-D/spec#142

[jbarth-ubhd]

My toughts about parallelization:

• would it be possible to parallelize single processing steps automatically within given bounds (CPU, RAM)
• this would make it easier to start multiple workflows at a time
• there could be negative effects: CPU+Disk Cache trashing, RAM bandwidth, slowed down i/o due to disk latency ...
• e. g. Imagemagick-Q16 needs about 2Copies * Pixels * 2BytePerChannel * 4 ChannelsPerPixel Bytes of RAM. Parallelization could be limited by environment variables

[stweil]

It is desirable to support this kind of parallel processing, because it allows to get a much faster OCR result for a certain book.

But we should not forget that it is also possible to parallelize on the book level, and that already works of course. As long as there is still a huge number of books waiting for OCR, that kind of parallelization might be sufficient for many users.

Nevertheless thinking about future improvements is reasonable, and thoughts discussed here could also trigger actions starting now. If a processor should process several pages parallelized, it is important to know the required resources. This is something which the maintainers of each processor know best. So the processor description (JSON) could be extended by information like required CPU cores, maximum RAM, number of GPUs and maybe more which allows a best use of the available resources. Some of those values might depend on image size or other factors, so they will be only rough estimates, but even that can help. That additional information is also useful for parallelization on the book level which would overwise have to guess or work with try and error.

[stweil]

A central problem for the page level parallelization seems to be updating the METS file.

Locking that file is only one of the possible solutions. Ideally it requires that the individual page results may be added to the METS file in random order. Writing them ordered could result in processes for higher page numbers which have to wait until lower page numbers are finished, so RAM would have to be large enough for running plus waiting processors.

If each processing of a single page would not write directly to the METS file, but delegate that to a central software instance, that central instance could manage orderly updates of the METS file without any locking.

[jbarth-ubhd]

Or some kind of database, perhaps sqlite3, with conversion from/to mets as first/final step.

[bertsky]

@jbarth-ubhd

* would it be possible to parallelize single processing steps automatically within given bounds (CPU, RAM)

Yes, that's what the above proposal is about. If we change the API for modules, moving from process to process_page, then we can do everything multiprocessing and other libraries can give us in core.

* this would make it easier to start multiple workflows at a time

What do you mean by that? You still cannot process multiple workflows on the same document, because we would need a METS locking/synchronization mechanism for that. And you can already run the same workflow on multiple documents with workflow-configuration now.

* there could be negative effects: CPU+Disk Cache trashing, RAM bandwidth, slowed down i/o due to disk latency ...

Of course, when running page-parallel, we must offer the user ways to say specifically what resources (not) to allocate. And the more intelligence we invest in automatic flow optimization, the less the user has to experiment with the settings.

* e. g. Imagemagick-Q16 needs about 2Copies * Pixels * 2BytePerChannel * 4 ChannelsPerPixel Bytes of RAM. Parallelization could be limited by environment variables

I don't think we should use environment variables for that (at least not exclusively). See here for various mechanisms we could use for global options like that.

@stweil

If a processor should process several pages parallelized, it is important to know the required resources. This is something which the maintainers of each processor know best. So the processor description (JSON) could be extended by information like required CPU cores, maximum RAM, number of GPUs and maybe more which allows a best use of the available resources. Some of those values might depend on image size or other factors, so they will be only rough estimates, but even that can help. That additional information is also useful for parallelization on the book level which would overwise have to guess or work with try and error.

That's a hard problem indeed. Even estimating the (usually non-linear) dependency between input size and resource allocation is a daunting task for implementors. And it also depends a lot on the parameters set by the workflow (e.g. model files or search depth). That's why I think realistically we should be agnostic about resource usage and let the user or workflow engine do the guesswork in real time. (But if there's one information we do need implementors to specify statically, it's whether or not the processor tries to make use of a GPU. See GPU = 1 in workflow-configuration rules for the GPU semaphore.)

A central problem for the page level parallelization seems to be updating the METS file.

That's not true if core itself provides the parallelization (as outline above). The Processor class can serialize the page results according to their pageId and only write back to the METS once afterwards.

This problem only arises if you want to run different workflows concurrently on the same data.