Liu 128

[pritchardn]

Full 'end-to-end' multiprocessing tests have been skipped on machines with < 4 (logical) cores.
The added components of a shared-memory manager and process wrapper are still present.
I now repeat the message written in preparation for the pull request:

The summary

• O-boy this sure was more than a 2-line fix. The overall idea remains to execute each drop in a separate process provided by Python.multiprocessing and to replace the memory-io InMemory drops with a new shared-memory-io implementation but the devil is in the details as with all things.

• We use a (slightly) custom Multiprocessing.processsub-class and implementation of shared memory to allow for named shared memory blocks

• In its current form this addition breaks Python 3.7 support (must be >=3.8) and well and truly breaks windows support (would need to implement shared-memory for Windows). Import guards are in place to continue supporting Python 3.7

Changes

• Drops now re-compute their checksum if it does not exist but the drop has COMPLETED. Shared memory drops present themselves to the main process as if they were written externally

• Drop Proxys now have their own context, rather than treating the node-manager itself as the RPC endpoint. While this consumes many more resources than is truly necessary, this is the fastest way (to me at least) to implement this functionality.

• The -t flag for node managers has changed behaviour:

  • 0 (default) - Single process - multi-threading (what was there before)
  • 1 - Maximum processes - multi-threading (maximal performance)
  • 1 - Single process specifically - equivalent to 0 (to avoid overhead)
  • n - Use at most n-processes
  • The decision to make the old behaviour the default is to avoid ‘breaking’ any sort of deployment, while tested, it feels judicious to opt-in to multithreading at first until this is shown to be robust in practice

• MemoryDrops now check for the existence of a thread-pool argument, which if present uses a shared_memory io handle instead of the normal handle. There is no ‘shared-memory drop' explicitly as such.

Additions

• process.py implements DlgProcess - a lightweight wrapper around Multiprocessing.process. The only major difference is that a pipe is opened between the parent and child process allowing exceptions to be passed back to the node manager for handling.

  • This offers an opportunity to hand other information up to the manager if needed later.

• shared_memory.py - (re)-implements the shared-memory primitive from Multiprocessing to allow for named shared memory in addition to catching some peculiar edge-cases this invites (files that exist in one process but not another for instance)

• io.py - Adds a shared-memory IO module that attempts to ensure a minimally sized shared memory chunk

• shared_memory_manager.py - A small module that handles the registration and unlinking of shared memory blocks on a per-session basis.

• Currently, there could be a name-collision for drops with the same oid and the same session name (only a problem if setting the session id explicitly).

  • Multiple managers referring to the same shared memory block will aggressively unlink the same location, this is a caught exception and is reported as a warning

• Some mention of this feature in the documentation, introduction.rst, graphs.rst and managers.rst specifically

• All of these features are tested separately, the test_dm suite has been duplicated for parallel testing - mileage seems to vary on Travis but works locally

I hope this provides a decent summary, and I am open to making further changes, but for now, this seems to suffice.

[awicenec]

Can you resolve the merge conflicts with master first, please?

[coveralls]

Coverage decreased (-0.3%) to 78.004% when pulling c8d7f15 on LIU-128 into 7fa67ff on master.