HICAT-994 Device servers using Python shared memory managers #243

jamienoss · 2021-11-30T14:39:19Z

Upstream PR for https://github.com/spacetelescope/hicat-package/pull/607.

See catkit/testbed/README.md

Changes to existing key features:

Infrastructure code has been factored out of Experiment into catkit.testbed.experiment.Testbed. This is mainly just the safety tests, however, the hicat derivation will also include ownership of all of the remote servers, i.e., the exception-handler, all device servers, the testbed state server, and the sim server.
Experiment.start() (that on the parent process) doesn't run the safety checks, these are now run on a child process by Experiment.Testbed.
Synchronization between processes, especially for halting upon a safety exception, is now implemented using events.
For concurrency to work, nothing should ideally be run on the parent process such that a call to Experiment.start() doesn't block the start of another experiment.
Experiment.start() doesn't wait and block until the experiment completes, it returns just as Process.start() does.
Experiment.join() has been added and mimics Process.join().
Experiment.run_experiment() (that run on the child process) monitors synchronization events from a dedicated monitor thread. This monitor thread will interrupt the main thread (that running Experiment.experiment()) upon certain events.
The outer script pattern is now:

with Testbed():
  experiment.start()
  experiment.join()

For concurrent usage, the out script pattern is now:

with Testbed():
  with experiment1:
    with experiment2:
      pass  # experment2.join() will then be called.

In the case that the desired start and wait/join order prevents the use of nested context managers, multiple try-finally blocks must be implemented such that all joins are executed in the correct finally statement.

jamienoss · 2021-12-01T05:40:58Z

catkit/interfaces/DeformableMirrorController.py

@@ -15,3 +18,24 @@ def apply_shape_to_both(self, dm1_shape, dm2_shape):
    @abstractmethod
    def apply_shape(self, dm_shape, dm_num):
        """Forms a command for a single DM, with zeros padded for the DM not in use."""
+
+#     # A "static" (non-autoproxy) proxy example. NOTE: Using this makes no difference to performance.


Kept as an example.

ehpor

Aside from the (trivial) merge conflict during the merge of #240, I don't see much wrong with this PR. I approve this PR for merging, and will not do detailed notes since that's totally uninteresting for everyone.

Some general comments.

A lot of the code in this PR is very advanced, to the point that maintenance will likely (read: certainly) become an issue in the future. However, I feel that this is to some degree unavoidable with the current state of catkit.
The mutex patterns are still unclear to me. This is the main concern for me for future unexpected bugs (from a user perspective at least).
I really like the new Testbed class, making it do the safety checks rather than an experiment. Great separation of concerns.
Thanks for the slight refactor of the data log into a context manager.
Documentation is a bit sparse. Most internal comments are not accessible to a non-expert, and I presume even an expert that is reading it for the first time. However, catkit/testbed/README.md is a good start for people wanting to learn about the internals of catkit multiprocessing. Especially the example in "Hardware access full stack flow" is a good first step.
The signal/events for experiment lifetime and safety are very well structured. See also the arguments of Experiment.__init__() for a further description.

ehpor · 2022-02-04T16:51:51Z

catkit/testbed/experiment.py

+STOP_EVENT = "catkit_stop_event"
+FINISH_EVENT = "catkit_soft_stop_event"
+SAFETY_EVENT = "catkit_safety_event"
+SAFETY_BARRIER = "catkit_safety_barrier"


No enum? No need to change this; just curious.

🤣 Touché

I'll punt this to another PR.

Signed-off-by: James Noss <jnoss@stsci.edu>

* Allow for daemonic experiments. * Return iterator proxy for Camera.stream_exposures. * Add base Instrument proxy that doesn't mutex all client sides calls. * Remove possible deadlock that could result in devices not being closed. * Add NamedEvent * Add FINISH_EVENT to aid user with synchronizing when to finish/stop concurrent experiments. Signed-off-by: James Noss <jnoss@stsci.edu>

Signed-off-by: James Noss <jnoss@stsci.edu>

…e we used to Signed-off-by: James Noss <jnoss@stsci.edu>

Signed-off-by: James Noss <jnoss@stsci.edu>

jamienoss · 2022-02-09T18:07:50Z

@ehpor Just for the sake of it, would you mind re-approving this PR, please? Note that it's just the upstream hicat CI that has failed, as expected.

ehpor

Reapproved.

jamienoss self-assigned this Nov 30, 2021

jamienoss added the new feature label Nov 30, 2021

jamienoss force-pushed the feature/HICAT-994-device-server-using-mem-manager branch from 38095ad to 859fd71 Compare December 1, 2021 05:37

jamienoss commented Dec 1, 2021

View reviewed changes

jamienoss force-pushed the feature/HICAT-994-device-server-using-mem-manager branch 3 times, most recently from f39ad0f to 7c762ac Compare December 3, 2021 22:04

ehpor self-requested a review February 4, 2022 16:33

ehpor previously approved these changes Feb 4, 2022

View reviewed changes

jamienoss added 16 commits February 4, 2022 17:31

HICAT-994 Device Servers

5ba0fb4

Signed-off-by: James Noss <jnoss@stsci.edu>

Add context manager to datalogger and have it call os.makedirs()

e64b320

Signed-off-by: James Noss <jnoss@stsci.edu>

Add authentication

3418ab1

Signed-off-by: James Noss <jnoss@stsci.edu>

Perf improvements

27245ea

Signed-off-by: James Noss <jnoss@stsci.edu>

Isolate DM Proxy

f01083d

Signed-off-by: James Noss <jnoss@stsci.edu>

Don't patch socket.socket

a1a6d2f

Signed-off-by: James Noss <jnoss@stsci.edu>

tidy ups

556e5a9

Signed-off-by: James Noss <jnoss@stsci.edu>

add readme

f51bdb8

Signed-off-by: James Noss <jnoss@stsci.edu>

Another perf test

d9a5bab

Signed-off-by: James Noss <jnoss@stsci.edu>

Perf proxy

47e3249

Signed-off-by: James Noss <jnoss@stsci.edu>

No parallel CI - requires auto porting CATKIT-100

f88559b

Signed-off-by: James Noss <jnoss@stsci.edu>

Add functionality to lock all members of device enum API.

a52405b

Signed-off-by: James Noss <jnoss@stsci.edu>

Extend inline comment

e842dee

Signed-off-by: James Noss <jnoss@stsci.edu>

_thread.interrupt_main() doesn't cut it so raise SIGINT instead - lik…

d2395fa

…e we used to Signed-off-by: James Noss <jnoss@stsci.edu>

Add dummy global authkey

e38a6da

Signed-off-by: James Noss <jnoss@stsci.edu>

jamienoss dismissed ehpor’s stale review via e38a6da February 4, 2022 22:35

jamienoss force-pushed the feature/HICAT-994-device-server-using-mem-manager branch from 47dc705 to e38a6da Compare February 4, 2022 22:35

jamienoss marked this pull request as ready for review February 4, 2022 22:35

fix rebase error

0f74f26

Signed-off-by: James Noss <jnoss@stsci.edu>

jamienoss marked this pull request as draft February 4, 2022 22:39

jamienoss marked this pull request as ready for review February 4, 2022 22:39

jamienoss marked this pull request as draft February 5, 2022 21:54

jamienoss marked this pull request as ready for review February 5, 2022 21:54

Skip perf tests for remote CI

b1532c2

Signed-off-by: James Noss <jnoss@stsci.edu>

jamienoss force-pushed the feature/HICAT-994-device-server-using-mem-manager branch from a26dc27 to b1532c2 Compare February 6, 2022 15:06

jamienoss marked this pull request as draft February 6, 2022 15:07

jamienoss marked this pull request as ready for review February 6, 2022 15:07

ehpor self-requested a review February 9, 2022 18:21

ehpor approved these changes Feb 9, 2022

View reviewed changes

jamienoss merged commit 54cd177 into develop Feb 9, 2022

jamienoss deleted the feature/HICAT-994-device-server-using-mem-manager branch February 9, 2022 19:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HICAT-994 Device servers using Python shared memory managers #243

HICAT-994 Device servers using Python shared memory managers #243

jamienoss commented Nov 30, 2021 •

edited

Loading

jamienoss Dec 1, 2021

ehpor left a comment •

edited

Loading

ehpor Feb 4, 2022 •

edited

Loading

jamienoss Feb 4, 2022

jamienoss Feb 9, 2022

jamienoss commented Feb 9, 2022

ehpor left a comment

HICAT-994 Device servers using Python shared memory managers #243

HICAT-994 Device servers using Python shared memory managers #243

Conversation

jamienoss commented Nov 30, 2021 • edited Loading

Changes to existing key features:

jamienoss Dec 1, 2021

Choose a reason for hiding this comment

ehpor left a comment • edited Loading

Choose a reason for hiding this comment

ehpor Feb 4, 2022 • edited Loading

Choose a reason for hiding this comment

jamienoss Feb 4, 2022

Choose a reason for hiding this comment

jamienoss Feb 9, 2022

Choose a reason for hiding this comment

jamienoss commented Feb 9, 2022

ehpor left a comment

Choose a reason for hiding this comment

jamienoss commented Nov 30, 2021 •

edited

Loading

ehpor left a comment •

edited

Loading

ehpor Feb 4, 2022 •

edited

Loading