v3.4.0

What's Changed

• Add tests for slice_along_axis and sliding_win_oneaxis. by @cboulay in #104
• Bugfix: ezmsg serve is now venv compatible by @griffinmilsap in #109
• Big docs update by @cboulay in #107
• Inspect tasks instead of assuming call signatures and return types by @griffinmilsap in #121
• Properly scope try/except block to not hide bad settings by @cboulay in #113
• Eliminate extensions; migrated to ezmsg-org. by @cboulay in #128

Functionally, this is a rather small change. Structurally, this is a big change because it moves the extensions out of this repo into their own respective repos, which makes maintenance a lot easier.

Full Changelog: v3.3.4...v3.4.0

v3.3.4

ezmsg 3.3.4 Changelog

Packaging changes

The src layout is a modern standard that isolates our source code from other project resources. This change minimizes the risk of accidental inclusion of unwanted files in the distribution, leads to cleaner packaging and install processes, and contributes to more reliable testing practices. For our developers and contributors, this translates to a more straightforward and error-resistant development workflow.

In conjunction with the migration to a src layout, ezmsg is also excited to convert to poetry for our packaging and dependency management needs. Poetry is a cutting-edge tool that offers several advantages over traditional packaging systems. Read about it here

Core Changes

• Bumped version to 3.3.4 for minor fixes and improvements.
• Improved debugging by adding log line for current log level in __init__.py.
• Enhanced logging for thread shutdown in backendprocess.py.
• Ensured graph_service initialization precedes shm_service in graphcontext.py.
• Enhanced subscriber notifications to include interface information in graphserver.py.
• Broadened exception catching in server.py to handle generic OSError.
• Added error handling for shared memory closure to shmserver.py.
• Simplified shared memory reattachment in subclient.py.
• Several refactors addressing graph context management, code readability, and FIXME comments in backend.py.
• BACKWARD INCOMPATIBILITY: Changes to underlying message logger format breaks loading old message logs.

Extensions (ezmsg-zmq)

• Version of ezmsg-zmq extension has been bumped to 1.1.5.
• Updated ezmsg-zmq with a wait_for_sub setting to control subscriber wait behavior in ZMQSenderUnit.
• Removed poller STATE variable from ZMQSenderUnit

Extensions (ezmsg-sigproc)

• Version of ezmsg-sigproc extension has been bumped to 1.2.3.
• Improvements to the ButterworthFilter to enforce type checking for message processing.
• Refactor of ewmfilter to support scale array broadcasting and customizable zero_offset.
• Enhancement of SampleTriggerMessage hashability by setting unsafe_hash to True.
• Adjustment in Spectrum class to rename output axis based on settings.
• Replaced Oscillator with Counter in test_downsample.py to better align with test requirements and added new assertions.
• Introduction of extensive unit tests for the Counter synthesizer.
• Introduction/reimplementation of almost every signal processing unit as a generator to allow for more comprehensive unit testing and better notebook experiences.
• Extensive unit testing (thank you @cboulay and Wyss!)

Utilities and Version Bumping

• Implemented message output rate limiting in MessageQueue through output_hz parameter.
• Fixed time jump handling in Rate utility.
• Added new transformation utility gen_to_unit for generator to async ezmsg Unit conversion.
• Additional generator utilities introduced into the ezmsg utility module.

Documentation and Examples

• Enhanced the README.md with installation instructions, a list of external extensions, and a new Publications section.
• Introduced an example script ezmsg_generator.py that showcases how to encapsulate computational logic within a generator and integrate it with ezmsg.

Acknowledgments

Thanks to the contributors who have played a role in enhancing the robustness and feature set of ezmsg.

New Contributors

• @hannahgooden made their first contribution in #32

Full Changelog: V3.3.3...v3.3.4

3.3.3: Minor enhancements and Bugfixes

ezmsg.core 3.3.3

• async initialize and shutdown for units
• ezmsg.util.messagequeue supports publishing at a particular rate

ezmsg-sigproc 1.2.2

• Bugfix: SampleTriggerMessages are no-longer frozen

3.3.2: Minor Bugfixes

ezmsg.core 3.3.2

• Global topics supported by passing Enums to Connections
• Clearing MessageCache after unit shutdown. This fixes BufferErrors on exit caused by references contained with the Caches.
• Removed local cache reference from Publisher.
• WIP: Remove proxy topics from graphviz output.

ezmsg-zmq 1.1.3

• Closing monitor socket of ZMQPoller/Sender prior to closing primary socket. This fixes ResourceWarning on shutdown of the ZMQ units.

ezmsg-sigproc 1.2.1

• Creating asyncio.Event with default factory for sigproc filter state. Fixes runtime loop error.

3.3.1: Bugfixes and MessageLogger/Replay Enhancements

ezmsg.core 3.3.1

This pull request primarily addresses a bug introduced in v3.3.0 where multiple components can be run using ez.run syntax. This new feature came with an unforeseen consequence where these components could possibly (and with startling frequency tended to) have duplicate names, resulting in multiple subscribers/publishers sharing the same topic names. This has been addressed with a change to the ez.run call signature, requiring unique names for every component to be run by ezmsg. Additionally, when run this way in v3.3.0, every component was run in its own process which could be circumvented by using force_single_process but ultimately did not give developers the flexibility they needed when running multiple components. To address this, we have added a process_components keyword argument to the ez.run call signature which specifies which components are to reside in dedicated processes. By default, all components passed to ez.run will now run in the same process (unless sub-components specify their own process_components).

Example v3.3.1 call signature for ez.run

ez.run(
    # All components are specified as name: component pairs.
    # This construction requires all components to have unique names at this level
    # All caps are used for these names in this example just for convention.
    REPLAY = replay, 
    PREPROC = preproc, 
    SEQ_COLL = sequence_collector, 
    FILT_COLL = filt_collector,
    TERM = term,

    # Use the root_name keyword argument to have all of these units reside under a designated root collection
    root_name = "PROJECT",

    # Connections between these components and streams can be specified just as before
    connections = [
        (replay.OUTPUT_MESSAGE, preproc.INPUT_SIGNAL),
        (preproc.OUTPUT_SEQUENCE, sequence_collector.INPUT_MESSAGE),
        (preproc.OUTPUT_FILTERED, filt_collector.INPUT_MESSAGE),
        (sequence_collector.OUTPUT_MESSAGE, term.INPUT)
    ],

    # Here, you can specify which of the components passed into ez.run should have a dedicated process.
    process_components = [preproc]
)

If you used to call ez.run with a single component, you should consider adapting your code from ez.run(system) to ez.run(SYSTEM = system). Although your code will still function when passing a single component, you should receive a little warning from your typechecker, and this usage is now considered deprecated, and you will receive a DeprecationWarning

Additional features introduced:

• ezmsg.util.messagelogger.MessageLogger now logs timestamps for every message in the text file
• ezmsg.util.messagereplay.MessageReplay can be run dynamically by publishing Path objects; message rate can be controlled to replay files as fast as possible, at specified message rates, or "as recorded" using these new timestamps
  • NOTE: MessageLogger has a new message at the start of every log file that just indicates the start time of the log file. This is currently used by MessageReplay when replaying messages "as recorded". If you've been using ezmsg.util.messagecodec.MessageDecoder to directly decode json messages from these files, you may now notice a LogStart message in your file that you didn't put there. To address this, ezmsg.util.messagecodec.message_log has been introduced as a generator that opens the log file, decodes the messages using MessageDecoder and skips over this LogStart message.

Bugfixes

• Typing for process_components only requires a typing.Collection[Component] instead of a typing.Tuple[Component, ...]
• ezmsg.util.messages.axisarray.AxisArray:
  • isel allows integers, slices, and index arrays
  • Users can specify indexers keyword argument, or just use kwargs to specify indexers, just like xarray
• ezmsg.util.messagereplay.MessageLoder is gone. It was not a meaningful addition to the core.
• Fixed tests/test_perf.py message channels buffers testing.
• Addressed #28

3.3.0: Quality of Life Enhancements

ezmsg.core v3.3.0

• Better extension packaging (with consistent naming for single-file extensions)
• Better flow control with asyncio.sleep(0) after every publish. Note: this halves performance of the perf test, but maintains flow without constant backpressure warnings for systems that publish really fast
• New message utilities in ezmsg.util
  • ezmsg.util.messages.axisarray -- AxisArray fully fleshed out
    • Includes xarray-like sel and isel functionality for slices along dimensions
  • Finally removed StampedMessage
  • ezmsg.util.terminate -- TerminateOnTotal (messages received) and TerminateOnTimeout units introduced
  • ezmsg.util.messagecodec -- Refactored JSONEncoder and JSONDecoder for ezmsg.util.messagelogger. This codec now properly serializes/deserializes message dataclasses just like pickle, but with a human-readable plaintext message log format! NB. MessageEncoder and MessageDecoder moved here from ezmsg.util.messagelogger and the json file format has changed ever-so-slightly (and backward compatibly) to support this.
  • ezmsg.util.messagereplay -- Introduces a MessageReplay unit that can deserialize and publish messages from message files written by MessageLogger. Also introduces a MessageCollector that just saves all received messages to a list in the STATE; can be accessed after execution completes from a notebook if run in the main process.
  • ezmsg.util.messagequeue -- HUGE enhancement that allows ezmsg systems to run with "squishiness" and "leakiness" -- allowing your system to accumulate a backlog of messages (that leaks if/when you want) to prevent a slow downstream unit from backing up your whole system.
• ez.run
  • force_single_process keyword argument which tries to run provided components all in one process -- particularly useful for notebook operation
  • Can now pass multiple components to run, and connect them directly
• GraphServer and SHMServer Enhancements
  • Servers are now run in threads with dedicated event loops as opposed to separate processes. This dramatically speeds up startup on Windows
  • Servers will check for $EZMSG_GRAPHSERVER_ADDR and $EZMSG_SHMSERVER_ADDR (strings formatted as hostname:12345) environment variables -- and if defined, ezmsg will force a connection to these addresses.
  • Reserved ports for GraphServer, SHMServer are 25978 and 25979 respectively. If the environment variables are not defined, ezmsg will first check these reserved ports on 127.0.0.1 and try to connect to servers running there. the ezmsg start command starts servers on the default ports and is a handy way for multiple systems on the same PC to dynamically interact with each-other
  • If the environment variables are not defined and there are no servers running on the default ports, ezmsg will force-start new servers on unoccupied ports starting from 10000 (or $EZMSG_SERVER_PORT_START) -- but avoiding default ports for the services.
• Command Line Changes
  • ezmsg start now reports forked PID
  • ezmsg graphviz writes a DOT-formatted graph specification to stdout; can be piped to graphviz to visualize the current running graph.
• Fixed bugs
  • Moved __version__ attribute to ezmsg.version fix pip install
  • "Single-process-mode" isn't special anymore; main process is always used for execution of one group of units
  • ezmsg will always start a new threaded eventloop for tasks; because of @ez.main code
  • Better KeyboardInterrupt handling (especially on Windows)
  • Unit.Initialize() and creation of State is now called from the task thread (and asyncio synchronization primitives will automatically be associated with the correct event loop
  • Significantly cleaned up code pertaining to connections to the GraphServer and SHMServer; reduced redundant code between these services significantly as well.
  • No-longer forcing WindowsSelectorEventLoop on Windows for asyncio.
  • All leaked ResourceWarnings resolved; all asyncio operations now threadsafe-ly implemented.
  • Resolved ConnectionResetError and BrokenPipeError on shutdown.
  • Settings now decorated with @dataclass_transform and type-checking should resolve settings fields appropriately.
    • This required introducing ezmsg's FIRST non-stdlib dependency; typing_extensions which is required to backport this typing PEP to Python 3.8. typing_extensions has no external deps and is pure-python; unlikely to break in the future.
  • Settings classes no-longer automatically generate unsafe_hash functions. This behavior can be reintroduced by creating your own dataclass definition with unsafe_hash=True, frozen=True and inheriting from Settings.
  • Tasks/Publishers/Subscribers/Threads, etc.. now have attributes that are less likely to collide with user-code
  • Deprecated ezmsg.testing

Extensions

ezmsg.sigproc v1.2.0

• All signal processing modules rewritten to operate on AxisArrays
  • All units allow you to specify named axis to operate on (usually time axis)
• New spectrum (FFT) unit in ezmsg.sigproc.spectral
• Window now allows newaxis accumulation for batching
• TimeseriesMessage deleted
• TSMessage is now a function that returns a properly formatted AxisArray

ezmsg.websocket v1.1.1

• __version__ attribute
• Moved units from ezmsg.websocket to ezmsg.websocket.units
• Added a pyproject.toml

ezmsg.zmq v1.1.1

• __version__ attribute
• Moved units from ezmsg.zmq.sender and ezmsg.zmq.poller to ezmsg.zmq.units
• Added a pyproject.toml

3.2.3: Refactored Extensions

This release contains slight refactors to the core ezmsg extensions in addition to bugfixes for a bunch of annoying behavior secondary to attaching/detaching from running graphs.

Extension updates:

• ezmsg-sigproc: V1.1.1: Fixes an issue where ezmsg.sigproc.sampler.Sampler was subscribing to an OutputStream. Also now has a __version__ attribute.
• ezmsg-zmq: V1.1.0: Slight refactor to permit inclusion of a __version__ attribute. ZMQPollerUnit and ZMQSender now exist in ezmsg.zmq.poller and ezmsg.zmq.sender respectively.
  * ezmsg-websocket: V1.1.0: Fixes typing information for websockets library, introduces a __version__ attribute. Also refactors WebsocketClient and WebsocketServer to exist in ezmsg.websocket.client and ezmsg.websocket.server respectively. WebsocketSettings is still located in ezmsg.websocket -- There was a packaging error, this package will be updated in a future release

Bugfixes:

• WARNING-level log messages for "high water mark" on individual pub-sub channels due to subscriber backpressure
• Fix for BufferError secondary to re-publishing a message attained with zero-copy=True. The backend will now check if a unit attempts to re-publish a zero-copy'd message and will instead broadcast a deepcopy of the message instead.
• Fix for spurious dictionary size change on iteration when detaching from a running system

Quality-of-life:

• ezmsg will now raise exceptions on subscribing to OutputStreams and publishing to InputStreams within a Unit.

3.2.2: Windows fixes

This bugfix release addresses some issues with using ezmsg on Windows. Specifically, Ctrl+C interrupting of multi-process systems was non-functional, and this has now been addressed. There's also been some changes to the servers and test backend that reduce the time necessary for GitHub CI to pass tests on Windows.

Accelerating tests on Windows required a small feature addition to the ezmsg command line interface. There is now an ezmsg start command that will fork the core ezmsg servers and return once they're fully up/initialized. The core ezmsg servers (GraphServer and SHMServer) take a pretty long time to start up on windows, so we now call ezmsg start to set them up before testing, then ezmsg shutdown once testing has concluded.

3.2.1: Attach Hotfix

Version 3.2.0 required a sync() operation when attaching to a running graph, which is not actually required. This sync was causing attach failures, so it's been removed. Additionally, this version has a fix for tests/test_attach.py which had a race condition and would sporadically fail/hang forever. There are also improvements to debug output (in the ezmsg logger at the DEBUG log level).

3.2.0: Single Process Mode

This release enables several new features:

• Single process mode: If all tasks would run in one process, and a GraphServer/SHMServer combo are already up, ez.run will not attempt multiprocessing. Critical for using ezmsg in REPL/notebook environments.
• Associate Publisher and Subscriber objects with OutputStream and InputStream instead of individual tasks:
  • Should drastically reduce the number of connections/sockets required for systems, enhancing performance
  • Allows for control over individual OutputStream...
    • hostname/port specification; allowing outbound TCP communications to other PCs
    • number of buffers; down to 1 buffer for direct control of channel latency and backpressure
    • initial SHM allocation size; to avoid resizing SHM while running
    • force TCP message passing; for small message sizes, there's no performance loss and this allows us to run perf benchmarks including TCP transmission without multi machine CI setups.
  • IMPORTANT BREAKING CHANGE -- it is no-longer possible to subscribe to OutputStreams and publish to InputStreams. This was previously possible (unintended) and should be fixed in downstream projects. If you can come up with a valid use-case, please let me know.
• Command line tool that allows for serving/shutting down GraphServer and SHMServer from command line. After installing ezmsg, simply run ezmsg serve from the command line to start up the ezmsg background servers

This pull request also:

• Fixes several small bugs
• Switches SHMServer and GraphServer ports (25978/9). GraphServer now sits at 25978 by default, and SHMServer now sits at 25979 by default.
• SHMServer, GraphServer and the publisher port-range start are now individually configurable by environment variables.
• GraphServer's pause and resume are no-longer asynchronous; don't need to be await'ed
• Shuts down immediately after NormalTermination is raised, rather than waiting up to 0.5 seconds
  • Tests run ever-so-slightly-faster because of this