Releases: ezmsg-org/ezmsg
v3.4.0
What's Changed
- Add tests for slice_along_axis and sliding_win_oneaxis. by @cboulay in #104
- Bugfix:
ezmsg serve
is now venv compatible by @griffinmilsap in #109 - Big docs update by @cboulay in #107
- Inspect tasks instead of assuming call signatures and return types by @griffinmilsap in #121
- Properly scope try/except block to not hide bad settings by @cboulay in #113
- Eliminate extensions; migrated to ezmsg-org. by @cboulay in #128
Functionally, this is a rather small change. Structurally, this is a big change because it moves the extensions out of this repo into their own respective repos, which makes maintenance a lot easier.
Full Changelog: v3.3.4...v3.4.0
v3.3.4
ezmsg
3.3.4 Changelog
Packaging changes
The src layout is a modern standard that isolates our source code from other project resources. This change minimizes the risk of accidental inclusion of unwanted files in the distribution, leads to cleaner packaging and install processes, and contributes to more reliable testing practices. For our developers and contributors, this translates to a more straightforward and error-resistant development workflow.
In conjunction with the migration to a src layout, ezmsg is also excited to convert to poetry for our packaging and dependency management needs. Poetry is a cutting-edge tool that offers several advantages over traditional packaging systems. Read about it here
Core Changes
- Bumped version to
3.3.4
for minor fixes and improvements. - Improved debugging by adding log line for current log level in
__init__.py
. - Enhanced logging for thread shutdown in
backendprocess.py
. - Ensured
graph_service
initialization precedesshm_service
ingraphcontext.py
. - Enhanced subscriber notifications to include interface information in
graphserver.py
. - Broadened exception catching in
server.py
to handle genericOSError
. - Added error handling for shared memory closure to
shmserver.py
. - Simplified shared memory reattachment in
subclient.py
. - Several refactors addressing graph context management, code readability, and
FIXME
comments inbackend.py
. - BACKWARD INCOMPATIBILITY: Changes to underlying message logger format breaks loading old message logs.
Extensions (ezmsg-zmq)
- Version of
ezmsg-zmq
extension has been bumped to1.1.5
. - Updated
ezmsg-zmq
with await_for_sub
setting to control subscriber wait behavior inZMQSenderUnit
. - Removed
poller
STATE variable fromZMQSenderUnit
Extensions (ezmsg-sigproc)
- Version of
ezmsg-sigproc
extension has been bumped to1.2.3
. - Improvements to the
ButterworthFilter
to enforce type checking for message processing. - Refactor of
ewmfilter
to support scale array broadcasting and customizablezero_offset
. - Enhancement of
SampleTriggerMessage
hashability by settingunsafe_hash
toTrue
. - Adjustment in
Spectrum
class to rename output axis based on settings. - Replaced
Oscillator
withCounter
intest_downsample.py
to better align with test requirements and added new assertions. - Introduction of extensive unit tests for the
Counter
synthesizer. - Introduction/reimplementation of almost every signal processing unit as a generator to allow for more comprehensive unit testing and better notebook experiences.
- Extensive unit testing (thank you @cboulay and Wyss!)
Utilities and Version Bumping
- Implemented message output rate limiting in
MessageQueue
throughoutput_hz
parameter. - Fixed time jump handling in
Rate
utility. - Added new transformation utility
gen_to_unit
for generator to asyncezmsg
Unit conversion. - Additional generator utilities introduced into the
ezmsg
utility module.
Documentation and Examples
- Enhanced the
README.md
with installation instructions, a list of external extensions, and a new Publications section. - Introduced an example script
ezmsg_generator.py
that showcases how to encapsulate computational logic within a generator and integrate it withezmsg
.
Acknowledgments
Thanks to the contributors who have played a role in enhancing the robustness and feature set of ezmsg
.
New Contributors
- @hannahgooden made their first contribution in #32
Full Changelog: V3.3.3...v3.3.4
3.3.3: Minor enhancements and Bugfixes
ezmsg.core 3.3.3
async
initialize
andshutdown
for unitsezmsg.util.messagequeue
supports publishing at a particular rate
ezmsg-sigproc 1.2.2
- Bugfix: SampleTriggerMessages are no-longer frozen
3.3.2: Minor Bugfixes
ezmsg.core 3.3.2
- Global topics supported by passing
Enums
toConnections
- Clearing
MessageCache
after unit shutdown. This fixesBufferErrors
on exit caused by references contained with theCache
s. - Removed local cache reference from
Publisher
. - WIP: Remove proxy topics from
graphviz
output.
ezmsg-zmq 1.1.3
- Closing monitor socket of
ZMQPoller/Sender
prior to closing primary socket. This fixesResourceWarning
on shutdown of the ZMQ units.
ezmsg-sigproc 1.2.1
- Creating
asyncio.Event
with default factory for sigproc filter state. Fixes runtime loop error.
3.3.1: Bugfixes and MessageLogger/Replay Enhancements
ezmsg.core 3.3.1
This pull request primarily addresses a bug introduced in v3.3.0 where multiple components can be run using ez.run
syntax. This new feature came with an unforeseen consequence where these components could possibly (and with startling frequency tended to) have duplicate names, resulting in multiple subscribers/publishers sharing the same topic names. This has been addressed with a change to the ez.run
call signature, requiring unique names for every component to be run by ezmsg
. Additionally, when run this way in v3.3.0, every component was run in its own process which could be circumvented by using force_single_process
but ultimately did not give developers the flexibility they needed when running multiple components. To address this, we have added a process_components
keyword argument to the ez.run
call signature which specifies which components are to reside in dedicated processes. By default, all components passed to ez.run
will now run in the same process (unless sub-components specify their own process_components
).
Example v3.3.1 call signature for ez.run
ez.run(
# All components are specified as name: component pairs.
# This construction requires all components to have unique names at this level
# All caps are used for these names in this example just for convention.
REPLAY = replay,
PREPROC = preproc,
SEQ_COLL = sequence_collector,
FILT_COLL = filt_collector,
TERM = term,
# Use the root_name keyword argument to have all of these units reside under a designated root collection
root_name = "PROJECT",
# Connections between these components and streams can be specified just as before
connections = [
(replay.OUTPUT_MESSAGE, preproc.INPUT_SIGNAL),
(preproc.OUTPUT_SEQUENCE, sequence_collector.INPUT_MESSAGE),
(preproc.OUTPUT_FILTERED, filt_collector.INPUT_MESSAGE),
(sequence_collector.OUTPUT_MESSAGE, term.INPUT)
],
# Here, you can specify which of the components passed into ez.run should have a dedicated process.
process_components = [preproc]
)
If you used to call ez.run
with a single component, you should consider adapting your code from ez.run(system)
to ez.run(SYSTEM = system)
. Although your code will still function when passing a single component, you should receive a little warning from your typechecker, and this usage is now considered deprecated, and you will receive a DeprecationWarning
Additional features introduced:
ezmsg.util.messagelogger.MessageLogger
now logs timestamps for every message in the text fileezmsg.util.messagereplay.MessageReplay
can be run dynamically by publishingPath
objects; message rate can be controlled to replay files as fast as possible, at specified message rates, or "as recorded" using these new timestamps- NOTE:
MessageLogger
has a new message at the start of every log file that just indicates the start time of the log file. This is currently used byMessageReplay
when replaying messages "as recorded". If you've been usingezmsg.util.messagecodec.MessageDecoder
to directly decode json messages from these files, you may now notice aLogStart
message in your file that you didn't put there. To address this,ezmsg.util.messagecodec.message_log
has been introduced as a generator that opens the log file, decodes the messages usingMessageDecoder
and skips over thisLogStart
message.
- NOTE:
Bugfixes
- Typing for
process_components
only requires atyping.Collection[Component]
instead of atyping.Tuple[Component, ...]
ezmsg.util.messages.axisarray.AxisArray
:isel
allows integers, slices, and index arrays- Users can specify
indexers
keyword argument, or just use kwargs to specify indexers, just likexarray
ezmsg.util.messagereplay.MessageLoder
is gone. It was not a meaningful addition to the core.- Fixed
tests/test_perf.py
message channels buffers testing. - Addressed #28
3.3.0: Quality of Life Enhancements
ezmsg.core v3.3.0
- Better extension packaging (with consistent naming for single-file extensions)
- Better flow control with
asyncio.sleep(0)
after every publish. Note: this halves performance of the perf test, but maintains flow without constant backpressure warnings for systems that publish really fast - New message utilities in
ezmsg.util
ezmsg.util.messages.axisarray
--AxisArray
fully fleshed out- Includes
xarray
-likesel
andisel
functionality for slices along dimensions
- Includes
- Finally removed
StampedMessage
ezmsg.util.terminate
--TerminateOnTotal
(messages received) andTerminateOnTimeout
units introducedezmsg.util.messagecodec
-- RefactoredJSONEncoder
andJSONDecoder
forezmsg.util.messagelogger
. This codec now properly serializes/deserializes message dataclasses just likepickle
, but with a human-readable plaintext message log format! NB.MessageEncoder
andMessageDecoder
moved here fromezmsg.util.messagelogger
and the json file format has changed ever-so-slightly (and backward compatibly) to support this.ezmsg.util.messagereplay
-- Introduces aMessageReplay
unit that can deserialize and publish messages from message files written byMessageLogger
. Also introduces aMessageCollector
that just saves all received messages to a list in theSTATE
; can be accessed after execution completes from a notebook if run in the main process.ezmsg.util.messagequeue
-- HUGE enhancement that allowsezmsg
systems to run with "squishiness" and "leakiness" -- allowing your system to accumulate a backlog of messages (that leaks if/when you want) to prevent a slow downstream unit from backing up your whole system.
ez.run
force_single_process
keyword argument which tries to run provided components all in one process -- particularly useful for notebook operation- Can now pass multiple components to run, and connect them directly
GraphServer
andSHMServer
Enhancements- Servers are now run in threads with dedicated event loops as opposed to separate processes. This dramatically speeds up startup on Windows
- Servers will check for
$EZMSG_GRAPHSERVER_ADDR
and$EZMSG_SHMSERVER_ADDR
(strings formatted ashostname:12345
) environment variables -- and if defined,ezmsg
will force a connection to these addresses. - Reserved ports for
GraphServer
,SHMServer
are 25978 and 25979 respectively. If the environment variables are not defined,ezmsg
will first check these reserved ports on127.0.0.1
and try to connect to servers running there. theezmsg start
command starts servers on the default ports and is a handy way for multiple systems on the same PC to dynamically interact with each-other - If the environment variables are not defined and there are no servers running on the default ports,
ezmsg
will force-start new servers on unoccupied ports starting from 10000 (or$EZMSG_SERVER_PORT_START
) -- but avoiding default ports for the services.
- Command Line Changes
ezmsg start
now reports forked PIDezmsg graphviz
writes a DOT-formatted graph specification to stdout; can be piped tographviz
to visualize the current running graph.
- Fixed bugs
- Moved
__version__
attribute toezmsg.version
fixpip install
- "Single-process-mode" isn't special anymore; main process is always used for execution of one group of units
ezmsg
will always start a new threaded eventloop for tasks; because of@ez.main
code- Better
KeyboardInterrupt
handling (especially on Windows) Unit.Initialize()
and creation ofState
is now called from the task thread (andasyncio
synchronization primitives will automatically be associated with the correct event loop- Significantly cleaned up code pertaining to connections to the
GraphServer
andSHMServer
; reduced redundant code between these services significantly as well. - No-longer forcing
WindowsSelectorEventLoop
on Windows forasyncio
. - All leaked
ResourceWarnings
resolved; allasyncio
operations now threadsafe-ly implemented. - Resolved
ConnectionResetError
andBrokenPipeError
on shutdown. Settings
now decorated with@dataclass_transform
and type-checking should resolve settings fields appropriately.- This required introducing
ezmsg
's FIRST non-stdlib dependency;typing_extensions
which is required to backport this typing PEP to Python 3.8.typing_extensions
has no external deps and is pure-python; unlikely to break in the future.
- This required introducing
Settings
classes no-longer automatically generateunsafe_hash
functions. This behavior can be reintroduced by creating your owndataclass
definition withunsafe_hash=True, frozen=True
and inheriting fromSettings
.- Tasks/Publishers/Subscribers/Threads, etc.. now have attributes that are less likely to collide with user-code
- Deprecated
ezmsg.testing
- Moved
Extensions
ezmsg.sigproc v1.2.0
- All signal processing modules rewritten to operate on
AxisArrays
- All units allow you to specify named axis to operate on (usually time axis)
- New spectrum (FFT) unit in
ezmsg.sigproc.spectral
- Window now allows
newaxis
accumulation for batching TimeseriesMessage
deletedTSMessage
is now a function that returns a properly formattedAxisArray
ezmsg.websocket v1.1.1
__version__
attribute- Moved units from
ezmsg.websocket
toezmsg.websocket.units
- Added a
pyproject.toml
ezmsg.zmq v1.1.1
__version__
attribute- Moved units from
ezmsg.zmq.sender
andezmsg.zmq.poller
toezmsg.zmq.units
- Added a
pyproject.toml
3.2.3: Refactored Extensions
This release contains slight refactors to the core ezmsg
extensions in addition to bugfixes for a bunch of annoying behavior secondary to attaching/detaching from running graphs.
Extension updates:
ezmsg-sigproc
: V1.1.1: Fixes an issue whereezmsg.sigproc.sampler.Sampler
was subscribing to anOutputStream
. Also now has a__version__
attribute.ezmsg-zmq
: V1.1.0: Slight refactor to permit inclusion of a__version__
attribute.ZMQPollerUnit
andZMQSender
now exist inezmsg.zmq.poller
andezmsg.zmq.sender
respectively.
*-- There was a packaging error, this package will be updated in a future releaseezmsg-websocket
: V1.1.0: Fixes typing information forwebsockets
library, introduces a__version__
attribute. Also refactorsWebsocketClient
andWebsocketServer
to exist inezmsg.websocket.client
andezmsg.websocket.server
respectively.WebsocketSettings
is still located inezmsg.websocket
Bugfixes:
WARNING
-level log messages for "high water mark" on individual pub-sub channels due to subscriber backpressure- Fix for
BufferError
secondary to re-publishing a message attained withzero-copy=True
. The backend will now check if a unit attempts to re-publish azero-copy
'd message and will instead broadcast adeepcopy
of the message instead. - Fix for spurious dictionary size change on iteration when detaching from a running system
Quality-of-life:
ezmsg
will now raise exceptions on subscribing toOutputStreams
and publishing toInputStreams
within aUnit
.
3.2.2: Windows fixes
This bugfix release addresses some issues with using ezmsg on Windows. Specifically, Ctrl+C interrupting of multi-process systems was non-functional, and this has now been addressed. There's also been some changes to the servers and test backend that reduce the time necessary for GitHub CI to pass tests on Windows.
Accelerating tests on Windows required a small feature addition to the ezmsg command line interface. There is now an ezmsg start
command that will fork the core ezmsg servers and return once they're fully up/initialized. The core ezmsg servers (GraphServer and SHMServer) take a pretty long time to start up on windows, so we now call ezmsg start
to set them up before testing, then ezmsg shutdown
once testing has concluded.
3.2.1: Attach Hotfix
Version 3.2.0
required a sync()
operation when attaching to a running graph, which is not actually required. This sync was causing attach failures, so it's been removed. Additionally, this version has a fix for tests/test_attach.py
which had a race condition and would sporadically fail/hang forever. There are also improvements to debug output (in the ezmsg
logger at the DEBUG
log level).
3.2.0: Single Process Mode
This release enables several new features:
- Single process mode: If all tasks would run in one process, and a
GraphServer
/SHMServer
combo are already up,ez.run
will not attempt multiprocessing. Critical for usingezmsg
in REPL/notebook environments. - Associate
Publisher
andSubscriber
objects withOutputStream
andInputStream
instead of individual tasks:- Should drastically reduce the number of connections/sockets required for systems, enhancing performance
- Allows for control over individual
OutputStream
...- hostname/port specification; allowing outbound TCP communications to other PCs
- number of buffers; down to 1 buffer for direct control of channel latency and backpressure
- initial SHM allocation size; to avoid resizing SHM while running
- force TCP message passing; for small message sizes, there's no performance loss and this allows us to run perf benchmarks including TCP transmission without multi machine CI setups.
- IMPORTANT BREAKING CHANGE -- it is no-longer possible to subscribe to
OutputStream
s and publish toInputStream
s. This was previously possible (unintended) and should be fixed in downstream projects. If you can come up with a valid use-case, please let me know.
- Command line tool that allows for serving/shutting down
GraphServer
andSHMServer
from command line. After installingezmsg
, simply runezmsg serve
from the command line to start up theezmsg
background servers
This pull request also:
- Fixes several small bugs
- Switches
SHMServer
andGraphServer
ports (25978/9). GraphServer now sits at 25978 by default, and SHMServer now sits at 25979 by default. SHMServer
,GraphServer
and the publisher port-range start are now individually configurable by environment variables.GraphServer
'spause
andresume
are no-longer asynchronous; don't need to beawait
'ed- Shuts down immediately after
NormalTermination
is raised, rather than waiting up to 0.5 seconds- Tests run ever-so-slightly-faster because of this