[docs] changelog for v0.5.0a0

[docs] further description of input
[docs] transition status for data module
[docs] splitting out gui.plots
[dependencies] make pynwb optional
[packaging] refresh lockfile.

autopilot/__init__.py

@@ -1,5 +1,4 @@
 __author__  = 'Jonny Saunders <j@nny.fyi>'
-__version__ = '0.4.4'
 
 import sys
 if sys.version_info < (3,8):

autopilot/gui/widgets/input.py

@@ -1,3 +1,26 @@
+"""
+Widgets for representing input widgets for different types.
+
+The metaclass {class}`.Input` represents the basic structure, which each of the inheriting classes
+override. Subclasses can be retrieved and instantiated with the overloaded :meth:`.from_type` method::
+
+    # retrieve the class
+    int_input_class = Input.from_type(int)
+    # instantiate the class -- some subtypes (see :class:`.LiteralInput` ) need arguments on instantiation.
+    int_input = int_input_class()
+
+and can then be used to make and manipulate Qt Widgets::
+
+    # make the literal ``QWidget`` to be used
+    widget = int_input.make()
+    # get/set the value of the created widget
+    value = int_input.value()
+    int_input.setValue(value)
+
+This allows the :class:`ModelWidget` class to provide a uniform interface to fill end edit models.
+
+"""
+
 import typing
 from abc import ABC, abstractmethod
 from ast import literal_eval
@@ -75,6 +98,18 @@ def from_type(cls, type_:Type[list]) -> Type['ListInput']: ...
 
     @classmethod
     def from_type(cls, type_:type) -> Type['Input']:
+        """
+        Get a subclass of ``Input`` that represents a given type.
+
+        Args:
+            type_ (:class:`typing.Type`): The type (eg. ``float``, ``int``) to be represented
+
+        Returns:
+            An appropriate ``Input`` subclass
+
+        Raises:
+            :class:`ValueError` if either no or more than 1 subclass has been declared for the given type.
+        """
         subclass = [c for c in cls.__subclasses__() if c.python_type is type_]
         if len(subclass) == 0:
             raise ValueError(f"No Input widget has been defined for type_ {type_}")
@@ -87,11 +122,20 @@ def from_type(cls, type_:type) -> Type['Input']:
     def setValue(self, value:typing.Any):
         """
         Set a value in the created widget
+
+        After the :meth:`Input.make` method is called, returning a widget, the Input instance will
+        store a reference to it. This method then allows its value to be set, doing appropriate
+        type conversions and invoking the correct methods.
         """
 
     @abstractmethod
     def value(self) -> typing.Any:
-        """Retreive the value from the widget!"""
+        """
+        Retrieve the value from the widget!
+
+        After the :meth:`Input.make` method is called, returning a widget, the Input instance will
+        store a reference to it. This method then returns the value, doing appropriate type conversion.
+        """
 
     def make(self,
              widget_kwargs:Optional[dict]=None,
@@ -100,7 +144,7 @@ def make(self,
         Make the appropriate widget for this input.
 
         Stores the made widget in the private :attr:`._widget` attr, which is then used
-        in subsequent :meth:`.value` and :meth:`.setValue` calls.
+        in subsequent :meth:`.Input.value` and :meth:`.Input.setValue` calls.
 
         Args:
             widget_kwargs (dict): Optional: kwargs given to the widget on instantiation

autopilot/gui/widgets/model.py

@@ -248,7 +248,7 @@ def validate(self, kwargs:Optional[dict] = None, dialog:bool=False) -> Union[Lis
 
 class ListModelWidget(QtWidgets.QWidget):
     """
-    Container class to make lists of :class:`.ModelWidget`s for when a field is a ``List``
+    Container class to make lists of :class:`.ModelWidget` s for when a field is a ``List``
     """
 
     def __init__(self, model:Union[BaseModel, Type[BaseModel]],

docs/changelog/v0.5.0.md

@@ -1,14 +1,169 @@
-# v0.5.0a - ~The Data Modeling Edition~
-(changelog_v050)=
+# v0.5.0a0 - ~The Data Modeling Edition~
+(changelog_v050a)=
 
 A lot. Namely a whole rewriting of the {mod}`autopilot.data` module.
 
-Jonny is working on finishing their dissertation and is really running up against time for writing docs
-and so needs to leave the changelog blank for now.
+This is being released as an alpha version while we finish working out a few of the kinks in the data modeling 
+system because Jonny is finishing their dissertation and has a deadline determined by the viscera of 
+institutional science rather than software development!
 
-One thing to be aware of is that the subject class will attempt to update any of your existing
-.h5 files to the new format -- it should be nondestructive (making a backup of the whole file first
-and then making backups up any tables that it can't create a new version of that fully preserves
-the data from the previous version), but don't be alarmed when your subject files start updating!
-It might be a good idea to make a full backup of your data directory before making this update
-(but that's something we recommend anyway).
+## Upgrading From <v0.5.0
+
+- The subject class will attempt to update any of your existing
+  .h5 files to the new format -- it should be nondestructive (making a backup of the whole file first
+  and then making backups up any tables that it can't create a new version of that fully preserves
+  the data from the previous version), but don't be alarmed when your subject files start updating!
+  We advise, as we always do, making a full backup of your data directory before making this update.
+- Change references in `launch_autopilot.sh` scripts from `autopilot.core.terminal` or `autopilot.core.pilot` 
+  to `autopilot.agents.terminal` or `autopilot.agents.pilot`, respectively
+- Update any `TrialData` declarations in plugin tasks to use the new {class}`.Trial_Data` model. See {attr}`.Nafc.TrialData` for an example.
+  The old pytables IsDescriptor style descriptions will still  work, but you will be given a warning every time.
+- We advise updating pigpio to at least [`3c23715`](https://github.com/sneakers-the-rat/pigpio/commit/3c237159e5995ec58cd673579bdd66a8d819b269) to
+  avoid warnings and make use of some new functionality.
+
+## Package Structure
+
+- The `autopilot.core` module was removed entirely, and was split into
+  - {mod}`autopilot.agent`: which contains the {class}`.Terminal` and {class}`.Pilot` classes and a stub metaclass
+  - {mod}`autopilot.utils.loggers`: formerly `autopilot.core.loggers`
+  - {mod}`autopilot.gui` (described in module documentation) now has all the GUI modules, split out by type rather than all in one file.
+  - {mod}`autopilot.data` was created from `autopilot.core.subject`, see below.
+- {mod}`autopilot.root` contains root classes: 
+  - {class}`.Autopilot_Type` - types and data models for handling and saving live data
+  - {class}`.Autopilot_Object` - General root metaclass, beneath which will be top-level metaclasses for each of the major
+    class hierarchies that are currently separate.
+  - {class}`.Autopilot_Pref` - Stub for later converting prefs and environmental configuration parameters from their scattered state.
+- {mod}`autopilot.exceptions` will be used for more explicit exception handling.
+- Code files were given `755` permissions to allow execution (as opposed to `644`)
+- The unused git submodules for pigpio and an old temperature sensor were removed.
+
+## Major updates
+
+- {mod}`autopilot.data` is the major change in this release! See the module-level doc page for more details
+  - `autopilot.core.subject` was moved to {mod}`autopilot.data.subject` and remains the main interface to autopilot data.
+    It was effectively rewritten to accomodate the use of data models, and many of its public methods were made private to
+    make its use more obvious.
+    - The subject structure that determines which things are located where has been made into its own class {class}`.Subject_Structure`
+    - Making new subjects is now done with the {meth}`.Subject.new` class method rather than from the `__init__` method
+    - The `open_hdf` method has been replaced with the `_h5f` private method that is a more explicit context manager allowing
+      locking read/write access and nonlocking read access
+    - The subject file now explicitly handles updates between different versions of the Subject class. 
+    - Subject will check if the protocol `.json` file has been updated from when it was assigned and automatically update to the new version,
+      logging the change.
+    - All trial data is now saved with a `session_uuid` unique identifier column that is an always-unique key in case of any
+      overlapping session IDs from reassignments, etc.
+    - Many attributes were replaced by properties that return data models:
+      - {attr}`.Subject.bio` and {attr}`.Subject.info` ->  {class}`.Biography`
+      - {attr}`.Subject.protocol` -> {class}`.Protocol_Status`, which manages the current trial, session, step, etc. rather than being treated separately
+  - {mod}`autopilot.data.interfaces` contains classes to relate abstract data models to different representations and formats
+    - {mod}`.interfaces.datajoint` allows creating datajoint schema definitions from autopilot models using the
+      companion [datajoint-babel](https://github.com/auto-pi-lot/datajoint-babel) project
+    - {mod}`.interfaces.tables` translates pydantic models to HDF5 files
+  - {mod}`autopilot.data.modeling` has basic types for use in data models
+  - {mod}`autopilot.data.models` has the models themselves
+  - {mod}`autopilot.data.units` stub module for using explicit units instead of anonymous floats
+- A new set of general model filling widgets - {mod}`.widgets.input` and {mod}`.widgets.model` - were created that will
+  eventually replace much of the existing GUI which suffers from code duplication problems from minor variations between
+  representations of parameters, etc.
+- {class}`~.loggers.Log`, {class}`.Log_Format`, {class}`.LogEntry`, {class}`.ParseError`, 
+  {data}`.MESSAGE_FORMATS`, {data}`.LOG_FORMATS` were added to allow programmatic loading of logfiles, and {mod}`.utils.log_parsers` 
+  was added to allow recovery of structured data (eg. from logged trial data). Loggers now use the [rich logging handler](https://rich.readthedocs.io/en/stable/logging.html)
+  for much more readable logs in stderr.
+- {class}`.Jack_Sound` classes now put all frames at once into the processing queue, rather than one at a time for much less
+  variability in sound onset jitter. The {class}`.JackClient` class was updated to reflect that as well by adding a `play_q_size`
+  argument that controls the size of the deque used to store frames to play (rather than pulling each from a Queue as was done before). {class}`.JackClient`
+  also has an optional `disable_gc` (False by default) parameter that allows the garbage collector to be disabled
+  in the process for further (unquantified) jitter reduction.
+
+## Minor updates
+
+- The bandwidth test ({class}`.Bandwidth_Test` widget and {meth}`.Pilot.l_bandwidth` listen) was updated to allow selecting
+  compression, use of a random array, or preserialized messages
+- The {mod}`.Terminal` now closes cleanly (still with a warning of leaked semaphors) without hanging.
+- External processes {func}`.external.start_pigpiod` and {func}`.external.start_jackd` first check if the process is already started
+- {meth}`.Digital_Out.set` now has a ``result`` argument for faster setting that doesn't confirm the result when ``False``
+- {class}`.Message` s now provide clearer support for compression in serialization, and automatically detect and handle
+  a compressed message when deserializing. This is also carried through to the {meth}`.Net_Node.send` method.
+- {meth}`.Station.l_kill` was moved up to the top level {class}`.Station` class, rather than being something the Terminal station owns.
+- The {mod}`~.setup.run_script` functions no longer uses the totally unreadable white on green ANSI code combo that I have no idea why i ever used in the first place.
+- A `picamera_legacy` script was added for enabling the picamera on bullseye.
+- The {class}`.sounds.Gammatone` sound now accepts a `filter_kwargs` dictionary that is passed on to the {class}`.timeseries.Gammatone` filter.
+- The {class}`.Task` and {class}`.Nafc` classes use the new {class}`.Trial_Data` style data declarations.
+- Two utility functions {func}`.common.walk_dicts` and {func}`.common.flatten_dicts` were added to make handling nested dictionaries a bit easier.
+- 
+
+## Bugfixes
+
+- The Subject class would incorrectly overwrite data given a mismatch in system times between the Terminal and Pilot.
+  The subject class should now ideally no longer overwrite anything ever.
+- The subject class would drop trial data silently if it was not in the TrialData description. En route to making the table
+  automatically expand to accomodate unexpected data, dropped data is now logged as an exception.
+- The prefs manager handles being launched from within ipython and other processes better, but is still a bit buggy.
+  Now it uses the check used internally in the multiprocessing module to see if a manager can be launched,
+  and falls back to using a standard dictionary if not.
+- `jackd_source` script uses correct `https://` rather than `git://` protocol.
+- A `_TASK_LIST` was added to {mod}`.utils.registry` to support deprecated task specifications.
+
+## Regressions
+
+- With additional checking for monotonic increases in `trial_num` and checks that prevent data overwriting,
+  individual writes of trial data are now a bit slower, which should be optimized for when we complete the transition to
+  uniform data models throughout the library.
+
+## Prefs
+
+- `PIGPIOD` - `bool` - if `True`, start pigpiod on pilot start
+- Stubs were created for converting the prefs to using data models, but they have not been filled yet.
+- A `AUTOPILOT_NO_PREFS_MANAGER` environment variable now controls the use of a multiprocessing manager explicit. Documentation
+  for environmental variables is forthcoming.
+- A `AUTOPILOT_WARN_DEFAULTS` environment variable controls whether warnings should be printed for when a default pref value
+  is retrieved, because that warning is a good one but can be really annoying.
+
+## Packaging & Dependencies
+
+- Autopilot is now packaged with Poetry! This allows for fully deterministic installation with the poetry.lock file and
+  updates from the old setuptools style of dependency specification.
+- The source repository has moved from https://github.com/wehr-lab/autopilot to https://github.com/auto-pi-lot/autopilot
+- `MANIFEST.in` has been replaced by the `include` field in `pyproject.toml`
+- `autopilot.__version__` is now determined by `importlib.metadata` and specified in the `pyproject.toml` file rather than in the `__init__.py` file
+- `blosc` was replaced with `blosc2`
+- New dependencies
+  - global
+    - pydantic (^1.9.0)
+    - parse (^1.19.0)
+    - rich (^11.2.0)
+    - validators (^0.18.2)
+  - docs
+    - autodoc_pydantic (^1.7.0)
+    - myst_parser (^0.17.2)
+- Version Bumps
+  - pyzmq 18.1.* -> ^22.3.0
+  - tornado >=5.0.0 -> ^6.1.0
+  - numpy >=1.16.5 -> ^1.20.0
+  - scipy >=1.6.0 -> ^1.7.0
+  - pandas >=0.19.2 -> ^1.3.0 on python 3.7 and ^1.4.0 on >=3.8
+  - tables >=3.4.2 -> ^3.7.0
+  - Sphinx >=3.1.2 -> ^4.3.1
+  - A lot more dependencies were taken from being implicit versions to explicit
+    by the conversion to using Poetry...
+
+## Docs
+
+- [Configuration](configuration) was moved to its own page, documenting setting up the system as well as the contents of the user directory.
+- A [faq](FAQ) page was stubbed out (but is still pretty skeletal)
+- The overview was updated with some more information in the module tour
+- Virtual environment usage was moved from the setup page to its own subpage linked from the FAQ.
+- A `make serve` option was added to the docs makefile that makes use of [sphinx-autobuild](https://pypi.org/project/sphinx-autobuild/)
+  to livereload docs while editing them.
+- `autopilot_theme.css` was updated to be compatible with the new version of sphinx-rtd-theme that apparently changed the way
+  that TOC buttons were made, as well as remove incorrect references to fonts no longer packaged.
+- The `autodoc_pydantic` and `myst_parser` extensions were added -- and we will be moving towards using MyST rather than 
+  hellish ReST for future narrative docs!
+- Private methods and functions are now no  longer rendered in the main documentation, and the library will over time
+  use the public/private distinction more systematically to make it more understandable.
+- Examples was split off into its own folder and links to wiki plugins. Blink was moved with it
+
+## Tests
+
+- We have started importing some of the pigpio mocking tools from the [People's Ventilator Project](https://peoplesvent.org)
+  to start writing GPIO tests!

docs/data/index.md

@@ -95,8 +95,20 @@ See the main {mod}`.data.subject` module page for further information.
 This too is just a stub, but we will be moving more of our models to using specific SI units when appropriate rather
 than using generic `float`s and `int`s with human-readable descriptions of when they are a mL or a ms vs. second or Liter, etc.
 
-
-
+# Transition Status
+
+Transitioning to a uniform data modeling system is in progress! The following need to still be transitioned to formal models.
+
+- `Task.PARAMS` and `Task.HARDWARE`
+- `Task.PLOT` which should be merged into the TrialData field descriptions
+- {mod}`autopilot.prefs` - which currently has a large dictionary of default prefs
+- Hardware parameter descriptions - Need to find better way of having models that represent
+  class arguments.
+- {mod}`.graduation` objects.
+- Verious GUI widgets need to use models rather than the zillions of ad-hoc representations:
+  - {class}`.Protocol_Wizard`
+- {mod}`.utils.plugins` needs its own model to handle dependencies, etc.
+- {mod}`.agents` needs models for defining basic agent attributes.
 
 ```{eval-rst}
 .. automodule:: autopilot.data

docs/gui/index.rst

@@ -1,6 +1,17 @@
 GUI
 =====
 
+GUI Modules used by the :class:`.Terminal` agent.
+
+Each of the submodules contains GUI objects of different types, and will continue to be refined
+to allow for extensions by plugins.
+
+* :mod:`.gui.menus` - Widgets and dialogues available from the Terminal menubar
+* :mod:`.gui.plots` - Widgets and graphical primitives used for task plots
+* :mod:`.gui.widgets` - General purpose widgets that make up the Terminal GUI
+* :mod:`.gui.dialog` - Convenience function for popping standard modal/nonmodal dialogues
+* :mod:`.gui.styles` - Qt Stylesheets (css-like) for the Terminal
+
 .. automodule:: autopilot.gui
     :members:
     :undoc-members:
@@ -9,6 +20,6 @@ GUI
 .. toctree::
 
     menus
-    plots
+    plots/index
     widgets/index
     dialog

docs/gui/plots/geom.md

@@ -0,0 +1,10 @@
+# geom
+
+Geometric primitives for plots.
+
+```{eval-rst}
+.. automodule:: autopilot.gui.plots.geom
+    :members:
+    :undoc-members:
+    :show-inheritance:
+```

docs/gui/plots/index.md

@@ -0,0 +1,15 @@
+# Plots
+
+```{eval-rst}
+.. automodule:: autopilot.gui.plots
+    :members:
+    :undoc-members:
+    :show-inheritance:
+```
+
+```{toctree}
+plot
+geom
+info
+video
+```

docs/gui/plots/info.md

@@ -0,0 +1,10 @@
+# info
+
+Widgets for displaying non-plot information from within the terminal
+
+```{eval-rst}
+.. automodule:: autopilot.gui.plots.info
+    :members:
+    :undoc-members:
+    :show-inheritance:
+```

docs/gui/plots/plot.md

@@ -0,0 +1,10 @@
+# plot 
+
+Primary plot widgets that contain items from {mod}`.geom` 
+
+```{eval-rst}
+.. automodule:: autopilot.gui.plots.plot
+    :members:
+    :undoc-members:
+    :show-inheritance:
+```

docs/gui/plots/video.md

@@ -0,0 +1,10 @@
+# video
+
+Classes for displaying streamed video
+
+```{eval-rst}
+.. automodule:: autopilot.gui.plots.video
+    :members:
+    :undoc-members:
+    :show-inheritance:
+```