Borealis v0.7 #403
Closed
Borealis v0.7 #403
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Negative cpid for embargoing files - approved by Kathryn to use positive CPID going forward, only embargoing for test experiment data and for modes that are still needing further development or post-processing (such as full FoV)
* Made site specific tmp.md5 file for scheduler sync * Adding logging, updated schedules directory * Made email file list optional to local_scd_server * Logging changes in remote_server.py - Removed plot creation - Now printing contents of email to console - Added extra log statements * Modified start_ and stop_radar.sh Added more descriptive and better checks for error logging within start_radar.sh and stop_radar.sh * Fixed some bugs in start/stop scripts * Small bugfix to scripts/ * Touched up scheduler_sync.daemon following code review * Update from code review suggestions * Fixed small bug checking variable definitions
* Changes to address IndexError when writing zero-sequence records. * Custom __repr__ for SliceData that should display all variables set * Added a check to write_hdf5_file() that should circumvent the IndexError * Added log warning when empty array encountered.
* Attempting to autofill experiment portions of the docs that are in code docstrings. * interface_types tuple docstring was basically verbatim repeated in building_an_experiment.rst * Fixed incorrect import paths in a code snippet * ExperimentPrototype init params are also repeated in docs, but worded more generally for new users. I've included added the docstrings to auto-load but it may be better to remove them. * Updating based on docs build of previous commit. * Use autoattribute to import interface_types docstring (avoiding module docstring) * Removed inclusion of ExperimentPrototype parameters via autoproperty * Testing out ":class-signature:" to remove function call on ExperimentSlice autodoc. * Removed some out of date lines from ExperimentPrototype docstring * * Testing updated dependencies to see if new options take effect. * Bumped sphinx minimum version to 5.1 * Removed sphinx-rtd-theme minimum version * Docs broke, testing sphinx==7.2.6 requirement. * Updating docutils reuquirement to match sphinx 7.2.6 requirement * Fixing build warnings/errors. * Index collision for interface_types * Unsupported ":class-signature:" directive for autoclass * Indentation error in remote_server.py module docstring. * Bad path in ExperimentSlice autodoc path. * Reverting requirements back to original to fix HTML rendering. * Fixed requirements specifications. * Newer versions of sphinx supported, docutils requirements bumped slightly. * Fixed noindex specification in autodoc uses. * Fixed deprecation warning on intersphinx_mapping in conf.py * Docs changes per code review. * Fixed list formatting in interfacing_types docstring. * Removed list of unique slice keys, which is not complete and redundant with ExperimentSlice fields later on page.
* Bugfix in scheduler, not getting correct past lines to schedule current experiment. * `get_relevant_lines()` of `remote_server.py` was conducting a minute-by-minute search, rather than stepping through the entries of the schedule file. This function has been removed. * `get_relevant_lines()` of `scd_utils.py` was updated to grab all future and current lines, as well as any past lines that run past the current time, and the most recent infinite-duration line from the past. * Added in a check to ensure that infinite-duration lines have priority zero (otherwise, they would supersede any lower-priority experiments scheduled after them, essentially removing those lower priority levels). * Moved all static parts of `SCDUtils` into class methods/variables. Updated docstrings accordingly. * `quick_test_scdutils.py` and `tests/scheduler/test_scheduler.py` now run successfully. Needed updates for `kwargs` and `embargo` fields. Co-authored-by: Theodore Kolkman <90067549+tjk584@users.noreply.github.com>
* Re-formatted use of tx and rx center frequency to be slice defined. * trying to get ctrfreqs passed properly * Adjusted validators, and added same center freq requirement on CONCURRENT slices * Re-ordered slice parameter definitions for validator functionality * moved tx and rx max/min freq defs to slice validators * Properly referenced center freqs in scan classes and sequence class * Enabled global N200 center freq defining through experiment init for legacy experiment compatibility * Revert last commit * Revert last commit * I do not understand how git commit reversion works. Did it manually. * Properly formated validators for potential failed check up the chain * Adjusted error statements * PR corrections * Managing if tx/rxctrfreq fails, and removing ref to max_freqs * Rem fixed bug in sequence interfacting * Set new tuning delay * Renamed scan class and ScanClassBase to interface class and InterfaceClassBase * Enforced common center freq for CONCURRENT and SEQUENCE interfacing * updated docs and fixed comment * Fix runtime issues and doc compiling issues * Added check to prevent freq from being too close to center freqs * Update src/experiment_prototype/experiment_slice.py Co-authored-by: Theodore Kolkman <90067549+tjk584@users.noreply.github.com> * Theo's comments * Comment change to determining transition bandwidth * Adjusted docs for documentation updates --------- Co-authored-by: Theodore Kolkman <90067549+tjk584@users.noreply.github.com> Implements retuning of rx and tx center frequencies. Refactors txctrfreq and rxctrfreq to be slice defined parameters
* Refactored DSP class as step towards making DSP test notebook. * Made all methods static, and moved shared memory allocation to `__init__` instead of instance methods. * Added return types to docstrings for methods * Removed `super.__init__(DSP, self)` from `__init__` since DSP is only child of `object`. * Moved some DecimationScheme checks to ExperimentSlice. * This avoids ExperimentSlice and Options imports from having to be in decimation_scheme.py * Changed ExperimentExceptions in decimation_scheme.py to ValueErrors. * Facilitates testing of decimation_scheme.py functions. * Added floating-point comparison for output rate of DecimationScheme object. * Created notebook for testing DSP class and filtering schemes. * Refactored rx_signal_processing.py to modularize DSP for simple testing in the notebook. * Created small numpy array of rawrf data that imports into notebook for realistic data example. * Notebook walks through DecimationScheme object, uses them in the DSP object, benchmarks performance of different DecimationSchemes, and shows the spectrum of transmitted pulses. * Modified docs and removed old testing scripts. * Added section about filter testing notebook to the tools.rst page in the docs * Found versions of docs package requirements which render bulleted lists and the docs itself correctly, pinned in requirements.txt * Replaced correlation matrix image in dsp_chain_in_detail.rst with newer image * Removed old scripts from tests/dsp_testing/ which have not been in use and are replaced by filters.ipynb * Changes per code review. * Moved DSP class to new file src/utils/signals.py * Moved sample_building.py functions into signals.py * Made windowed_view() function a staticmethod of DSP class * Updated all imports accordingly in sequences.py, rx_signal_processing.py, and filters.ipynb. * Update docs/source/dsp_chain_in_detail.rst Co-authored-by: Draven Galeschuk <doreban@gmail.com> * Changes per code review. * Suppress RuntimeWarning in notebook * Made figure size configurable in notebook * Add titles to some plots in notebook * Added figure which was meant for display in docs * Updated commit for borealis_experiments. --------- Co-authored-by: Draven Galeschuk <doreban@gmail.com>
* Hotfix for calculation error from #441 * Forgot to correct phase of output samples after bandpass filter. * Lock docutils version in requirements.txt (docs not rendered correctly on ReadTheDocs) * Lock docutils and sphinx-rtd-theme versions for docs.
* Fix docs and sort interfaced slices. * Docs formatted slightly incorrectly due to InterfaceClassBase refactor. * Sort in slice_combos_sorter by first slice in each combo. * Added clarifying comment
* Major update to logging functionality. * Created a script json_to_rich_logs.py for converting JSON-rendered log files to the console-rendered style. * Made log levels configurable by renderer (can have different levels for console, logfile, and aggregator) * Added new log level VERBOSE, between DEBUG and INFO levels. The intent is to minimize the console output by switching most log entries to VERBOSE, while keeping VERBOSE logs in the logfiles and sent to the aggregator. * Updated config file format so each log handler has its own enable and log level field, plus grouped some other existing fields with their respective log handler. * Attempting to fix bugs in adding VERBOSE log option. * Fixed log option name (was TRACE, now VERBOSE) * Added docstrings to ConfigurableLevel processor class. * Fixed bug with experiment_handler screen crashing due to bad log config args. * Modifying some logs to VERBOSE level * INFO level now is much sparser, both in information and frequency. * VERBOSE level used for most logs that were previously INFO. * Fixed bugs with defining new VERBOSE log level. * Had to hack in the attribute to structlog.stdlib, very dirty fix * Moved addition of VERBOSE log level to within log() function to avoid multiple calls from experiment_handler which calls log() twice. * Fixed more bugs with defining new VERBOSE log level. * Force upper-case name for log level and lower-case name for log function when setting attrs in logging and structlog modules * Ignore utils.log_config in frame for CallsiteParameter, to avoid `log_config fn` being attributed as the function that calls verbose logs * Use `info` style when rendering `verbose` logs to the console * Minor modification to data_write.py log of writing a record * Alphabetized imports in experiment_handler.py * Removed `log = log_config.log()` at module level in experiment_handler.py, replaced with call to getLogger() for use when running experiment_unittests.py * Configured logger for experiment_unittests.py, with all renderers set to False. * Reverted change in borealis_experiments commit, now points to develop. * Minor change to ensure logfile defined if logfile not None. * Major simplification of multi-level logging, and bugfix for multiple handlers. * Set the level for each handler using logging.Handler.setLevel(), removing custom ConfigurableLevel class. * Removed unnecessary setattrs() in add_logging_level(), replaced with logging.addLevelName() instead * Removed addition of log level number in processor chain * Fixed bug with using structlog 24.1.0. * Added structlog<24.1.0, graypy, and rich to install_radar_deps.py pip list. * Created VerboseLogger class that extends BoundLoggerBase. This is used as a wrapper class. It has a new `verbose` method that can be used like `log.verbose()`, and circumvents the previous hack which modified the `structlog.stdlib._NAME_TO_LEVEL` attribute which was renamed `NAME_TO_LEVEL` between 23.1.0 and 24.1.0. * Modified the json_to_rich_logs.py script to output to stdout. This can be piped to file if the user desires. Updated the handler in log_config.log() to a StreamHandler. * Reorganized the Borealis screens. * This is visually easier to look up, and gives each screen more room before line-wrapping occurs. * Added verbose level to json_to_rich_logs.py. * Also added verbose style for ConsoleRenderer when using json_to_console feature of logging. * Minor updates to what is logged. * Reordered screens in `steamed_hams.py` * Fixed tabbing in verbose logs of `radar_control.py` * Changed parsing log of `data_write.py` to info, and added slice_ids as a field * Include total processing time of sequence and slice_ids to info log in `rx_signal_processing.py` * Adding slice_ids to radar_control.py num_sequences log. * Disabled aggregator in config files, and froze structlog version for docs. * Update src/utils/log_config.py Co-authored-by: Theodore Kolkman <90067549+tjk584@users.noreply.github.com> * Minor changes per code review. * Update log_config.py * Updated docstring for `log()` with new `VERBOSE` mode for log levels. --------- Co-authored-by: Theodore Kolkman <90067549+tjk584@users.noreply.github.com>
Moved emailer class creation within while loop so a new one is made every loop.
* Fixed logic regarding no intf antennas. * Previously, rx_signal_processing checked options.intf_antenna_count to determine whether intf antennas present. * Now, rx_int_antennas field from slices passed along in order to determine whether intf antennas present. * Fixed a bug with intf_acfs and xcfs not desired. * Raising an exception during calculating expectation value for acfs, circumvented by checking if intf_acfs/xcfs available first before calling this. * Fixed some validators in experiment_slice.py. * Now, it is possible for the user to specify an empty list for tx_antennas, rx_main_antennas, rx_int_antennas. * None values will default to full antenna list, empty lists will be checked that they place nice with other fields. * Fixed a bug with concurrent slices that have different rx_antennas. * send_dsp_metadata wasn't sending beam information for _all_ antennas, only the rx_main_antennas and rx_int_antennas of the slice. * Fixed a bug with beam phases in radar_control. **NOTE** ALWAYS USE EXPLICIT DTYPE IN NUMPY ARRAYS * Trim floats in console logs to 3 decimal places * Config file refactor for accurate N200 channel to antenna mapping. * Each n200 entry in the config file now has four fields: - addr: the IP address (unchanged) - rx_channel_0 - rx_channel_1 - tx_channel_0 * The last three fields all expect a value like "m0" or "i2", where the "m" and "i" indicate main and interferometer array, respectively, while the following number indicates the antenna number in that array. The channels correspond to the physical connections of the N200 device. * The list of N200 configs is parsed to determine the rx_main, rx_intf, and tx_main antennas for the radar * The C++ options code was updated to handle this new format * Careful refactoring has been done to ensure that the tx and rx phases are properly handled in sequences.py given the antennas in the config file and the antennas specified by a given slice * `rx_int_antennas` was renamed `rx_intf_antennas` in experiment_slice.py * Antenna validators were updated to check that antennas specified for a slice have a corresponding N200 connection in the config file * main_antenna_count_ and intf_antenna_count_ were removed as variables from C++ DriverOptions class, as well as their getters * Logic for zeroing beam phases for unused antennas in a slice was moved from radar_control.py to sequences.py, so it is done once when the experiment is instantiated instead of on-the-fly. This also moves it to a more logical place, right where the beam phases are calculated. * Bugfix for subdev spec in C++ options. * Limit the acceptable subdev specs for rx, since channel number configuration based on an assumed value. * Fixing some bugs (compiling on lab computer) * Forgot semicolons. Sigh * Bugfixes from testing on lab computer. * Yet more bugfixes from testing on the lab. * Fixed bug that arose from merge conflicts. * Simplification of driveroptions.cpp based on code review. * Updated config_options.rst to reflect new format. * Added script for testing config files. * Can either test a specific config file or list of config files, or unit tests the Options class parsing of a config file. * Created a test config file base_config.ini that is read in by the unit tests. * Removed redundant tests from options.py * Updated the docs to include the new config testing script, and fix the sample_building autodoc. * Bugfix in driveroptions.cpp and updating docs. * Added tests for experiment slice and config file interplay. * Test that exceptions are raised if an experiment is specified in a slice but not in the config file. * Fixed base_config.ini after merging in develop. * Potential fix for ReadTheDocs failure. * Potential fix for ReadTheDocs failure take 2. * Bugfix with testing config files. * Right now config_testing.py does not care about the restrict.dat or hdw.dat fields, so the script uses the SAS files to populate those fields of the MockOptions class. This is required to ensure the tests that use ExperimentSlice will have all the required fields. * Fixed bugs with log.exception and concurrent multi-beam slices. * `log.exception()` was not defined for VerboseLogger * `CONCURRENT` slices can have different numbers of simultaneous beams. This was not handled correctly, and would raise an Exception. This commit fixes it, and ensures that all beams are correctly generated. * Laid foundation for including main and intf antenna phases in data files in future release. * Only pass phases of used antennas to data_write from radar_control * This will ensure that the phases for a slice `rx_main_phases` and `rx_intf_phases` match up with the antennas specified in `antenna_arrays_order`. * Fixed bug with extracting correct antenna phases for slice. * Sort antennas for slice in ExperimentSlice validators. * Changes per code review.
* Changes to only test relevant experiments. * Experiment unit tests only run on experiments which are relevant (possibly will be run in future) or when `add_line()` is called. * Refactored the scheduler code. * Removed remote_server_options.py and email_utils.py * Raise error in local_scd_server.py if schedule line can't be scheduled * Added site_id as required param for initializing `scd_utils.SCDUtils` object * renamed `SCDUtils.check_line()` to` SCDUtils.create_line()`, and moved all checks to `SCDUtils.test_line()` * Changed experiment_unittests.py `build_experiment_tests()` function to add a failing test if a specified experiment does not exist * Significant changes to test_scheduler.py to handle the above refactors. * Removed mention of scheduler emails in docs and fixed small local_scd_server.py bug. * Fixed logging bug in experiment_unittests.py * Changes per code review.
…r. (#468) * Moved "Driver Ready" message to radar_control into transmit thread from main thread
* Blocked out unused variables to be removed and fizxed unmatched types * Removed unused variables * Fixed ZMQ warnings and header file name changes * Fixed accidental deletion, just removed storage variable
* First attempt at option to seamlessly run without realtime module. * Minor bugfix for specifying realtime screen * Fixed bug in steamed_hams.py and attempting to intelligently drop realtime messages. * Drop any packets for realtime if host unreachable. * Documented how to run the radar without realtime * Update docs/source/realtime.rst Co-authored-by: Theodore Kolkman <90067549+tjk584@users.noreply.github.com> * Update docs/source/starting_the_radar.rst Co-authored-by: Theodore Kolkman <90067549+tjk584@users.noreply.github.com> * Changes per code review. * Simplified logging for brian failures and refactored realtime. * Removed the queue, since shared state is a bad idea for threads. Instead, use zmq sockets to send fitted record between threads for service over web socket. * Create all sockets within the threads (best practice) * Refactor of realtime. * Break out conversion and fitting of records into a function * Renamed functions that run forever to better reflect their behaviour * Pass the zmq context and options into each thread * Thread functions defined outside main function * Small bugfix in logging failed frames in brian.py * Minor bugfixes from testing in lab. * Forgot to pass `--realtime-off` to brian from steamed_hams * Check the number of failed frames before logging in brian * Another major refactor of realtime * Organized imports * Renamed `main()` to `realtime_server()` * Removed threads, put all functionality into `realtime_server()` * Added interrupt handling as per https://zguide.zeromq.org/docs/chapter2/#Handling-Interrupt-Signals * Created simulator for realtime, `realtime_sim.py`. * Reads in a single record and passes it through `realtime.realtime_server()` to ensure that it is processed and served. * `rawacf_record.pkl` file contains the record * Added additional ignore for log_config's `CallsiteParameterAdder` * Refactored realtime to handle being imported, pass in sockets to `realtime_server()` function so it can be tested externally. * Fixed relative imports to work when realtime is imported. * Also fixed bad path in realtime_sim.py that came from changing the directory structure * Use absolute path for reading in pickled rawacf data * Log an error if realtime tries to bind to non-existent network device * Simplified error handling for realtime.py and realtime_sim.py. * `realtime_server()` now correctly handles closed context and interruptions, closing up its sockets and returning * `realtime` server socket created with no linger * If `realtime.py` run as main, it will correctly terminate when an interrupt or error is encountered * Switched the fitacf socket in `realtime_sim.py` to match what `realtime` uses (PUB-SUB) * Put the data service in a loop, to simulate multiple data messages being sent. Verified that no messages are lost. * Added information about realtime_sim script to the docs. Moved some tests from tools.rst to lab_testing.rst. * Update docs/source/lab_testing.rst Co-authored-by: Theodore Kolkman <90067549+tjk584@users.noreply.github.com> --------- Co-authored-by: Theodore Kolkman <90067549+tjk584@users.noreply.github.com>
* Fixed sample buffer calculations. * Since DecimationScheme object is not fixed per slice, the number of extra samples required may be Slice dependent. This should be calculated each sequence for correctness. * Small bugfix with initializing lists. * Small bugfix for floating-point equality check. * Updated commit that borealis_experiments submodule points to. * Minor change to print other experiment in CPID collision. * Updated borealis_experiments commit. * Fixed bug with buffer size mismatch. * Due to `sequence_worker` thread using `dm_rates` and `dm_scheme_taps` variables from outer scope, which could be updated between processing main and intf data.
* Updated index.rst with changes in v0.7 * Updating docs and docstrings. * Minor change to monitoring.rst
* Added in pre-commit and black git integrations. * pre-commit uses some general file hooks, black hooks, and rst hooks * black configured for github workflows to run on push and pull request * Added more pre-commit hooks. * autoflake for removing unused imports in Python files * clang-format for formatting C++ code * cppcheck for static analysis of C++ code * check-toml and check-yaml for parsing those filetypes * requirements-txt-fixer for sorting requirements.txt files * check-hooks-apply and check-useless-excludes to verify that pre-commit hooks and excludes are applicable * Added Black badge to README.md * Applied clang-format hook to C++ code. * Ignore C++ clang-format refactor in git blame. * Added workflow to auto-link docs build to PRs. * Added docs page on contributing to Borealis. * Attempting to fix Black GitHub action. * Attempting to fix Black GitHub action - take 2. * Exclude directories for black github action * Exclude directories for black github action - take 2 * Fail docs build on warnings * Removed docs-link workflow that didn't work. * Amending some lines that were formatted by Black. * Minor reformat of experiment_handler.py * Fixed bug in writing bfiq files. * Fixed formatting of files from merge resolution. * Updated borealis_experiments commit after PR merged in that repo. * Removed Python specification in Black hook of pre-commit yaml. * formatting modifed by clang --------- Co-authored-by: Draven G <doreban@gmail.com>
Squash-and-merge does not preserve the commit SHA, so updated to merge commit SHA
* Updating installation scripts/dependencies. * Quick install fix * fixed pre-commit install and bad user error * removed CUDA installation and updated docs * Fixed hard coded python version in env setup * Small documentation changes * Print warning if trying to install cupy and cuda not installed. * Fixed check if cuda installed before installing cupy --------- Co-authored-by: Draven <doreban@gmail.com> Co-authored-by: Theodore Kolkman <tjk584@usask.ca>
Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 3.19.4 to 3.19.5. - [Release notes](https://github.com/protocolbuffers/protobuf/releases) - [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/protobuf_release.bzl) - [Commits](protocolbuffers/protobuf@v3.19.4...v3.19.5) --- updated-dependencies: - dependency-name: protobuf dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Release
Major Features
tx_antenna_phasesas a field in the data filesrxonlyfield in experiment slices REQUIRED for listening experimentsMinor features