A collection of tools for offline processing of data from a robotic black box (as implemented in https://github.com/ropod-project/black-box).
The toolbox primarily uses Jupyter notebooks for data plotting and analysis with the hope that this simplifies the sharing, presentation, and above all, processing of data.
- Data processing relies on the
black_box_utilspackage that is included here as a standalone Python library
The following notebooks are provided with this toolbox:
sw_data_plot: Plots data from the smart wheels of a platform
cmd_vel_plot: Plots velocity commands
Plots various smart wheel measurements, in particular:
- wheel currents
- wheel voltage measurements
- wheel velocities
- IMU acceleration measurements per wheel
- IMU gyroscope measurements per wheel
Plots planar base velocity commands, namely:
- linear velocity along x
- linear velocity along y
- angular velocity
A Python package implementing various utilities for working with black box data. There are three scripts that are part of this package:
DBUtils class with the following static methods:
restore_db: Restores a MongoDB database dumped in a given directory
get_all_docs: Returns all documents contained in a specified collection of a given database
restore_subdbs: Restores multiple dumps into a single database.
dump_db: Dumps a MongoDB database in the specified directory.
get_data_collection_names: Returns the names of all black box data collections in the specified database.
clear_db: Drop all collections in the given database.
drop_db: Drops the given database.
get_subdb_metadata: Returns a list of dictionaries containing metadata about any sub-databases stored in the given database.
get_subdb_docs: Returns a list of dictionaries in which the keys are sub-database names and the values are lists of documents in the given collection corresponding to the respective sub-databases.
get_docs: Returns all documents contained in the specific collection of the given database within given time duration.
get_docs_of_last_n_secs: Return documents from given collection name of last n seconds from given db name.
get_doc_cursor: Returns a cursor for all documents in the specified collection of the given database which have the given 'timestamp' value in the given range.
get_collection_metadata: Returns the entry of the 'black_box_metadata' collection for the specified collection.
get_oldest_doc: Returns the oldest document in the given collection name.
get_newest_doc: Returns the newest document in the given collection name.
get_last_doc_before: Returns the last document in the collection with the given name that is before given timestamp.
get_db_oldest_timestamp: Gets the oldest record in the mongo db and returns the corresponding timestamp.
get_db_newest_timestamp: Gets the newest record in the mongo db and returns the corresponding timestamp.
get_db_client: Returns a MongoDB client at :.
get_db_host_and_port: Returns a (host, port) tuple which is ("localhost", 27017) by default, but the values can be overridden by setting the environment variables "DB_HOST" and "DB_PORT" respectively.
DataUtils class with the following static methods:
get_all_measurements: Returns all measurements of a single variables (that potentially has multiple instances - e.g. one instance per wheel)
filter_data: Filters data using a given data filter and returns the filtered array.
find_correlated_variables: Returns a list of variable name pairs where each pair denotes that two measurements in a given measurement matrix are correlated (based on the Pearson correlation coefficient).
get_windowed_correlations: Returns a list of variable name pairs and a 2D list of windowed pairwise correlations, given a list of variable names and a data array.
get_var_value: Returns the value of a given variable in the given dictionary;
get_variable_list: Returns a list of the names of all variables stored in a given collection.
get_flattened_variable_names: Recursive method that returns a flattened list of the variable names in a given dictionary
expand_var_names: Generates a new list of variable names from a given list of such that the * character in each entry of the original list is replaced by a zero-based index
get_bb_query_msg_template: 'Returns a dictionary which represents a template for a black box query message.
get_bb_query_msg: Returns a black box data query message.
get_bb_latest_data_query_msg: Returns a black box latest data query message.
parse_bb_variable_msg: Returns a nested dictionary that reconstructs the structure of the data represented by the variables in the list of a given black box variable message dictionary.
parse_bb_data_msg: Returns a tuple (variables, data), where variables is a list of variables that were queried and data a list of variable values.
parse_bb_latest_data_msg: Returns a tuple (variables, data), where variables is a list of variables that were queried and data a list of the latest variable values.
split_into_windows: Given a one-dimensional list of elements "data", returns a 2D numpy array of sliding windows of size "window_size".
safe_literal_eval: Uses ast.literal_eval to parse a given string.
PlotUtils class with the following static methods:
subplot_data: Plots a single time series on a subplot (and potentially annotates any events of interest)
subplot_data_lists: Plots multiple time series on a single subplot (and potentially annotates any events of interest)
plot_position_velocity: Plots the position of robot with dots and its velocity at each position with a line. Colors represent the intensity of velocity.