Skip to content

Filter Bags#37

Merged
contagon merged 4 commits into
masterfrom
filter-bags
Jul 18, 2025
Merged

Filter Bags#37
contagon merged 4 commits into
masterfrom
filter-bags

Conversation

@contagon
Copy link
Copy Markdown
Owner

@contagon contagon commented Jul 9, 2025

This PR adds a (destructive) command to remove all data from downloaded rosbags that aren't the required IMU or LiDAR data. This greatly reduces storage size (by 50% for newer college)!

I'm on the fence about the command name. I considered filter, clean, or prune, but felt that filter best represents what is happening here.

@contagon contagon requested a review from Copilot July 10, 2025 13:39
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Adds a destructive CLI command to strip non-IMU/LiDAR data from ROS1 and ROS2 bags, reducing storage size by ~50%.

  • Implements filter_ros1 and filter_ros2 helper functions to rewrite bags with only the specified topics.
  • Adds a filter command in dataset_manager.py, wiring up topic-based filtering for downloaded datasets.
  • Bumps rosbags dependency to >=0.10.10 in pyproject.toml.

Reviewed Changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 3 comments.

File Description
python/evalio/cli/dataset_manager.py New filter command, plus filter_ros1 and filter_ros2 functions for topic-based bag rewriting.
pyproject.toml Updated rosbags dependency to >=0.10.10 for required bugfixes.
Comments suppressed due to low confidence (2)

python/evalio/cli/dataset_manager.py:191

  • [nitpick] Naming this function filter shadows the built-in filter; consider renaming to avoid confusion (e.g., filter_bags).
def filter(

python/evalio/cli/dataset_manager.py:190

  • No tests have been added for the new filter command; consider adding unit or integration tests to verify destructive behavior and error paths.
@app.command(no_args_is_help=True)

Comment thread python/evalio/cli/dataset_manager.py
Comment thread python/evalio/cli/dataset_manager.py Outdated
Comment thread python/evalio/cli/dataset_manager.py Outdated
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@contagon contagon merged commit 42ae1d4 into master Jul 18, 2025
12 checks passed
@contagon contagon deleted the filter-bags branch July 19, 2025 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants