Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Utils for discovering attached data and dropping in-memory data #1361

Merged
merged 1 commit into from
Jun 24, 2024

Conversation

pzelasko
Copy link
Collaborator

This PR adds the following methods and properties:

  • cut.iter_data() which iterates over (key, manifest) pairs of all data items attached to a given cut (e.g., ("recording", Recording(...)), ("custom_features", TemporalArray(...)))
  • is_in_memory property for all manifest types to indicate if it contains data that is held in memory
  • is_placeholder for non-cut manifests to indicate if a manifest is just a placeholder (has some metadata, but can't be used to load data)
  • cut.drop_in_memory_data() which converts manifests with in-memory data to placeholders (this is useful for manifests that live longer than just dataloading to avoid blowing up CPU memory and/or slowing down the program)

@pzelasko pzelasko added this to the v1.25.0 milestone Jun 24, 2024
@pzelasko
Copy link
Collaborator Author

Skipping the last test that fails due to numpy 2.0 release

@pzelasko pzelasko merged commit e3bed73 into master Jun 24, 2024
8 of 9 checks passed
@pzelasko pzelasko deleted the feature/utils-for-in-memory-data branch June 24, 2024 20:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant