-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] Luigi pipeline update #78
Conversation
Added NSynth pitch dataset with the new builder method |
heareval/tasks/nsynth_pitch.py
Outdated
|
||
# Filter out pitches that are not within the range of a standard piano | ||
metadata = metadata[metadata["pitch"] >= 21] | ||
metadata = metadata[metadata["pitch"] <= 108] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Make these config per @khumairraj's suggestion
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. Maybe we can define it in the NsynthPitchConfig
after the super()
. Basically, everything in the super is something that is required across all datasets. Any config specific to the dataset can be after the super. Mostly such a kind of config attribute will be affecting the ConfigureProcessMetadata
like this pitch thing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks awesome
task_class = new_class( | ||
name=name, | ||
bases=(base,), | ||
exec_body=partial(self.add_requirements, requirements), | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is v cool.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These refactors are great.
# train: 85,111; valid: 10,102; test: 4890. These values will be a bit less | ||
# after filter the pitches to be only within the piano range. | ||
# To subsample a partition, set the max_files to an integer. | ||
# TODO: Should we subsample NSynth? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Full nsynth is quite large. Should we subsample it?
Idea for updating the config and creating a more generic pipeline that can run based on the config files. To help keep the config files consistent I made them python classes -- these can be instantiated / modified by json in the future if we want.