version 0.3 alpha #88

- add the following to REQUIRED: pandas, torch, torchvision, dask - bump version of crowsetta in REQUIRED to >=2.1.0

with keras.utils.Sequence class for Vak datasets

after changing vak.dataset.spect to vak.dataset.dataframe

so it adds a 'split' column to a DataFrame, instead of splitting a Dataset into subsets and then returning each subset as a separate Dataset.

- instead of having 'train_vds_path', 'val_vds_path', etc. - the 'split' column in the csv tells vak what splits there are and which samples belong to them

to emphasize this is a higher-level function that makes a dataset from **either** audio or spectrogram files

because they're not file input-output functions, so they don't belong in .io

to avoid namespace clash with Dataset classes used when working with networks

- validate_and_get_timebin_dur -- checks there is a single unique value for the duration of time bins in all the spectograms in the dataset, using the 'timebin_dur' column from the dataframe. Returns that value. - split_dur -- get duration of a split in the dataset. Sums value of 'duration' column for only the rows where 'split' column == split specified by user in call to function. - both functions make vak.cli.train more concise and can be used elsewhere

- class method 'fit_df' lets cli functions + user get a fit SpectScaler by passing it a pandas.DataFrame - avoid loading the whole dataset this way, just loop through spectrograms - cleaned up docstrings, match numpy spec

- because it's outside of scope of function - user or other functions can just call df.to_csv()

instead of using convoluted calls to dataframe.from_files; removed csv-saving capability from that function in previous commit

- they are now all absolute paths - so dataframe "works" from anywhere

- models and model_config_parsers entry points

- import models module, use renamed functions from that module - change relevant variable names - TrainConfig now has 'models' attribute - fix docstring for attribute that was wrong anyway

- batch_size, optimizer, learning_rate, loss

because it defaults to None and it's dumb to specify a default, esp. if doing so would require converting from string 'None' to NoneType and confusing the attrs.converters.optional(int) that tries to convert 'None' to an int and then raises an error

so that it doesn't say sections that have names of models are invalid

so its signature matches that of other sections. This allows logic for unpacking sections can be written more concisely in vak.config.parse.parse_config

- change config_file argument to config_path - use Pathlib instead of os.path - remove networks attribute from Config class + loading model configs will happen within cli functions + because they know which section's list of model is appropriate for them to load - rewrite function logic to be more succint + possibly too clever by half but there was a lot of repeated code

- import entire parse module, so user writes config.parse.from_file - import Config class from .parse module, just so it's there

in learncurve, train, and prep, change 'config_file' to 'config_path', for consistency with other code and because it's clearer (to my mind at least).

- convert SpectConfig NamedTuple to an attrs class - rewrite parse_spect_config to simply convert the SPECTROGRAM section of the config into a dict and then let the attrs class do all the work of converting, validating, etc - add attributes 'spect_key', 'timebins_key', 'freqbins_key' + so that there are defaults that always exist at the level of the config if some function needs them, e.g. vak.cli.train

now it gets the default Spectrogram values even when the config.ini file pointed to by config_path does not contain a Spectrogram section this way the default spect_key/timebins_key/freqbins_key options are always there

seems like least painful way to share parameters across train/predict etc for dataloader even though it is adding yet another section to the config.ini file. Will have a default so in theory some users might not have to think about it at first

- to use the same logic + structure as the rewritten train/prep modules + let attrs do as much as work as possible for converting/validating - stop checking for train/predict_vds paths

to a directory that exists, so config.parse.from_file won't throw an error

that lets calling functions supply a list of section names that should be parsed from config.ini files makes it possible for vak.cli.prep to get just the prep section without parsing train/learncurve/predict sections just yet, and in doing so throwing a bunch of errors about paths that don't exist (yet but will after running train or setting up the config.ini file for predict -- would be annoying for user to get these errors all the time) this used to be hard-wired but the way the functions are written now seems more concise yet general (I'm telling myself?)

to use 'sections' argument

from SPECTROGRAM to SPECT_PARAMS

- change SPECTROGRAM section to SPECT_PARAMS - and add new options to that section related to array files

- remove separate arguments for spect_key, timebins_key, freqbins_key, and audio_path - instead use them from spect_params passed in - assume spect_params will be a SpectParamsConfig object, or that we can convert it into one (i.e. it's a dict with keyword arg-value pairs) - this blurs the lines around what Config classes are for -- not just for parsing config.ini files - but it keeps this function from having three thousand arguments

to use prep with single config_path argument

with 'has_unlabeled' function -- used to determine whether segments without labels are present in dataset, in which case a class label needs to be assigned for "unlabeled" time bins

add converter that automatically turns it into a set

- remove the bespoke classes in vak/io/dataset + MetaSpect, Vocalization, and VocalizationDataset - remove tests for those classes - remove imports for both from __init__.py files

- dataset is now "datasets" to match convention switch torch + torchvision - dataloaders.py is now spectrogram_window_dataset.py - datasets/__init__.py imports SpectrogramWindowDataset from its module

renaming Dataset to SpectrogramWindowDataset made me realize that some unit tests were checking for the wrong return type -- should now be a pandas Dataframe, not any of the vak-specific classes

- in a model module - and add a util module + that has function that Model uses to validate attributes of subclasses - idea is that all Models must declare netowork, optimizer, loss, and metrics - and use a common API with fit, evaluate, predict + subclasses can also use this function if they are also abstract classes meant for sub-classing

move code chunk so adding the option 'results_dir_made_by_main_script' happens before running training instead of after, so it will be there even if training gets stopped in the middle

may add back later but right now can't figure out if engine + cli even works with all the stupid broken tensorflow imports

that will be attributes of any Model sub-class and so will be specified in that Model's section of a config.toml file

to work with toml files

- removes code not directly related to validation from config/validators - moves files_from_dir from io.audio into a util module, so that just the main (and relatively lengthy) 'to_spect' function lives in io.audio - wraps evfuncs.load_cbin in a decorator that switches the returned argument order, so it's the same order as wavfile.read, i.e. sampling frequency is element 0 of the return tuple, and the actual data is element 1 of the return tuple + this was not a problem before with the 'built-in' evfuncs module that I tore out - config/validators now just uses AUDIO_FORMAT_FUNC_MAP from util.audio

- remove constant + functions having to do with config parsers, users will not have to write their own config parsers - rewrite from_model_config_map to just use dicts, not expect any attrs-based config class it needs to cast to a dict, and not have to parse any section using some config_parser function. Just use the dict straight from toml as kwargs to the model's __init__ function

make comma_separated_list converter just return the input value if it is already a list, or convert to list if it is a str, else raise a TypeError instead of crashing by trying to convert a list to a list

because no other functions actually use this attribute. User can just look at the path they specified as "root_results_dir" to figure out where the results are.

e.g. if there are no args passed to the loss function when instantiating it, just map the 'loss' key to an empty dictionary which will the get passed as the kwargs

from 'results' command, because 'config' command doesn't exist anymore

with iter_ helper function, that was in vak/models/util.py

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

version 0.3 alpha #88

version 0.3 alpha #88

Commits on Dec 17, 2019

Commits on Jan 14, 2020

Commits on Jan 19, 2020

Commits on Feb 9, 2020