Expose dataset loading functions as public API

[grovduck]

There are functions in sknnr.datasets._base that manage the loading of CSV files into Dataset objects (similar to sklearn's Bunch objects). At present, however, these utility functions (e.g. _load_dataset_from_csv_filenames and _load_csv_data) are meant for internal use and are currently used in load_moscow_stjoes and load_swo_ecoplot. As @aazuspan points out in lemma-osu/synthetic-knn#2, these functions (and callees) rely on "specific restrictions like requiring an integer index in the first column, and floats in the remaining columns" (see here).

Nonetheless, even with these restrictions, this functionality is likely useful across other packages that require sknnr or for users that may have similarly formatted CSV files. In order to use these functions in a public API, we would need to generalize _load_csv_data so that it possibly accepts DATA_MODULE as a keyword argument (as in sklearn.datasets.load_csv_data). This would work well for other packages that want to use this functionality, although perhaps users might have to create a module structure with their data to use it as well.

To consider:

• Should these data loading functions continue to live in sknnr.datasets or in a more generic "utilities" module?
• Do we need to support loading not only from modules (using importlib.resources) or from generic directories as well?

@aazuspan, further thoughts on this?

[aazuspan]

In order to use these functions in a public API, we would need to generalize _load_csv_data so that it possibly accepts DATA_MODULE as a keyword argument (as in sklearn.datasets.load_csv_data.

This sounds like an easy improvement without any downsides to me! Would that get passed through _load_dataset_from_csv_filenames as well? I assume that would be the most commonly used function for other packages making their own dataset loading functions? If so, we could probably come up with a slightly catchier name...

Should these data loading functions continue to live in sknnr.datasets or in a more generic "utilities" module?

I don't see anything wrong with having dataset-related utils like this in the sknnr.datasets module, but something like sknnr.utils.datasets also seems reasonable.

this functionality is likely useful across other packages that require sknnr or for users that may have similarly formatted CSV files.

I agree that there's utility to these functions for both other packages and end users, but I'm a little on the fence about the cost-benefit equation when it comes to making this work for the latter group. On the cost side, I see some complexity arising from a) probably needing to allow reading from paths as well as modules, and b) probably needing to validate data files to ensure they're compatible with the expected format.

Meanwhile, the benefits seem less significant for end users. As I see it, a lot of the complexity of these functions comes from 1) avoiding a strict dependency on pandas, and 2) creating a similar API to sklearn in terms of the different return type options, i.e. a Dataset of arrays vs. a Dataset of dataframes vs. X-y arrays vs. X-y dataframes, etc. Those both seem like worthwhile features for a dataset loading function in a package, but for users that don't mind importing pandas and just want X-y dataframes (presumably that's the majority of use cases?), I'm a little less confident that it's worth the effort and maintenance, if it could effectively be replaced with a couple calls to pd.read_csv.

Let me know what you think! If I'm underestimating the value of these functions for end users, I'm not opposed to adding some complexity to make them more flexible.

[grovduck]

This sounds like an easy improvement without any downsides to me! Would that get passed through _load_dataset_from_csv_filenames as well? I assume that would be the most commonly used function for other packages making their own dataset loading functions? If so, we could probably come up with a slightly catchier name...

Yes, passing it through _load_dataset_from_csv_filenames was my thought as well and I agree that would be the most commonly used function. I'm all for catchy names!

I don't see anything wrong with having dataset-related utils like this in the sknnr.datasets module, but something like sknnr.utils.datasets also seems reasonable.

sknnr.datasets seems like a good first place to start, especially when we don't necessarily have other functions that fit under a utils umbrella yet.

I'm a little on the fence about the cost-benefit equation when it comes to making this work for the latter group

Your entire argument makes perfect sense here. I agree that the complexity involved making it available for other users is not worth it when pd.read_csv is available. I think you're right on here, so let's focus on loading from a module.

I'm happy to take a first cut at this, but happy to defer to you as well.

[aazuspan]

sknnr.datasets seems like a good first place to start, especially when we don't necessarily have other functions that fit under a utils umbrella yet.

Makes sense! We can always re-evaluate if we end up with more utils.

I'm happy to take a first cut at this

Sounds great!