Small subset #28
Small subset #28
Conversation
| @@ -537,6 +538,13 @@ def fit(self, dataset: LabeledDataset) -> "PytorchSupervised": | |||
| train_dataset, tune_dataset = dataset.train_test_split(train_fraction=train_fraction, | |||
| train_study_count=train_count, | |||
| seed=seed) | |||
| # For very small training sets the tune dataset will be empty | |||
There was a problem hiding this comment.
Do you have to hold out entire studies for your tune dataset? I get why that's important for your test data (you want to evaluate if your model generalizes to unseen studies), but for your tune data maybe it's okay to split up studies, at least in the case when your training set is small.
Just an idea, not necessary to change anything now.
There was a problem hiding this comment.
That's a good idea, but given the differences between studies I imagine I would end up overfitting if I did early stopping using the tune loss. I could probably do the splitting differently though. Maybe I should just designate a single study at the beginning of the run to be a tuning set shrug.
I'll open an issue and add your comment to it for thinking about in the future
| from saged import utils, datasets, models | ||
|
|
||
|
|
||
| def subset_to_equal_ratio(train_data: datasets.LabeledDataset, |
There was a problem hiding this comment.
This is similar to the function in saged/keep_ratios.py, right? Could you combine them into a single function?
There was a problem hiding this comment.
Right, it's an updated version of the one in saged/keep_ratios.py. When I refactor I'm moving this one to utils.py and calling changing both benchmarks to just call the utils version
| print(plot) | ||
|
|
||
| # %% | ||
| plot = ggplot(tb_metrics, aes(x='train_count', y='balanced_accuracy', color='supervised')) |
There was a problem hiding this comment.
You could try plotting this with error bars (e.g. standard error) to emphasize that the crossover isn't really significant, if you want. (At least I assume it won't be, looking at the data points)
There was a problem hiding this comment.
I think geom_smooth is supposed to plot confidence intervals by default but didn't here for some reason. Maybe the fact that the x axis isn't really continuous?
This PR finds the performance crossover point between a three layer neural network and logistic regression by looking at small training datasets.
Interestingly, we find that tb increases to a point then flatlines, while sepsis seems to take more data to saturate. Also it looks like the training of
deep_netis very broken, which is unsurprising given past results.Changes:
small_subsets.pysmall_subsets.py