HF-preprocessed provider

[dsocolobsky]

Adds a new HuggingFacePreprocessedDataProvider Data Provider which fetches pre-tokenized datasets from HuggingFace via their streaming interface (so, on-demand).

I added a new test config hf-preprocessed-config.toml using emozilla/Hermes-3-Preprocessed-Llama3, training seemed to work alright but I'm not that familiar with these datasets/api so requires further testing.

[pefontana]

Nice one @dsocolobsky !
Just one thing, maybe we can check
in the PreprocessedDataProvider we check the sequence length
shared/data-provider/src/preprocessed.rs:99

                                let input_ids = list_to_vec(
                                    &row,
                                    inputs_column,
                                    Some(num_tokens_per_sequence),
                                )?;

Maybe we can do the same here, but only if doesnt involve a lot of code

[IAvecilla]

I wonder how this works with the shuffled indices. There’s a small chance we could end up with consecutive numbers, which would mean making one request per index, so we’d likely get rate-limited sooner rather than later. Maybe we could request the data using ordered indices and then shuffle the results afterward.

[dsocolobsky]

I've taken a look at this but at a first glance it seems like we can't avoid fetching 1 by 1 in the case the indices are shuffled. It's very unlikely that if we have e.g. 4 shuffled indices in the range [0,40000] some of them will be consecutive.

[IAvecilla]

I think we're already checking this in the get_samples function before calling this, can this be removed?

[dsocolobsky]

I think it's worth it to leave the check just in case at some point we re-use it somewhere. If we ever call this before checking before if num_rows=0 it will access out of bounds and panic. I prefer not to have this hidden condition.

[IAvecilla]

I think in the train.rs example we use the Preprocessed variant but it expects the data to be available locally. I think is not a priority and can be done in another PR but I think at some point we want to add this variant to be able to run the train example without needing the data locally beforehand #26

[philrhc]

it would be useful for me to learn about this if you could add information about this new data provider under the Provider Configuration subheading in psyche-book/src/explain/data-provider.md