Skip to content

Conversation

@ved1beta
Copy link
Contributor

What does this PR do?

This PR adds support for custom types in data loaders, allowing any iterable dataloader-like object to be used with proper device placement. This enhancement improves flexibility when working with custom data structures while maintaining compatibility with Accelerate's device management features.

Specifically, this PR:

  1. Introduces a new CustomTypesDataLoader class that can handle any iterable dataloader-like object
  2. Modifies the DataLoaderConfiguration to support custom types
  3. Adds proper tensor conversion and device placement handling
  4. Includes comprehensive tests to validate the functionality

Fixes #2975

Before submitting

Who can review?

@SunMarc @zach-huggingface as this touches core parts of the library, specifically the data loader functionality.

The changes include:

  • New data loader class in core library
  • Device placement handling
  • Test suite additions

@SunMarc
Copy link
Member

SunMarc commented May 15, 2025

Sorry for the wait, I'll have a look at this soon !

@github-actions
Copy link
Contributor

github-actions bot commented Jun 8, 2025

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

@github-actions github-actions bot closed this Jun 16, 2025
@ved1beta ved1beta deleted the Fix_3008_customtypes_dataloader branch June 16, 2025 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Barebones dataloader to allow for any type of iterable dataloader-like object to be used. Should just handle device placement

2 participants