Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable large file support #60448

Open
Flamefire opened this issue Jun 22, 2021 · 0 comments
Open

Enable large file support #60448

Flamefire opened this issue Jun 22, 2021 · 0 comments
Labels
feature A request for a proper, new feature. module: build Build system issues triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@Flamefire
Copy link
Collaborator

Flamefire commented Jun 22, 2021 •
edited by pytorch-probot bot

馃殌 Feature

Some dependencies (e.g. minizip) will benefit from the large file support that is enabled by defining _LARGEFILE64_SOURCE=1, see their respective docus.
It might be a good idea to enable this globally in the PyTorch main CMake file.

Motivation

Especially in ML I expect large files to occur often enough that it is worth to enable their support by default. E.g. the Imagenet files, when compressed into an archive, are already really huge.

cc @malfet @seemethere @walterddr
@mrshenli mrshenli added feature A request for a proper, new feature. module: build Build system issues triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Jun 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature A request for a proper, new feature. module: build Build system issues triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Flamefire @mrshenli