Huggingface dataset integration by fsiino-nvidia · Pull Request #101 · NVIDIA-NeMo/Gym

fsiino-nvidia · 2025-09-26T21:40:29Z

This change adds support for Huggingface dataset management (upload/download/delete Gitlab artifact(s))

Addresses items 2 and 3 from #81

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

…r-organization Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

copy-pr-bot · 2025-09-26T21:40:33Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # resources_servers/comp_coding/configs/comp_coding.yaml

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

banghuaz-nvidia · 2025-10-17T17:58:45Z

Hey @fsiino-nvidia , can we resolve the conflict as well? Left some comments there also. Thx!

banghuaz-nvidia · 2025-10-16T20:07:17Z

+        print(f"[Nemo-Gym] - Repo '{repo_id}' already exists")
+    except HfHubHTTPError as e:
+        if e.response is not None and e.response.status_code == 404:
+            client.create_repo(repo_id=repo_id, token=config.hf_token, repo_type="dataset", private=True)


This seems to be a slightly unnatural way to check existing repo and then create one. Does create_repo not automatically check the existence?

added exist_ok=True to simplify this.

banghuaz-nvidia · 2025-10-17T17:58:21Z

+```
+
+Naming convention for Huggingface datasets is as follows:
+`{hf_organization}/{hf_collection_name}-{domain}–{resource_server_name}-{your dataset name}`


Can we decouple collection vs dataset prefix? There might be cases where we want to put it under "Nemo-Gym" collection without having a prefix "Nemo-Gym" there. So it would be nice to have two fields for these.

I changed this to hf_dataset_prefix. By default it is NeMo-Gym- and can be overridden with even an empty string if desired.

Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # .pre-commit-config.yaml # README.md # nemo_gym/config_types.py # resources_servers/google_search/configs/google_search.yaml # scripts/update_resource_servers.py

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # README.md # nemo_gym/config_types.py # scripts/update_resource_servers.py

banghuaz-nvidia · 2025-11-10T05:25:08Z

Hey @fsiino-nvidia , the PR should be ready to merge. Could you please resolve the current conflicts?

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

banghuaz-nvidia

LGTM. Thanks!

This change adds support for Huggingface dataset management (upload/download/delete Gitlab artifact(s)) Addresses items 2 and 3 from #81 --------- Signed-off-by: Frankie Siino <fsiino@nvidia.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

This change adds support for Huggingface dataset management (upload/download/delete Gitlab artifact(s)) Addresses items 2 and 3 from NVIDIA-NeMo#81 --------- Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia added 24 commits September 22, 2025 14:29

Init

cb63a92

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Rename script, pretty print table, add error handling

452c8e1

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Copyright

82f6930

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Enforce license and domain in resource server yaml

07ca159

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Extract training license, use enum for domains

23d59e3

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Update script

443cb2c

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Simplify domain check

c1c2e72

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Update readme

011b22b

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Recall hf_utils

1a9b2b5

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Test precommit order

28d998d

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Revert precommits, remove extra readme section

b847d19

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Test idempotent write

7ffa731

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Make sorting consistent

feb0e22

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

More sorting conflicts

c3f4104

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Merge remote-tracking branch 'github/main' into fsiino/resource-serve…

a902713

…r-organization Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Simplify

a5acbcb

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Add docstring

4e292f8

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Empty commit for tests

d5f2448

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Strip domain from non-resource server

eb19f4f

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Upload dataset to hf collection repo

97ccc8d

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Download hf dataset

7e91859

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Check format, use dataset attribute getter util

ee39c78

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Remove hardcoded vars, improve flow

bc6ca8b

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Delete from gitlab, refactor

6bbdc9f

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia added 2 commits September 26, 2025 16:25

Update readme

ac38e2c

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Add standalone delete, confirmation dialog

808024d

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia marked this pull request as ready for review September 27, 2025 00:29

fsiino-nvidia requested a review from a team as a code owner September 27, 2025 00:29

Merge remote-tracking branch 'github/main' into fsiino/hf-migration

cdc99ea

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia added 5 commits September 28, 2025 12:42

Add test

af32526

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Skip

eba405b

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Merge remote-tracking branch 'github/main' into fsiino/hf-migration

e3815eb

Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # resources_servers/comp_coding/configs/comp_coding.yaml

Remove name

bc26711

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Update tests

7afc186

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia linked an issue Sep 29, 2025 that may be closed by this pull request

HF Dataset Format Checking & Migration, Auto summary table generation #81

Closed

banghuaz-nvidia reviewed Oct 17, 2025

View reviewed changes

fsiino-nvidia added 2 commits October 17, 2025 12:26

Fix coverage, simplify repo creation, make dataset prefix flexible

e379fc0

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia requested a review from bxyu-nvidia October 17, 2025 22:29

fsiino-nvidia force-pushed the fsiino/hf-migration branch from 8af32c4 to f8fd7bc Compare October 22, 2025 00:02

Update docs

7d668f4

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia force-pushed the fsiino/hf-migration branch from f8fd7bc to 7d668f4 Compare October 22, 2025 00:08

Merge remote-tracking branch 'github/main' into fsiino/hf-migration

10a0ecb

Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # README.md # nemo_gym/config_types.py # scripts/update_resource_servers.py

fsiino-nvidia added 4 commits November 10, 2025 11:36

Move yaml related functions to utils

da09801

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Revert, keep visit functions exportable

7601b97

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Fix test for cli

f2a37d3

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Merge remote-tracking branch 'github/main' into fsiino/hf-migration

c52f84a

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia force-pushed the fsiino/hf-migration branch from f5fb64b to c52f84a Compare November 10, 2025 20:08

Add fallback

4195811

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

banghuaz-nvidia approved these changes Nov 11, 2025

View reviewed changes

banghuaz-nvidia removed the request for review from bxyu-nvidia November 11, 2025 00:11

banghuaz-nvidia merged commit 4a19206 into main Nov 11, 2025
6 checks passed

banghuaz-nvidia deleted the fsiino/hf-migration branch November 11, 2025 00:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Huggingface dataset integration#101

Huggingface dataset integration#101
banghuaz-nvidia merged 43 commits intomainfrom
fsiino/hf-migration

fsiino-nvidia commented Sep 26, 2025 •

edited

Loading

Uh oh!

copy-pr-bot bot commented Sep 26, 2025

Uh oh!

banghuaz-nvidia commented Oct 17, 2025

Uh oh!

banghuaz-nvidia Oct 16, 2025

Uh oh!

fsiino-nvidia Oct 17, 2025

Uh oh!

banghuaz-nvidia Oct 17, 2025

Uh oh!

fsiino-nvidia Oct 17, 2025

Uh oh!

banghuaz-nvidia commented Nov 10, 2025

Uh oh!

banghuaz-nvidia left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

fsiino-nvidia commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot bot commented Sep 26, 2025

Uh oh!

banghuaz-nvidia commented Oct 17, 2025

Uh oh!

banghuaz-nvidia Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

fsiino-nvidia Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

banghuaz-nvidia Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

fsiino-nvidia Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

banghuaz-nvidia commented Nov 10, 2025

Uh oh!

banghuaz-nvidia left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fsiino-nvidia commented Sep 26, 2025 •

edited

Loading