NeMo RL training tutorial errors

[cwing-nvidia]

1. The sanity test script used to validate the training environment fails with the following assertion error: AssertionError: assert {'final_batch...': None, ...}} == {'final_batch...': None, ...}}.

2. The training tutorial for Nemo-RL fails because the NeMo RL container does not include Megatron. This leads to a “module not found” error when attempting to run the Nemo Gym training example on a single A100 GPU.

[cmunley1]

NeMo RL container does not include Megatron

seems wrong. you did submodule update --init --recursive? what container is this? can u share full steps to reproduce?

[ffrujeri]

I also followed the whole tutorial and didn't run into any issues

ng_prepare_data "+config_paths=[${config_paths}]" \
    +output_dirpath=data/workplace_assistant \
    +mode=train_preparation \
    +should_download=true \
    +data_source=huggingface


####################################################################################################
#
# Load and validate server instance configs
#
####################################################################################################

Found 3 server instance configs (1 agent configs):
- policy_model (responses_api_models)
- workplace_assistant (resources_servers)
- workplace_assistant_simple_agent (responses_api_agents)


Found 0 agent server instance configs WITHOUT datasets:


Found 1 agent server instance configs WITH datasets:
- workplace_assistant_simple_agent
  - train (train)
  - validation (validation)
  - example (example)


In scope dataset types for `train_preparation` mode: ['train', 'validation']
Found 1 agent server instance configs with in-scope datasets:
- workplace_assistant_simple_agent
  - train (train)
  - validation (validation)


####################################################################################################
#
# Load datasets. Missing datasets **WILL** be downloaded.
#
####################################################################################################

FOUND the following datasets at their local paths:


MISSING the following datasets:
- workplace_assistant_simple_agent
  - train (train)
  - validation (validation)


Downloading 'train' split from nvidia/Nemotron-RL-agent-workplace_assistant to resources_servers/workplace_assistant/data/train.jsonl...
train.jsonl: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20.9M/20.9M [00:00<00:00, 34.6MB/s]
[Nemo-Gym] - Downloaded train.jsonl to: resources_servers/workplace_assistant/data/train.jsonl
Downloading 'validation' split from nvidia/Nemotron-RL-agent-workplace_assistant to resources_servers/workplace_assistant/data/validation.jsonl...
validation.jsonl: 9.09MB [00:00, 44.6MB/s]
[Nemo-Gym] - Downloaded validation.jsonl to: resources_servers/workplace_assistant/data/validation.jsonl


####################################################################################################
#
# Validate samples and aggregate metrics
#
####################################################################################################

resources_servers/workplace_assistant/data/train.jsonl: 1255it [00:00, 1619.27it/s]
Aggregate metrics for resources_servers/workplace_assistant/data/train_metrics.json
{
    'name': 'train',
    'type': 'train',
    'jsonl_fpath': 'resources_servers/workplace_assistant/data/train.jsonl',
    'num_repeats': 1,
    'gitlab_identifier': None,
    'huggingface_identifier': {
        'repo_id': 'nvidia/Nemotron-RL-agent-workplace_assistant',
        'artifact_fpath': 'train.jsonl',
    },
    'license': 'Apache 2.0',
    'Number of examples': 1255,
    'Number of tools': {
        'Total # non-null values': 1255,
        'Average': 27.0,
        'Min': 27.0,
        'Max': 27.0,
        'Standard deviation': 0.0,
    },
    'Json-dumped number of words (proxy for token count)': {
        'Total # non-null values': 1255,
        'Average': 1724.95,
        'Min': 1713.0,
        'Max': 1747.0,
        'Standard deviation': 6.2,
    },
    'Number of turns': {
        'Total # non-null values': 1255,
        'Average': 1.0,
        'Min': 1.0,
        'Max': 1.0,
        'Standard deviation': 0.0,
    },
    'Temperature': {
        'Total # non-null values': 1255,
        'Average': 1.0,
        'Min': 1.0,
        'Max': 1.0,
        'Standard deviation': 0.0,
    },
    'id': {
        'Total # non-null values': 1255,
        'Average': 627.0,
        'Min': 0.0,
        'Max': 1254.0,
        'Standard deviation': 362.43,
    },
    'category': {
        'unique_count': 5,
        'total_count': 1255,
    },
    'environment_name': {
        'unique_count': 1,
        'total_count': 1255,
    },
}
resources_servers/workplace_assistant/data/validation.jsonl: 545it [00:00, 1603.18it/s]
Aggregate metrics for resources_servers/workplace_assistant/data/validation_metrics.json
{
    'name': 'validation',
    'type': 'validation',
    'jsonl_fpath': 'resources_servers/workplace_assistant/data/validation.jsonl',
    'num_repeats': 1,
    'gitlab_identifier': None,
    'huggingface_identifier': {
        'repo_id': 'nvidia/Nemotron-RL-agent-workplace_assistant',
        'artifact_fpath': 'validation.jsonl',
    },
    'license': 'Apache 2.0',
    'Number of examples': 545,
    'Number of tools': {
        'Total # non-null values': 545,
        'Average': 27.0,
        'Min': 27.0,
        'Max': 27.0,
        'Standard deviation': 0.0,
    },
    'Json-dumped number of words (proxy for token count)': {
        'Total # non-null values': 545,
        'Average': 1724.98,
        'Min': 1713.0,
        'Max': 1746.0,
        'Standard deviation': 6.27,
    },
    'Number of turns': {
        'Total # non-null values': 545,
        'Average': 1.0,
        'Min': 1.0,
        'Max': 1.0,
        'Standard deviation': 0.0,
    },
    'Temperature': {
        'Total # non-null values': 545,
        'Average': 1.0,
        'Min': 1.0,
        'Max': 1.0,
        'Standard deviation': 0.0,
    },
    'id': {
        'Total # non-null values': 545,
        'Average': 272.0,
        'Min': 0.0,
        'Max': 544.0,
        'Standard deviation': 157.47,
    },
    'category': {
        'unique_count': 5,
        'total_count': 545,
    },
    'environment_name': {
        'unique_count': 1,
        'total_count': 545,
    },
}


####################################################################################################
#
# Collate samples and aggregate metrics
#
####################################################################################################

resources_servers/workplace_assistant/data/train.jsonl: 1255it [00:00, 5751.65it/s]
Collating train datasets: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 72.15it/s]
Aggregate metrics for data/workplace_assistant/train_metrics.json
{
    'Number of examples': 1255,
    'Number of tools': {
        'Total # non-null values': 1255,
        'Average': 27.0,
        'Min': 27.0,
        'Max': 27.0,
        'Standard deviation': 0.0,
    },
    'Json-dumped number of words (proxy for token count)': {
        'Total # non-null values': 1255,
        'Average': 1724.95,
        'Min': 1713.0,
        'Max': 1747.0,
        'Standard deviation': 6.2,
    },
    'Number of turns': {
        'Total # non-null values': 1255,
        'Average': 1.0,
        'Min': 1.0,
        'Max': 1.0,
        'Standard deviation': 0.0,
    },
    'Temperature': {
        'Total # non-null values': 1255,
        'Average': 1.0,
        'Min': 1.0,
        'Max': 1.0,
        'Standard deviation': 0.0,
    },
    'id': {
        'Total # non-null values': 1255,
        'Average': 627.0,
        'Min': 0.0,
        'Max': 1254.0,
        'Standard deviation': 362.43,
    },
    'category': {
        'unique_count': 5,
        'total_count': 1255,
    },
    'environment_name': {
        'unique_count': 1,
        'total_count': 1255,
    },
}
resources_servers/workplace_assistant/data/validation.jsonl: 545it [00:00, 5849.78it/s]
Collating validation datasets: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 150.89it/s]
Aggregate metrics for data/workplace_assistant/validation_metrics.json
{
    'Number of examples': 545,
    'Number of tools': {
        'Total # non-null values': 545,
        'Average': 27.0,
        'Min': 27.0,
        'Max': 27.0,
        'Standard deviation': 0.0,
    },
    'Json-dumped number of words (proxy for token count)': {
        'Total # non-null values': 545,
        'Average': 1724.98,
        'Min': 1713.0,
        'Max': 1746.0,
        'Standard deviation': 6.27,
    },
    'Number of turns': {
        'Total # non-null values': 545,
        'Average': 1.0,
        'Min': 1.0,
        'Max': 1.0,
        'Standard deviation': 0.0,
    },
    'Temperature': {
        'Total # non-null values': 545,
        'Average': 1.0,
        'Min': 1.0,
        'Max': 1.0,
        'Standard deviation': 0.0,
    },
    'id': {
        'Total # non-null values': 545,
        'Average': 272.0,
        'Min': 0.0,
        'Max': 544.0,
        'Standard deviation': 157.47,
    },
    'category': {
        'unique_count': 5,
        'total_count': 545,
    },
    'environment_name': {
        'unique_count': 1,
        'total_count': 545,
    },
}
View your final data!
- train: data/workplace_assistant/train.jsonl
- validation: data/workplace_assistant/validation.jsonl


####################################################################################################
#
# Finished!
#
####################################################################################################