Skip to content

Changes to support R2 Downloader #548

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Jul 29, 2025
Merged

Changes to support R2 Downloader #548

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
624ecb9
Add r2 download linnke for llama2 and 3
anandhu-eng Jul 28, 2025
a9010d9
fix typo
anandhu-eng Jul 28, 2025
67cd653
set r2 downloader as default download tool
anandhu-eng Jul 28, 2025
69f0e4d
Add r2 download method for waymo validation dataset
anandhu-eng Jul 28, 2025
b3c4fcd
support r2 download for pointpainting
anandhu-eng Jul 28, 2025
0ab30af
[Automated Commit] Format Codebase [skip ci]
github-actions[bot] Jul 28, 2025
5b58564
r2_downloader -> r2-downloader
anandhu-eng Jul 29, 2025
0048b20
r2_downloader -> r2-downloader
anandhu-eng Jul 29, 2025
e8429e0
Update meta.yaml
anandhu-eng Jul 29, 2025
8dd575b
Update meta.yaml
anandhu-eng Jul 29, 2025
71e1e4c
Update meta.yaml
anandhu-eng Jul 29, 2025
84e9b0a
Merge branch 'dev' into #529
anandhu-eng Jul 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions script/get-dataset-mlperf-inference-llama3/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,28 @@ variations:
MLC_RCLONE_URL: mlc-inference:mlcommons-inference-wg-public/llama3.1_405b/mlperf_llama3.1_405b_dataset_8313_processed_fp16_eval.pkl
MLC_DATASET_TYPE: validation
MLC_DATASET_FILE_NAME: mlperf_llama3.1_405b_dataset_8313_processed_fp16_eval.pkl
validation,rclone:
env:
MLC_RCLONE_URL: mlc-inference:mlcommons-inference-wg-public/llama3.1_405b/mlperf_llama3.1_405b_dataset_8313_processed_fp16_eval.pkl
MLC_DATASET_FILE_NAME: mlperf_llama3.1_405b_dataset_8313_processed_fp16_eval.pkl
validation,r2-downloader:
env:
MLC_RCLONE_URL: ""
MLC_DATASET_FILE_NAME: ""
calibration:
group: dataset-type
env:
MLC_RCLONE_URL: mlc-inference:mlcommons-inference-wg-public/llama3.1_405b/mlperf_llama3.1_405b_calibration_dataset_512_processed_fp16_eval.pkl
MLC_DATASET_TYPE: calibration
MLC_DATASET_FILE_NAME: mlperf_llama3.1_405b_calibration_dataset_512_processed_fp16_eval.pkl
calibration,rclone:
env:
MLC_RCLONE_URL: mlc-inference:mlcommons-inference-wg-public/llama3.1_405b/mlperf_llama3.1_405b_calibration_dataset_512_processed_fp16_eval.pkl
MLC_DATASET_FILE_NAME: mlperf_llama3.1_405b_calibration_dataset_512_processed_fp16_eval.pkl
calibration,r2-downloader:
env:
MLC_RCLONE_URL: ""
MLC_DATASET_FILE_NAME: ""
rclone:
add_deps_recursive:
dae:
Expand All @@ -52,5 +68,13 @@ variations:
MLC_DOWNLOAD_URL: <<<MLC_RCLONE_URL>>>
MLC_RCLONE_CONFIG_NAME: mlc-inference
group: download-tool
r2-downloader:
add_deps_recursive:
dae:
tags: _r2-downloader
env:
MLC_DOWNLOAD_FILENAME: checkpoint
MLC_RCLONE_CONFIG_NAME: mlc-inference
group: download-tool
print_env_at_the_end:
MLC_DATASET_LLAMA3_PATH: Path to the dataset
34 changes: 24 additions & 10 deletions script/get-dataset-waymo/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,22 +21,12 @@ variations:
group: download-src
default: true
prehook_deps:
- tags: get,rclone
enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- yes
- tags: get,rclone-config,_waymo
force_cache: true
enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- yes
- enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- 'yes'
env:
MLC_DOWNLOAD_FINAL_ENV_NAME: MLC_DATASET_WAYMO_PATH
MLC_EXTRACT_FINAL_ENV_NAME: MLC_DATASET_WAYMO_PATH
MLC_DOWNLOAD_URL: mlc_waymo:waymo_preprocessed_dataset/kitti_format
extra_cache_tags: waymo,dataset
force_cache: true
names:
Expand All @@ -54,6 +44,21 @@ variations:
add_deps_recursive:
dae:
tags: _rclone
prehook_deps:
- tags: get,rclone
enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- yes
- tags: get,rclone-config,_waymo
force_cache: true
enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- yes
r2-downloader:
group: download-tool
add_deps_recursive:
dae:
tags: _r2-downloader
default: true
dry-run:
group: run-mode
Expand All @@ -62,3 +67,12 @@ variations:
dry-run,rclone:
env:
MLC_DOWNLOAD_EXTRA_OPTIONS: --dry-run
dry-run,r2-downloader:
env:
MLC_DOWNLOAD_EXTRA_OPTIONS: -x
rclone,mlc:
env:
MLC_DOWNLOAD_URL: mlc_waymo:waymo_preprocessed_dataset/kitti_format
r2-downloader,mlc:
env:
MLC_DOWNLOAD_URL: https://waymo.mlcommons-storage.org/metadata/dataset.uri
19 changes: 7 additions & 12 deletions script/get-ml-model-llama2/customize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,24 +29,19 @@ def preprocess(i):
else:
if path == '' or not os.path.exists(path):
env['MLC_TMP_REQUIRE_DOWNLOAD'] = 'yes'
if env['MLC_DOWNLOAD_SRC'] == "mlcommons":
i['run_script_input']['script_name'] = 'run-rclone'
if env.get('MLC_OUTDIRNAME', '') != '':
env['LLAMA2_CHECKPOINT_PATH'] = env['MLC_OUTDIRNAME']
else:
env['LLAMA2_CHECKPOINT_PATH'] = os.getcwd()

return {'return': 0}


def postprocess(i):

env = i['env']
if env.get('LLAMA2_CHECKPOINT_PATH', '') == '':
env['LLAMA2_CHECKPOINT_PATH'] = env['MLC_ML_MODEL_PATH']
else:
env['MLC_ML_MODEL_PATH'] = env['LLAMA2_CHECKPOINT_PATH']
env['MLC_ML_MODEL_LLAMA2_FILE_WITH_PATH'] = env['LLAMA2_CHECKPOINT_PATH']
env['MLC_GET_DEPENDENT_CACHED_PATH'] = env['MLC_ML_MODEL_PATH']
if env.get('MLC_DOWNLOAD_MODE', '') != "dry":
if env.get('LLAMA2_CHECKPOINT_PATH', '') == '':
env['LLAMA2_CHECKPOINT_PATH'] = env['MLC_ML_MODEL_PATH']
else:
env['MLC_ML_MODEL_PATH'] = env['LLAMA2_CHECKPOINT_PATH']
env['MLC_ML_MODEL_LLAMA2_FILE_WITH_PATH'] = env['LLAMA2_CHECKPOINT_PATH']
env['MLC_GET_DEPENDENT_CACHED_PATH'] = env['MLC_ML_MODEL_PATH']

return {'return': 0}
65 changes: 60 additions & 5 deletions script/get-ml-model-llama2/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,12 @@ tags:
- llama2-70b
- text-summarization
uid: 5db97be9f61244c6
tests:
needs_pat: true
run_inputs:
- variations_list:
- r2-downloader,70b,mlc,dry-run
- r2-downloader,7b,mlc,dry-run
variations:
L40s:
env:
Expand Down Expand Up @@ -84,11 +90,11 @@ variations:
MLC_ML_MODEL_PRECISION: int8
MLC_ML_MODEL_WEIGHT_DATA_TYPES: int8
group: precision
mlc:
group: download-source
default: true
env:
MLC_DOWNLOAD_SRC: mlcommons
rclone:
group: download-tool
add_deps_recursive:
dae:
tags: _rclone
prehook_deps:
- tags: get,rclone
enable_if_env:
Expand All @@ -99,6 +105,55 @@ variations:
enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- yes
r2-downloader:
group: download-tool
default: true
add_deps_recursive:
dae:
tags: _r2-downloader
dry-run:
group: run-mode
env:
MLC_DOWNLOAD_MODE: dry
dry-run,rclone:
env:
MLC_DOWNLOAD_EXTRA_OPTIONS: --dry-run
dry-run,r2-downloader:
env:
MLC_DOWNLOAD_EXTRA_OPTIONS: -x
mlc:
group: download-source
default: true
env:
MLC_DOWNLOAD_SRC: mlcommons
prehook_deps:
- enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- 'yes'
env:
MLC_DOWNLOAD_FINAL_ENV_NAME: LLAMA2_CHECKPOINT_PATH
MLC_EXTRACT_FINAL_ENV_NAME: LLAMA2_CHECKPOINT_PATH
force_cache: true
names:
- dae
tags: download-and-extract
force_env_keys:
- MLC_OUTDIRNAME
update_tags_from_env_with_prefix:
_url.:
- MLC_DOWNLOAD_URL
mlc,rclone,70b:
env:
MLC_DOWNLOAD_URL: mlc-llama2:Llama-2-70b-chat-hf
mlc,rclone,7b:
env:
MLC_DOWNLOAD_URL: mlc-llama2:Llama-2-7b-chat-hf
mlc,r2-downloader,70b:
env:
MLC_DOWNLOAD_URL: https://llama2.mlcommons-storage.org/metadata/llama-2-70b-chat-hf.uri
mlc,r2-downloader,7b:
env:
MLC_DOWNLOAD_URL: https://llama2.mlcommons-storage.org/metadata/llama-2-7b-chat-hf.uri
hf:
group: download-source
env:
Expand Down
4 changes: 0 additions & 4 deletions script/get-ml-model-llama2/run-rclone.sh
View file
Open in desktop

This file was deleted.

42 changes: 21 additions & 21 deletions script/get-ml-model-llama3/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,27 @@ variations:
env:
MLC_ML_MODEL_NAME: Llama-3.1-8b-Instruct
MLC_ML_MODEL_R2_HOSTED_NAME: llama3-1-8b-instruct
rclone:
group: download-tool
add_deps_recursive:
dae:
tags: _rclone
prehook_deps:
- tags: get,rclone
enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- yes
- tags: get,rclone-config,_mlperf-llama3-1
force_cache: true
enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- yes
default: true
r2-downloader:
group: download-tool
add_deps_recursive:
dae:
tags: _r2-downloader
mlc:
group: download-src
default: true
Expand Down Expand Up @@ -92,27 +113,6 @@ variations:
adr:
dae:
extra_cache_tags: llama3,dataset,rclone
rclone:
group: download-tool
add_deps_recursive:
dae:
tags: _rclone
prehook_deps:
- tags: get,rclone
enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- yes
- tags: get,rclone-config,_mlperf-llama3-1
force_cache: true
enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- yes
default: true
r2-downloader:
group: download-tool
add_deps_recursive:
dae:
tags: _r2-downloader
dry-run:
group: run-mode
env:
Expand Down
40 changes: 17 additions & 23 deletions script/get-ml-model-pointpainting/customize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,6 @@ def preprocess(i):
if env.get('MLC_ML_MODEL_POINT_PAINTING_PATH', '') == '' or env.get(
'MLC_ML_MODEL_DPLAB_RESNET50_PATH', '') == '':
env['MLC_TMP_REQUIRE_DOWNLOAD'] = "yes"
if env['MLC_DOWNLOAD_SRC'] == "mlcommons":
i['run_script_input']['script_name'] = 'run-rclone'
if env.get('MLC_OUTDIRNAME', '') != '':
env['MLC_ML_MODEL_POINT_PAINTING_TMP_PATH'] = env['MLC_OUTDIRNAME']
else:
env['MLC_ML_MODEL_POINT_PAINTING_TMP_PATH'] = os.path.join(
os.getcwd(), "model")

return {'return': 0}

Expand All @@ -39,21 +32,22 @@ def postprocess(i):

env = i['env']

if env.get('MLC_ML_MODEL_POINT_PAINTING_PATH', '') == '':
if env['MLC_ML_MODEL_PP_FORMAT'] == "onnx":
env['MLC_ML_MODEL_POINT_PAINTING_PATH'] = os.path.join(
env['MLC_ML_MODEL_POINT_PAINTING_TMP_PATH'], "pp.onnx")
else:
env['MLC_ML_MODEL_POINT_PAINTING_PATH'] = os.path.join(
env['MLC_ML_MODEL_POINT_PAINTING_TMP_PATH'], "pp_ep36.pth")

if env.get('MLC_ML_MODEL_DPLAB_RESNET50_PATH', '') == '':
if env['MLC_ML_MODEL_DPLAB_RESNET50_FORMAT'] == "onnx":
env['MLC_ML_MODEL_DPLAB_RESNET50_PATH'] = os.path.join(
env['MLC_ML_MODEL_POINT_PAINTING_TMP_PATH'], "deeplabv3+.onnx")
else:
env['MLC_ML_MODEL_DPLAB_RESNET50_PATH'] = os.path.join(
env['MLC_ML_MODEL_POINT_PAINTING_TMP_PATH'],
"best_deeplabv3plus_resnet50_waymo_os16.pth")
if env.get('MLC_DOWNLOAD_MODE', '') != "dry":
if env.get('MLC_ML_MODEL_POINT_PAINTING_PATH', '') == '':
if env['MLC_ML_MODEL_PP_FORMAT'] == "onnx":
env['MLC_ML_MODEL_POINT_PAINTING_PATH'] = os.path.join(
env['MLC_ML_MODEL_POINT_PAINTING_TMP_PATH'], "pp.onnx")
else:
env['MLC_ML_MODEL_POINT_PAINTING_PATH'] = os.path.join(
env['MLC_ML_MODEL_POINT_PAINTING_TMP_PATH'], "pp_ep36.pth")

if env.get('MLC_ML_MODEL_DPLAB_RESNET50_PATH', '') == '':
if env['MLC_ML_MODEL_DPLAB_RESNET50_FORMAT'] == "onnx":
env['MLC_ML_MODEL_DPLAB_RESNET50_PATH'] = os.path.join(
env['MLC_ML_MODEL_POINT_PAINTING_TMP_PATH'], "deeplabv3+.onnx")
else:
env['MLC_ML_MODEL_DPLAB_RESNET50_PATH'] = os.path.join(
env['MLC_ML_MODEL_POINT_PAINTING_TMP_PATH'],
"best_deeplabv3plus_resnet50_waymo_os16.pth")

return {'return': 0}
51 changes: 47 additions & 4 deletions script/get-ml-model-pointpainting/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,11 @@ variations:
env:
MLC_ML_MODEL_PP_FORMAT: onnx
MLC_ML_MODEL_DPLAB_RESNET50_FORMAT: onnx
mlc:
group: download-src
default: true
rclone:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we giving rclone as a backup download option? If so why delete run-rclone.sh?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @arjunsuresh,

Since RClone downloads are typically handled through the meta variations in most of our scripts, I’ve updated this accordingly. The download tool is now passed to the download and extract script (lines 55–70) via adr.

In short, both RClone and R2 Downloader are supported through this setup.

group: download-tool
add_deps_recursive:
dae:
tags: _rclone
prehook_deps:
- tags: get,rclone
enable_if_env:
Expand All @@ -40,6 +42,47 @@ variations:
enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- yes
r2-downloader:
group: download-tool
add_deps_recursive:
dae:
tags: _r2-downloader
default: true
mlc:
group: download-src
default: true
prehook_deps:
- enable_if_env:
MLC_TMP_REQUIRE_DOWNLOAD:
- 'yes'
env:
MLC_DOWNLOAD_FINAL_ENV_NAME: MLC_ML_MODEL_POINT_PAINTING_TMP_PATH
MLC_EXTRACT_FINAL_ENV_NAME: MLC_ML_MODEL_POINT_PAINTING_TMP_PATH
extra_cache_tags: waymo,dataset
force_cache: true
names:
- dae
tags: download-and-extract
force_env_keys:
- MLC_OUTDIRNAME
update_tags_from_env_with_prefix:
_url.:
- MLC_DOWNLOAD_URL
env:
MLC_DOWNLOAD_SRC: mlcommons

dry-run:
group: run-mode
env:
MLC_DOWNLOAD_MODE: dry
dry-run,rclone:
env:
MLC_DOWNLOAD_EXTRA_OPTIONS: --dry-run
dry-run,r2-downloader:
env:
MLC_DOWNLOAD_EXTRA_OPTIONS: -x
rclone,mlc:
env:
MLC_DOWNLOAD_URL: mlc-waymo:waymo_preprocessed_dataset/model
r2-downloader,mlc:
env:
MLC_DOWNLOAD_URL: https://waymo.mlcommons-storage.org/metadata/model.uri
Loading
Loading