diff --git a/.agents/skills/cosmos3-codebase-nav/SKILL.md b/.agents/skills/cosmos3-codebase-nav/SKILL.md new file mode 100644 index 0000000..ede78e0 --- /dev/null +++ b/.agents/skills/cosmos3-codebase-nav/SKILL.md @@ -0,0 +1,122 @@ +--- +name: cosmos3-codebase-nav +description: > + Navigate the Cosmos3 package codebase to find where parameters, configs, defaults, + scripts, and documentation live. Use when the user asks "where is X in cosmos3", + "how do I find the config for Y", "where are the defaults", "where do I change a + parameter", or any question about locating files, modules, or settings. Also use + when the user opens or edits files and needs orientation. +--- + +# Cosmos3 Codebase Navigation + +## When to use this skill + +- Use this skill when an agent is navigating the Cosmos3 package +- Use this skill to answer "where is X", "how do I find the config for Y", or any file-location question +- Use this skill when the user opens or edits cosmos3 files and needs orientation + +## Path convention + +All paths below are relative to this file's location (`.agents/skills/cosmos3-codebase-nav/`). The repo is laid out as: + +- `cosmos_framework/` — main training package (data, model, trainer, callbacks, checkpoint, utils, …). +- `cosmos_framework/configs/base/experiment/` — vfm (generator) experiment SKUs referenced by `[train.train_policy].experiment` in the recipe TOMLs. +- `cosmos_framework/configs/base/vlm/experiment/` — vlm (reasoner) experiment SKUs. +- `cosmos_framework/inference/` — inference subpackage (args, model, inference engine, defaults, Ray serving, common helpers). +- `cosmos_framework/scripts/` — top-level entry-point scripts (train, inference, eval, export_model, convert_model_to_dcp, upsample_prompts, caption_from_video, captions_to_sft_jsonl, action_policy_server, …). Invoked as `python -m cosmos_framework.scripts.`. +- `examples/toml/sft_config/.toml` + `examples/launch_sft_.sh` — paired SFT recipes (training entry-point input). The shell sources `examples/_sft_launcher_common.sh`, which forwards into `cosmos_framework.scripts.train --sft-toml=...`. +- `cosmos_framework/configs/toml_config/` — pydantic schemas (`sft_config.py`) and helpers that validate the recipe TOML at load time. + +## Quick Reference + +### Where parameters and defaults live + +| What you're looking for | File | +| --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | +| Sampling params (num_steps, guidance, shift, fps, etc.) | `../../../cosmos_framework/inference/args.py` → `SamplingArgs`, `SamplingOverrides` | +| Per-modality default values | `../../../cosmos_framework/inference/defaults//sample_args.json` | +| Setup params (parallelism, checkpoints, model path) | `../../../cosmos_framework/inference/args.py` → `OmniSetupArgs`, `OmniSetupOverrides` | +| Common args base classes | `../../../cosmos_framework/inference/common/args.py` → `ArgsBase`, `OverridesBase` | +| Ray serving parallelism presets | `../../../cosmos_framework/inference/ray/configs/latency.yaml`, `../../../cosmos_framework/inference/ray/configs/throughput.yaml` | +| Feature flags | `../../../cosmos_framework/utils/flags.py` | +| Prompt upsampler system prompt | `../../../cosmos_framework/inference/defaults/prompt_upsampler.txt` | +| Video captioner system prompt | `../../../cosmos_framework/inference/defaults/video_captioner.txt` | +| SFT recipe TOMLs (paired with `examples/launch_sft_*.sh`) | `../../../examples/toml/sft_config/.toml` | +| SFT pydantic schema (validates the recipe TOML) | `../../../cosmos_framework/configs/toml_config/sft_config.py` | +| Training experiment SKUs (vfm) | `../../../cosmos_framework/configs/base/experiment/` | +| Training experiment SKUs (vlm / reasoner) | `../../../cosmos_framework/configs/base/vlm/experiment/` | +| Example inputs | `../../../inputs/omni/t2i.json`, `../../../inputs/omni/t2v.json`, `../../../inputs/omni/i2v.json`, … | + +Available modality modes for defaults: `text2image`, `text2video`, `image2video`, `image2image`, `video2video`, `forward_dynamics`, `inverse_dynamics`, `policy`. + +### Config defaults resolution chain + +When a user runs inference, default parameter values are resolved in this order: + +``` +cosmos_framework/inference/defaults//sample_args.json # 1. Per-modality JSON defaults (num_steps, guidance, shift, fps, etc.) + ↓ +_load_modality_defaults() in cosmos_framework/inference/args.py # 2. Loaded and cached at import time + ↓ +SamplingArgs / SamplingOverrides # 3. Pydantic models with field-level validation + ↓ +OmniSampleOverrides.build_sample() # 4. Merges user overrides → final resolved args + ↓ +_RESOLUTION_SHIFT_DEFAULTS[model_size, resolution] # 5. Model+resolution shift override (if user didn't set shift) + ↓ +CLI flags (--guidance, --shift, etc.) # 6. User overrides from command line +``` + +The `_RESOLUTION_SHIFT_DEFAULTS` table in `../../../cosmos_framework/inference/args.py` (on `OmniSampleOverrides`) overrides the default `shift` based on model size and resolution, unless the user explicitly specified `--shift`. + +| Mode | Default file | Key defaults | +| ------------- | --------------------------------------------------------------------------- | ---------------------------------------------- | +| `text2image` | `../../../cosmos_framework/inference/defaults/text2image/sample_args.json` | `num_frames=1`, `guidance=6.0`, `shift=10.0` | +| `text2video` | `../../../cosmos_framework/inference/defaults/text2video/sample_args.json` | `num_frames=189`, `guidance=6.0`, `shift=10.0` | +| `image2video` | `../../../cosmos_framework/inference/defaults/image2video/sample_args.json` | `num_frames=189`, `guidance=6.0`, `shift=10.0` | + +Action and video2video modes also have defaults under `cosmos_framework/inference/defaults/{image2image,video2video,forward_dynamics,inverse_dynamics,policy}/sample_args.json`. + +Users can also supply a custom defaults file per-request via the `defaults_file` field in sample arguments (see `../../../docs/inference.md`). + +### Where to make changes + +| Task | Edit | +| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | +| Change a built-in default value | `../../../cosmos_framework/inference/defaults//sample_args.json` | +| Add a new CLI parameter | `SamplingArgs` + `SamplingOverrides` in `../../../cosmos_framework/inference/args.py`, then add to each `sample_args.json` | +| Change parallelism presets | `../../../cosmos_framework/inference/ray/configs/latency.yaml` or `throughput.yaml` | +| Add a new script | `../../../cosmos_framework/scripts/` — follow `inference.py` as the pattern | + +### Key entry points + +| Entry point | How to run | +| -------------------- | -------------------------------------------------------------------------------------------- | +| Batch inference | `python -m cosmos_framework.scripts.inference` | +| Training | `python -m cosmos_framework.scripts.train --sft-toml=examples/toml/sft_config/.toml` | +| Action evaluation | `python -m cosmos_framework.scripts.eval` | +| Online serving (Ray) | `python -m cosmos_framework.inference.ray.serve` | +| Submit to Ray server | `python -m cosmos_framework.inference.ray.submit` | +| Gradio UI | `python -m cosmos_framework.inference.ray.gradio` | +| Prompt upsampling | `python -m cosmos_framework.scripts.upsample_prompts` | +| Model export (HF) | `python -m cosmos_framework.scripts.export_model` | +| DCP conversion | `python -m cosmos_framework.scripts.convert_model_to_dcp` | +| Diffusers conversion | `python -m cosmos_framework.scripts.convert_model_to_diffusers` | +| Video captioning | `python -m cosmos_framework.scripts.caption_from_video` | +| Captions → SFT JSONL | `python -m cosmos_framework.scripts.captions_to_sft_jsonl` | +| Action policy server | `python -m cosmos_framework.scripts.action_policy_server` | + +### Documentation + +| Doc | Covers | +| ----------------------------------- | ---------------------------------------------------------- | +| `../../../AGENTS.md` | Commands, rules, key file locations (read this first) | +| `../../../README.md` | Overview, quickstart, examples | +| `../../../docs/setup.md` | Installation, environment, checkpoints | +| `../../../docs/code_structure.md` | Repo layout and per-subpackage tour of `cosmos_framework/` | +| `../../../docs/inference.md` | Sample args, default values, custom defaults | +| `../../../docs/inference_online.md` | Ray Serve and Gradio | +| `../../../docs/prompting.md` | Prompt engineering, upsampling | +| `../../../docs/training.md` | SFT / post-training workflow | +| `../../../docs/faq.md` | FAQ, tips, and troubleshooting | diff --git a/.agents/skills/cosmos3-env-troubleshoot/SKILL.md b/.agents/skills/cosmos3-env-troubleshoot/SKILL.md new file mode 100644 index 0000000..0251a2c --- /dev/null +++ b/.agents/skills/cosmos3-env-troubleshoot/SKILL.md @@ -0,0 +1,106 @@ +--- +name: cosmos3-env-troubleshoot +description: > + Diagnose and fix Cosmos3 environment, installation, and runtime errors. + Use when the user encounters an ImportError, ModuleNotFoundError, CUDA error, + Docker error, checkpoint download failure, or any traceback during setup or inference. +--- + +# Cosmos3 Environment Troubleshooting + +## When to use this skill + +- Use when a user hits an error during installation, environment setup, or first run +- Use when a traceback mentions torch, CUDA, missing modules, or shared libraries +- Use when Docker or container setup fails +- Use when checkpoint downloads fail or HuggingFace auth errors appear + +## Path convention + +All paths below are relative to this file's location (`.agents/skills/cosmos3-env-troubleshoot/`). + +## Step 1: Match against known errors + +Check the error message against the table below. Each row links to the canonical fix in the docs. + +| Error signature | Cause | Fix location | +| ----------------------------------------------------------------------------- | ------------------------------------ | -------------------------------------------------------------------------------------------------------- | +| `ImportError: cannot import name '_functionalization' from 'torch._C'` | NGC container library conflict | `../../../docs/setup.md` § PyTorch Import Issue — run `export LD_LIBRARY_PATH=''` | +| `ModuleNotFoundError: No module named 'cosmos_framework'` | Package not installed | `../../../docs/setup.md` § Dependency Issue — run `uv sync --all-extras --group=cu130-train --reinstall` | +| `ModuleNotFoundError: No module named ` | Dependency missing | `../../../docs/setup.md` § Dependency Issue — reinstall venv | +| `fatal error: Python.h: No such file or directory` | Broken Python / uv install | `../../../docs/setup.md` § Python Issue — reinstall uv + venv from scratch | +| `OSError: : cannot open shared object file` | CUDA version mismatch | `../../../docs/setup.md` § CUDA Issue — install matching `cuda-toolkit-` | +| `docker: Error response from daemon: unknown or invalid runtime name: nvidia` | Docker nvidia runtime not configured | `../../../docs/setup.md` § Docker Container — run `sudo nvidia-ctk runtime configure --runtime=docker` | +| HuggingFace 401 / download failures | Auth or license not accepted | `../../../docs/setup.md` § Downloading Base Checkpoints — check `HF_TOKEN`, accept license agreement | + +## Step 2: If no documented fix matches, try common remediation + +Run these diagnostic commands to collect information, then attempt fixes in order: + +### Diagnostic commands + +```shell +# System +uname -a +cat /etc/os-release | head -5 + +# Python +python --version +which python + +# CUDA +nvidia-smi +python -c "import torch; print(f'torch={torch.__version__}, cuda={torch.version.cuda}')" + +# Package +uv pip list | head -20 +``` + +### Remediation ladder (try in order) + +1. **Clear library path**: `export LD_LIBRARY_PATH=''` +2. **Reinstall venv**: `uv sync --all-extras --group=cu130-train --reinstall` (or `cu128-train` on older drivers; drop `-train` only if you intentionally want the inference-only group) +3. **Reinstall uv + venv from scratch**: + + ```shell + curl -LsSf https://astral.sh/uv/install.sh | sh + uv python install --reinstall + rm -rf .venv + uv sync --all-extras --group=cu130-train --reinstall + source .venv/bin/activate + ``` + +4. **Check CUDA version alignment**: the major CUDA version from `nvidia-smi` must match `torch.version.cuda` +5. **Try Docker**: if the host environment is too broken, fall back to the Docker container (see `../../../docs/setup.md`) + +## Step 3: If still unresolved, generate a bug report + +If none of the above resolves the issue, collect environment information and present the user with a pre-filled bug report they can submit as a GitHub issue. + +Fill in the template below by running the diagnostic commands and inserting the results: + +````markdown +## Environment + +- **OS**: +- **Python**: +- **CUDA (system)**: +- **CUDA (torch)**: `> +- **torch version**: `> +- **cosmos_framework version**: +- **Installation method**: + +## Error + +``` + +``` + +## What was tried + +1. + +## Additional context + + +```` diff --git a/.agents/skills/cosmos3-inference/SKILL.md b/.agents/skills/cosmos3-inference/SKILL.md new file mode 100644 index 0000000..9419856 --- /dev/null +++ b/.agents/skills/cosmos3-inference/SKILL.md @@ -0,0 +1,59 @@ +--- +name: cosmos3-inference +description: > + Guide users through running Cosmos3 inference — offline batch generation, online + serving with Ray and Gradio, parallelism options, input formats, sampling parameters, + and prompt upsampling. Use when the user asks "how do I run inference", + "how do I generate a video", "how do I serve the model", "what parameters should I use", + or any question about running the model to produce outputs. +--- + +# Cosmos3 Inference + +## When to use this skill + +- Use when a user wants to generate images or videos with Cosmos3 +- Use when a user asks about inference parameters, input formats, or parallelism +- Use when a user wants to set up online serving (Ray Serve, Gradio) +- Use when a user asks about prompt engineering or upsampling +- For environment or import errors, hand off to **cosmos3-env-troubleshoot** + +## Path convention + +All paths below are relative to the cosmos3 package root (`../../../` from this skill file). All `uv run` / `python` commands should also be run from there. + +## Where to find answers + +| User question | Go to | +| ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | +| How do I run inference? (single-GPU, multi-GPU) | `README.md` § Inference | +| Which model should I use? (Nano vs Super, memory, shift) | `README.md` § Models | +| Which modality? (t2i, t2v, i2v, examples) | `README.md` § Modalities | +| What parallelism preset? (latency vs throughput) | `README.md` § Inference | +| What input fields are available? (prompt, vision_path, num_frames, ...) | `docs/inference.md` § Sample Arguments | +| What are the default parameter values? | `cosmos_framework/inference/defaults//sample_args.json` (per-modality JSON) | +| How do I use custom defaults? | `docs/inference.md` § Custom Defaults | +| How do I override a parameter? (precedence) | `docs/faq.md` § How do I override a default parameter? | +| What is the shift parameter? | `docs/faq.md` § What is the `shift` parameter? | +| How many frames can I generate? (resolution caps) | `docs/faq.md` § How many frames can I generate? | +| How do I start Ray Serve / Gradio / submit requests? | `docs/inference_online.md` | +| How do I write good prompts? | `docs/prompting.md` | +| How do I upsample short prompts? | `docs/prompting.md` § Upsampling | +| How do I use the low-level API? (examples/) | `examples/inference.py` (model API) / `examples/inference_pipeline.py` (pipeline API) | +| All CLI flags | `uv run --all-extras --group=cu130 python -m cosmos_framework.scripts.inference --help` | + +## Things not obvious from the docs + +- **Path resolution**: relative paths in input JSON files are resolved relative to the **JSON file's directory**, not the working directory. +- **Seed**: always pass `--seed` for reproducible results. Without it, a random seed is used each time. +- **Resume**: interrupted runs can be resumed by re-running the same command — existing outputs are skipped automatically. +- **`--keep-going`**: continues processing remaining samples after a per-sample failure (e.g. guardrail rejection). Used in online serving by default. +- **Unique names**: every sample in a run must have a unique `name` field, or the script will error. + +## Related skills + +| Skill | When to use | +| -------------------------------------- | ---------------------------------------------- | +| `../cosmos3-setup/SKILL.md` | Installation and environment setup | +| `../cosmos3-codebase-nav/SKILL.md` | Finding files, parameters, and configs in code | +| `../cosmos3-env-troubleshoot/SKILL.md` | Debugging environment and runtime errors | diff --git a/.agents/skills/cosmos3-post-training/SKILL.md b/.agents/skills/cosmos3-post-training/SKILL.md new file mode 100644 index 0000000..26dd167 --- /dev/null +++ b/.agents/skills/cosmos3-post-training/SKILL.md @@ -0,0 +1,103 @@ +--- +name: cosmos3-post-training +description: > + Guide users through Cosmos3 supervised fine-tuning (SFT) post-training: + preparing the example dataset and Wan2.2 VAE, converting the base + checkpoint to DCP, launching distributed training (paired launch shell + recommended, raw `torchrun` as an alternative), running T2V/I2V/V2V + inference with the trained DCP checkpoint, optionally exporting it to + Hugging Face safetensors, and running **action evaluation** + (`cosmos_framework.scripts.eval`) on action checkpoints (forward / inverse + dynamics, policy) for PSNR / action MSE. Use when the user asks how to + post-train Cosmos3, fine-tune on a custom video dataset, export a + trained checkpoint, evaluate an action checkpoint, run `eval.py`, or + invoke one of the recipe launch shells (`launch_sft_action_fdm_nano.sh`, + `launch_sft_action_policy_nano.sh`, `launch_sft_vision_nano.sh`, + `launch_sft_llava_ov.sh`, plus their `_super` LoRA variants) + — or any question about `cu130-train` / `cu128-train`, + `convert_model_to_dcp` / `export_model` / `train` / `eval`, action-eval + metrics, or SFT output paths. `eval.py` is action-only; for T2V/I2V/V2V + use inference. For dataset captioning / JSONL assembly, see + `docs/dataset_jsonl.md`. +--- + +# Cosmos3 Post-Training (SFT) + +## When to use this skill + +- User wants to fine-tune Cosmos3-Nano (or Cosmos3-Super via LoRA) on a Bridge / LIBERO / custom video dataset (SFT) +- User asks which fields in a recipe TOML to override (`[model.parallelism].data_parallel_shard_degree`, `[dataloader_train].max_samples_per_batch`, `[optimizer].lr`, `[trainer].max_iter`, `[checkpoint].load_path`, ...) or which experiment SKU to pick +- User wants to convert a base Hugging Face checkpoint to DCP, or convert a trained DCP back to safetensors +- User wants to score an **action** checkpoint (forward / inverse dynamics, policy) against a held-out dataset with `cosmos_framework.scripts.eval` — per-sample PSNR / action MSE plus an aggregate. Eval is action-only; do not invoke this skill's eval guidance for T2V/I2V/V2V checkpoints +- For installation, `--group=cu130-train` / `cu128-train`, or LD_LIBRARY_PATH issues, hand off to **cosmos3-setup** +- For inference parameters, parallelism presets, or online serving, hand off to **cosmos3-inference** +- For raw-video captioning or assembling a SFT JSONL, see `docs/dataset_jsonl.md` (the captioning flow has moved out of `docs/training.md`) +- For LIBERO closed-loop policy evaluation (simulator-in-the-loop), see `docs/action_policy_closed_loop_eval.md` — separate from `cosmos_framework.scripts.eval` + +## Path convention + +All paths below are relative to the cosmos3 package root (`../../../` from this skill file). All `uv run` / `python` / `torchrun` / `bash` commands should also be run from there. + +## Where to find answers + +The canonical reference is `docs/training.md`. Use this table to route questions: + +| User question | Go to | +| --------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | +| Full step-by-step SFT workflow | `docs/training.md` | +| Which install group? (`cu130-train` vs `cu128-train`) | `docs/setup.md` § CUDA Variants | +| Which recipes exist? (action FDM / Policy / Vision SFT / Reasoner) | `docs/training.md` § Step 1 - Prepare data and config | +| How do I download the example dataset / Wan VAE? | `docs/training.md` § Step 1 - Prepare data and config | +| How do I convert a base HF checkpoint to DCP? | `docs/training.md` § Step 2 — Prepare checkpoint | +| How do I launch training (paired shell, recommended)? | `docs/training.md` § Step 3 → Generator Post-Training → Option A | +| How do I launch training with raw `torchrun`? | `docs/training.md` § Step 3 → Generator Post-Training → Option B | +| How do I override `DATASET_PATH` / `BASE_CHECKPOINT_PATH` / `WAN_VAE_PATH`? | `docs/training.md` § Step 3 → Generator Post-Training → Overriding the defaults | +| Which TOML keys are commonly tuned? | `docs/training.md` § Config | +| LoRA knobs (`lora_enabled` / `lora_rank` / `lora_alpha`) | `docs/training.md` § Config — `[model]` block (VFM only) | +| How do I validate the config without actually training? | `cosmos_framework/scripts/train.py` `--dryrun` flag (not in docs) | +| How do I export the trained DCP back to safetensors? | `docs/training.md` § Export checkpoint to Hugging Face safetensors | +| How do I run inference with the trained checkpoint? | `cosmos3-inference` skill (point at `$RUN_DIR/checkpoints/iter_`) | +| How do I evaluate an action checkpoint (forward/inverse/policy)? | `docs/training.md` § Evaluation | +| How do I run `cosmos_framework.scripts.eval` / `eval.py`? | `docs/training.md` § Evaluation | +| What metrics does action eval report (PSNR, MSE)? | `docs/training.md` § Evaluation | +| Where do training and action-eval artifacts land? | `docs/training.md` § Outputs | +| How do I caption raw videos / build a SFT JSONL? | `docs/dataset_jsonl.md` | +| LIBERO closed-loop policy eval | `docs/action_policy_closed_loop_eval.md` | + +## Workflow at a glance + +1. **Setup** — install the training extras: `uv sync --all-extras --group=cu130-train` (or `cu128-train` on older drivers), then `source .venv/bin/activate && export LD_LIBRARY_PATH=`. +2. **Step 1 - Prepare data and config** — for the recipe you're running, download the HF dataset to `examples/data//` and the Wan2.2 VAE to `examples/checkpoints/wan22_vae/Wan2.2_VAE.pth` via `uvx hf@latest download …`. The Reasoner recipe streams its dataset from HF Hub at startup — no Step 1 download needed. +3. **Step 2 — Prepare checkpoint** — set `BASE_CHECKPOINT_NAME` (`Cosmos3-Nano` or `Cosmos3-Super`, matching the recipe) and run `python -m cosmos_framework.scripts.convert_model_to_dcp -o examples/checkpoints/$BASE_CHECKPOINT_NAME --checkpoint-path $BASE_CHECKPOINT_NAME`. Skip for the Reasoner recipe (the Qwen3-VL backbone is fetched from HF Hub at startup). +4. **Step 3 — Run training (Option A, recommended)** — from the repo root, `bash examples/launch_sft_.sh` (e.g. `launch_sft_action_fdm_nano.sh`). The launcher resolves `DATASET_PATH`, `BASE_CHECKPOINT_PATH`, `WAN_VAE_PATH` from the default `examples/` locations populated by Steps 1+2; export any of them in the shell first to override. +5. **Step 3 — Run training (Option B, raw `torchrun`)** — export the env vars yourself, then `IMAGINAIRE_OUTPUT_ROOT=outputs/train PYTHONPATH=. torchrun --nproc_per_node=8 -m cosmos_framework.scripts.train --sft-toml=examples/toml/sft_config/.toml`. Unlike Option A, raw `torchrun` does NOT auto-resolve the env-var trio from `examples/` — they must come from the shell, or you must hand-edit the TOML to inline literal paths. +6. **Outputs** — `$RUN_DIR = $IMAGINAIRE_OUTPUT_ROOT///`. DCP checkpoints land under `$RUN_DIR/checkpoints/iter_/`; the latest iter name is in `$RUN_DIR/checkpoints/latest_checkpoint.txt`. `$RUN_DIR/config.yaml` next to the checkpoints is what inference + eval consume. +7. **Inference** — point `cosmos_framework.scripts.inference` at `$RUN_DIR/checkpoints/iter_` together with `--config-file $RUN_DIR/config.yaml` (see `cosmos3-inference` skill for presets / input formats). +8. **Action evaluation (action checkpoints only)** — `torchrun --nproc-per-node=8 -m cosmos_framework.scripts.eval -o outputs/train_eval --checkpoint-path $RUN_DIR/model --dataset.config-file $RUN_DIR/config.yaml`. Generates each held-out sample through the inference engine and scores against GT — PSNR for vision modes, MSE for action modes. Per-sample `metrics.json` lives next to each `vision.mp4`; rank-0 aggregate is `outputs/train_eval/metrics_aggregate.json`. Skip for T2V/I2V/V2V checkpoints. +9. **Export (optional)** — `python -m cosmos_framework.scripts.export_model --checkpoint-path $RUN_DIR/checkpoints/$(cat $RUN_DIR/checkpoints/latest_checkpoint.txt) --config-file $RUN_DIR/config.yaml -o $RUN_DIR/model` writes a portable HF safetensors checkpoint to `$RUN_DIR/model`. + +## Things not obvious from the docs + +- **Training extras are a separate group**: SFT requires the `cu130-train` / `cu128-train` install group, not the inference-only `cu130` / `cu128`. Re-running `uv sync` with the wrong group silently leaves training deps uninstalled. +- **Recipe = paired `examples/launch_sft_.sh` + `examples/toml/sft_config/.toml`**: the `.sh` declares `TOML_FILE` directly (full repo-relative path) plus `: "${DATASET_PATH:=…}"` / `: "${BASE_CHECKPOINT_PATH:=…}"` defaults that line up with where Steps 1+2 land, then sources `examples/_sft_launcher_common.sh`. `export`ed values in the user's shell win over the defaults. The helper forwards into `cosmos_framework.scripts.train --sft-toml=$TOML_FILE`, with any `TAIL_OVERRIDES` bash-array entries appended after `--` as Hydra-style `key.path=value` overrides (applied last on top of the pydantic-validated TOML schema in `cosmos_framework/configs/toml_config/sft_config.py`). `MASTER_PORT` defaults to `50012` in the helper; set it in the launcher (or `export`) only if you need to co-launch multiple jobs on one node. +- **Option A (paired launch shell) vs Option B (raw `torchrun`)**: Option A resolves the env-var trio (`DATASET_PATH` / `BASE_CHECKPOINT_PATH` / `WAN_VAE_PATH`) from `examples/` defaults so unset env runs out of the box. Option B requires you to either export those env vars yourself or hand-edit the TOML to inline the paths; the TOMLs use `${ENV:DATASET_PATH}` interpolation that's resolved at TOML load time. +- **`IMAGINAIRE_OUTPUT_ROOT` controls the entire output tree**: setting `IMAGINAIRE_OUTPUT_ROOT=outputs/train` makes everything land under `outputs/train////` (logs, `config.yaml`, `checkpoints/iter_`, callback outputs). Unset, training falls back to `/tmp/imaginaire4-output/...`. +- **W&B is disabled by default**: every recipe TOML sets `[job].wandb_mode = "disabled"`. To log to W&B, flip it to `"online"` in the TOML and export `WANDB_API_KEY` before launching. +- **Inference uses the DCP checkpoint directly**: the standard flow points `cosmos_framework.scripts.inference` at `$RUN_DIR/checkpoints/iter_` together with `--config-file $RUN_DIR/config.yaml`. The Hugging Face safetensors export (`$RUN_DIR/model`) is optional — only needed if you want a portable single-file checkpoint. +- **Parallelism degree must match topology**: in the TOML `[model.parallelism]` block, `data_parallel_shard_degree × data_parallel_replicate_degree × context_parallel_shard_degree` must equal `WORLD_SIZE`. `-1` autoselects `data_parallel_shard_degree` from torchrun world size. Mismatch → FSDP init failure. +- **`--dryrun`**: `cosmos_framework.scripts.train` accepts `--dryrun` to validate the config end-to-end without launching training. Use it whenever iterating on TOML keys or Hydra overrides. +- **`eval.py` is action-only**: `cosmos_framework.scripts.eval` only scores action-mode generations (PSNR for predicted video, MSE for predicted action). It does *not* score T2V/I2V/V2V (vision SFT) checkpoints — those use `cosmos_framework.scripts.inference` (no GT scoring). Pointing `eval.py` at a non-action dataloader fails with "mode requires GT video/action but data_batch had none". +- **Eval flag names are nested under `--dataset.*`**: `--dataset.config-file`, `--dataset.model-mode`, `--dataset.num-samples`, `--dataset.sample-stride`, `--dataset.root-override`, `--dataset.gcs-root-override`, `--dataset.gcs-path-map`, `--dataset.dataset` (key-or-name override). The training docs only call out `--dataset.config-file`, `--dataset.model-mode`, `--dataset.num-samples`; the rest still work. +- **`--dataset.model-mode` defaults to `joint`**: every dataset entry is evaluated under all three action modes — total generation count = `len(modes) × ceil(len(val_split) / sample_stride)`, capped by `--dataset.num-samples`. Restrict during development with `--dataset.num-samples N`, `--dataset.sample-stride K`, or `--dataset.model-mode `. Mode is also encoded in each sample's name (`//`) and is what the metric dispatcher reads back when scoring. +- **Action eval reuses the training dataloader via `--dataset.config-file`**: pointing `eval.py` at `$RUN_DIR/config.yaml` resolves the same dataloader the model was trained against (`val` split by default; falls back to `dataloader_train` when there is no `dataloader_val`). Use `--dataset.root-override /path/to/eval/dataset` to swap in held-out data without editing the config; alternatives are `--dataset.gcs-root-override ` (downloads via `--dataset.cache-dir` / `-c`) and `--dataset.gcs-path-map`. +- **Throughput preset for full-dataset action eval**: `--parallelism-preset` defaults to `latency` (model sharded across all ranks, one sample at a time — required when the checkpoint is too large to fit on a single GPU). When the model fits on one GPU, pass `--parallelism-preset=throughput` so wall-clock scales as `N / num_gpus × per_sample_time` instead of `N × per_sample_time`. Not promoted in `docs/training.md` but still supported. +- **Eval `--checkpoint-path` expects the exported safetensors dir, not the DCP iter**: the command in `docs/training.md` (`--checkpoint-path outputs/train/model`) assumes you ran the export step first. To eval directly off a DCP iter, point `--checkpoint-path` at `$RUN_DIR/checkpoints/iter_` instead. + +## Related skills + +| Skill | When to use | +| -------------------------------------- | ---------------------------------------------------------------------------- | +| `../cosmos3-setup/SKILL.md` | Initial install, CUDA variant selection, container/`LD_LIBRARY_PATH` setup | +| `../cosmos3-inference/SKILL.md` | Inference parameters, parallelism presets, input JSON format, online serving | +| `../cosmos3-codebase-nav/SKILL.md` | Locating configs, scripts, and defaults inside the package | +| `../cosmos3-env-troubleshoot/SKILL.md` | Debugging environment / runtime errors during training | diff --git a/.agents/skills/cosmos3-setup/SKILL.md b/.agents/skills/cosmos3-setup/SKILL.md new file mode 100644 index 0000000..aae4ba9 --- /dev/null +++ b/.agents/skills/cosmos3-setup/SKILL.md @@ -0,0 +1,61 @@ +--- +name: cosmos3-setup +description: > + Guide users through Cosmos3 installation, environment setup, checkpoint downloading, + and verification. Use when the user asks "how do I install cosmos3", "how do I set up + the environment", "how do I download checkpoints", "how do I use Docker", or any + question about getting the package running for the first time. +--- + +# Cosmos3 Setup + +## When to use this skill + +- Use when a user wants to install Cosmos3 or set up a development environment +- Use when a user asks about system requirements, CUDA versions, or GPU compatibility +- Use when a user needs to download model checkpoints or configure HuggingFace auth +- Use when a user wants to run Cosmos3 inside a Docker container or NGC container +- For errors during setup, hand off to the **cosmos3-env-troubleshoot** skill + +## Path convention + +All paths below are relative to the cosmos3 package root (`../../../` from this skill file). All `uv run` / `python` commands should also be run from there. + +## Where to find answers + +The canonical setup reference is `docs/setup.md`. The README (`README.md` § Setup) has the shortest quickstart. + +| User question | Go to | +| ------------------------------------------------------ | -------------------------------------------------------------------- | +| What are the system requirements? | `docs/setup.md` § System Requirements | +| How do I install with uv? (sync, pip venv, pip system) | `docs/setup.md` § Virtual Environment | +| How do I install with Docker? | `docs/setup.md` § Docker Container | +| Custom torch/CUDA versions or attention backends? | `docs/setup.md` § Advanced | +| Which CUDA version? (cu130 vs cu128) | `docs/setup.md` § CUDA Variants, `docs/faq.md` § Which CUDA version? | +| How do I download checkpoints? | `docs/setup.md` § Downloading Base Checkpoints | +| NGC container issues? | `docs/setup.md` § PyTorch Import Issue | +| Any installation error | `../cosmos3-env-troubleshoot/SKILL.md` | + +## Setup steps at a glance + +1. **Clone** the repository and `cd` into the project root (the directory containing `pyproject.toml`) +2. **System deps**: `sudo apt-get install -y --no-install-recommends curl ffmpeg git-lfs libx11-dev tree wget` +3. **Install uv**: `curl -LsSf https://astral.sh/uv/install.sh | sh && source $HOME/.local/bin/env` +4. **Install package**: `uv sync --all-extras --group=cu130-train && source .venv/bin/activate && export LD_LIBRARY_PATH=` (use `cu128-train` on older drivers; the inference-only `cu130` / `cu128` groups omit the training extras) +5. **Checkpoints**: auto-downloaded during inference; requires HuggingFace auth (see docs) +6. **Verify**: `uv run --all-extras --group=cu130-train python -c "import cosmos_framework; print('ok')"` + +## Things not obvious from the docs + +- **NGC container caveat**: you must run `export LD_LIBRARY_PATH=''` *before* any Python imports when inside an NGC PyTorch container. Easy to miss. +- **CUDA version alignment**: the major CUDA version from `nvidia-smi` must match `torch.version.cuda`. Mismatches cause cryptic shared-library errors. +- **`HF_HOME`**: controls where checkpoints are cached (default: `~/.cache/huggingface`). Set this if disk space is tight or you want a shared cache. +- **Conflicting env vars**: stale `HF_TOKEN` or `HUGGING_FACE_HUB_TOKEN` env vars can silently override CLI auth. Check with `printenv | grep HF_`. + +## Related skills + +| Skill | When to use | +| -------------------------------------- | ---------------------------------------------- | +| `../cosmos3-inference/SKILL.md` | Running inference after setup is complete | +| `../cosmos3-codebase-nav/SKILL.md` | Finding files, parameters, and configs in code | +| `../cosmos3-env-troubleshoot/SKILL.md` | Debugging environment and runtime errors | diff --git a/.claude/skills/cosmos3-codebase-nav/SKILL.md b/.claude/skills/cosmos3-codebase-nav/SKILL.md new file mode 100644 index 0000000..b9e708b --- /dev/null +++ b/.claude/skills/cosmos3-codebase-nav/SKILL.md @@ -0,0 +1,122 @@ +--- +name: cosmos3-codebase-nav +description: > + Navigate the Cosmos3 package codebase to find where parameters, configs, defaults, + scripts, and documentation live. Use when the user asks "where is X in cosmos3", + "how do I find the config for Y", "where are the defaults", "where do I change a + parameter", or any question about locating files, modules, or settings. Also use + when the user opens or edits files and needs orientation. +--- + +# Cosmos3 Codebase Navigation + +## When to use this skill + +- Use this skill when an agent is navigating the Cosmos3 package +- Use this skill to answer "where is X", "how do I find the config for Y", or any file-location question +- Use this skill when the user opens or edits cosmos3 files and needs orientation + +## Path convention + +All paths below are relative to this file's location (`.claude/skills/cosmos3-codebase-nav/`). The repo is laid out as: + +- `cosmos_framework/` — main training package (data, model, trainer, callbacks, checkpoint, utils, …). +- `cosmos_framework/configs/base/experiment/` — vfm (generator) experiment SKUs referenced by `[train.train_policy].experiment` in the recipe TOMLs. +- `cosmos_framework/configs/base/vlm/experiment/` — vlm (reasoner) experiment SKUs. +- `cosmos_framework/inference/` — inference subpackage (args, model, inference engine, defaults, Ray serving, common helpers). +- `cosmos_framework/scripts/` — top-level entry-point scripts (train, inference, eval, export_model, convert_model_to_dcp, upsample_prompts, caption_from_video, captions_to_sft_jsonl, action_policy_server, …). Invoked as `python -m cosmos_framework.scripts.`. +- `examples/toml/sft_config/.toml` + `examples/launch_sft_.sh` — paired SFT recipes (training entry-point input). The shell sources `examples/_sft_launcher_common.sh`, which forwards into `cosmos_framework.scripts.train --sft-toml=...`. +- `cosmos_framework/configs/toml_config/` — pydantic schemas (`sft_config.py`) and helpers that validate the recipe TOML at load time. + +## Quick Reference + +### Where parameters and defaults live + +| What you're looking for | File | +| --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | +| Sampling params (num_steps, guidance, shift, fps, etc.) | `../../../cosmos_framework/inference/args.py` → `SamplingArgs`, `SamplingOverrides` | +| Per-modality default values | `../../../cosmos_framework/inference/defaults//sample_args.json` | +| Setup params (parallelism, checkpoints, model path) | `../../../cosmos_framework/inference/args.py` → `OmniSetupArgs`, `OmniSetupOverrides` | +| Common args base classes | `../../../cosmos_framework/inference/common/args.py` → `ArgsBase`, `OverridesBase` | +| Ray serving parallelism presets | `../../../cosmos_framework/inference/ray/configs/latency.yaml`, `../../../cosmos_framework/inference/ray/configs/throughput.yaml` | +| Feature flags | `../../../cosmos_framework/utils/flags.py` | +| Prompt upsampler system prompt | `../../../cosmos_framework/inference/defaults/prompt_upsampler.txt` | +| Video captioner system prompt | `../../../cosmos_framework/inference/defaults/video_captioner.txt` | +| SFT recipe TOMLs (paired with `examples/launch_sft_*.sh`) | `../../../examples/toml/sft_config/.toml` | +| SFT pydantic schema (validates the recipe TOML) | `../../../cosmos_framework/configs/toml_config/sft_config.py` | +| Training experiment SKUs (vfm) | `../../../cosmos_framework/configs/base/experiment/` | +| Training experiment SKUs (vlm / reasoner) | `../../../cosmos_framework/configs/base/vlm/experiment/` | +| Example inputs | `../../../inputs/omni/t2i.json`, `../../../inputs/omni/t2v.json`, `../../../inputs/omni/i2v.json`, … | + +Available modality modes for defaults: `text2image`, `text2video`, `image2video`, `image2image`, `video2video`, `forward_dynamics`, `inverse_dynamics`, `policy`. + +### Config defaults resolution chain + +When a user runs inference, default parameter values are resolved in this order: + +``` +cosmos_framework/inference/defaults//sample_args.json # 1. Per-modality JSON defaults (num_steps, guidance, shift, fps, etc.) + ↓ +_load_modality_defaults() in cosmos_framework/inference/args.py # 2. Loaded and cached at import time + ↓ +SamplingArgs / SamplingOverrides # 3. Pydantic models with field-level validation + ↓ +OmniSampleOverrides.build_sample() # 4. Merges user overrides → final resolved args + ↓ +_RESOLUTION_SHIFT_DEFAULTS[model_size, resolution] # 5. Model+resolution shift override (if user didn't set shift) + ↓ +CLI flags (--guidance, --shift, etc.) # 6. User overrides from command line +``` + +The `_RESOLUTION_SHIFT_DEFAULTS` table in `../../../cosmos_framework/inference/args.py` (on `OmniSampleOverrides`) overrides the default `shift` based on model size and resolution, unless the user explicitly specified `--shift`. + +| Mode | Default file | Key defaults | +| ------------- | --------------------------------------------------------------------------- | ---------------------------------------------- | +| `text2image` | `../../../cosmos_framework/inference/defaults/text2image/sample_args.json` | `num_frames=1`, `guidance=6.0`, `shift=10.0` | +| `text2video` | `../../../cosmos_framework/inference/defaults/text2video/sample_args.json` | `num_frames=189`, `guidance=6.0`, `shift=10.0` | +| `image2video` | `../../../cosmos_framework/inference/defaults/image2video/sample_args.json` | `num_frames=189`, `guidance=6.0`, `shift=10.0` | + +Action and video2video modes also have defaults under `cosmos_framework/inference/defaults/{image2image,video2video,forward_dynamics,inverse_dynamics,policy}/sample_args.json`. + +Users can also supply a custom defaults file per-request via the `defaults_file` field in sample arguments (see `../../../docs/inference.md`). + +### Where to make changes + +| Task | Edit | +| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | +| Change a built-in default value | `../../../cosmos_framework/inference/defaults//sample_args.json` | +| Add a new CLI parameter | `SamplingArgs` + `SamplingOverrides` in `../../../cosmos_framework/inference/args.py`, then add to each `sample_args.json` | +| Change parallelism presets | `../../../cosmos_framework/inference/ray/configs/latency.yaml` or `throughput.yaml` | +| Add a new script | `../../../cosmos_framework/scripts/` — follow `inference.py` as the pattern | + +### Key entry points + +| Entry point | How to run | +| -------------------- | -------------------------------------------------------------------------------------------- | +| Batch inference | `python -m cosmos_framework.scripts.inference` | +| Training | `python -m cosmos_framework.scripts.train --sft-toml=examples/toml/sft_config/.toml` | +| Action evaluation | `python -m cosmos_framework.scripts.eval` | +| Online serving (Ray) | `python -m cosmos_framework.inference.ray.serve` | +| Submit to Ray server | `python -m cosmos_framework.inference.ray.submit` | +| Gradio UI | `python -m cosmos_framework.inference.ray.gradio` | +| Prompt upsampling | `python -m cosmos_framework.scripts.upsample_prompts` | +| Model export (HF) | `python -m cosmos_framework.scripts.export_model` | +| DCP conversion | `python -m cosmos_framework.scripts.convert_model_to_dcp` | +| Diffusers conversion | `python -m cosmos_framework.scripts.convert_model_to_diffusers` | +| Video captioning | `python -m cosmos_framework.scripts.caption_from_video` | +| Captions → SFT JSONL | `python -m cosmos_framework.scripts.captions_to_sft_jsonl` | +| Action policy server | `python -m cosmos_framework.scripts.action_policy_server` | + +### Documentation + +| Doc | Covers | +| ----------------------------------- | ---------------------------------------------------------- | +| `../../../AGENTS.md` | Commands, rules, key file locations (read this first) | +| `../../../README.md` | Overview, quickstart, examples | +| `../../../docs/setup.md` | Installation, environment, checkpoints | +| `../../../docs/code_structure.md` | Repo layout and per-subpackage tour of `cosmos_framework/` | +| `../../../docs/inference.md` | Sample args, default values, custom defaults | +| `../../../docs/inference_online.md` | Ray Serve and Gradio | +| `../../../docs/prompting.md` | Prompt engineering, upsampling | +| `../../../docs/training.md` | SFT / post-training workflow | +| `../../../docs/faq.md` | FAQ, tips, and troubleshooting | diff --git a/.claude/skills/cosmos3-env-troubleshoot/SKILL.md b/.claude/skills/cosmos3-env-troubleshoot/SKILL.md new file mode 100644 index 0000000..7a9b591 --- /dev/null +++ b/.claude/skills/cosmos3-env-troubleshoot/SKILL.md @@ -0,0 +1,106 @@ +--- +name: cosmos3-env-troubleshoot +description: > + Diagnose and fix Cosmos3 environment, installation, and runtime errors. + Use when the user encounters an ImportError, ModuleNotFoundError, CUDA error, + Docker error, checkpoint download failure, or any traceback during setup or inference. +--- + +# Cosmos3 Environment Troubleshooting + +## When to use this skill + +- Use when a user hits an error during installation, environment setup, or first run +- Use when a traceback mentions torch, CUDA, missing modules, or shared libraries +- Use when Docker or container setup fails +- Use when checkpoint downloads fail or HuggingFace auth errors appear + +## Path convention + +All paths below are relative to this file's location (`.claude/skills/cosmos3-env-troubleshoot/`). + +## Step 1: Match against known errors + +Check the error message against the table below. Each row links to the canonical fix in the docs. + +| Error signature | Cause | Fix location | +| ----------------------------------------------------------------------------- | ------------------------------------ | -------------------------------------------------------------------------------------------------------- | +| `ImportError: cannot import name '_functionalization' from 'torch._C'` | NGC container library conflict | `../../../docs/setup.md` § PyTorch Import Issue — run `export LD_LIBRARY_PATH=''` | +| `ModuleNotFoundError: No module named 'cosmos_framework'` | Package not installed | `../../../docs/setup.md` § Dependency Issue — run `uv sync --all-extras --group=cu130-train --reinstall` | +| `ModuleNotFoundError: No module named ` | Dependency missing | `../../../docs/setup.md` § Dependency Issue — reinstall venv | +| `fatal error: Python.h: No such file or directory` | Broken Python / uv install | `../../../docs/setup.md` § Python Issue — reinstall uv + venv from scratch | +| `OSError: : cannot open shared object file` | CUDA version mismatch | `../../../docs/setup.md` § CUDA Issue — install matching `cuda-toolkit-` | +| `docker: Error response from daemon: unknown or invalid runtime name: nvidia` | Docker nvidia runtime not configured | `../../../docs/setup.md` § Docker Container — run `sudo nvidia-ctk runtime configure --runtime=docker` | +| HuggingFace 401 / download failures | Auth or license not accepted | `../../../docs/setup.md` § Downloading Base Checkpoints — check `HF_TOKEN`, accept license agreement | + +## Step 2: If no documented fix matches, try common remediation + +Run these diagnostic commands to collect information, then attempt fixes in order: + +### Diagnostic commands + +```shell +# System +uname -a +cat /etc/os-release | head -5 + +# Python +python --version +which python + +# CUDA +nvidia-smi +python -c "import torch; print(f'torch={torch.__version__}, cuda={torch.version.cuda}')" + +# Package +uv pip list | head -20 +``` + +### Remediation ladder (try in order) + +1. **Clear library path**: `export LD_LIBRARY_PATH=''` +2. **Reinstall venv**: `uv sync --all-extras --group=cu130-train --reinstall` (or `cu128-train` on older drivers; drop `-train` only if you intentionally want the inference-only group) +3. **Reinstall uv + venv from scratch**: + + ```shell + curl -LsSf https://astral.sh/uv/install.sh | sh + uv python install --reinstall + rm -rf .venv + uv sync --all-extras --group=cu130-train --reinstall + source .venv/bin/activate + ``` + +4. **Check CUDA version alignment**: the major CUDA version from `nvidia-smi` must match `torch.version.cuda` +5. **Try Docker**: if the host environment is too broken, fall back to the Docker container (see `../../../docs/setup.md`) + +## Step 3: If still unresolved, generate a bug report + +If none of the above resolves the issue, collect environment information and present the user with a pre-filled bug report they can submit as a GitHub issue. + +Fill in the template below by running the diagnostic commands and inserting the results: + +````markdown +## Environment + +- **OS**: +- **Python**: +- **CUDA (system)**: +- **CUDA (torch)**: `> +- **torch version**: `> +- **cosmos_framework version**: +- **Installation method**: + +## Error + +``` + +``` + +## What was tried + +1. + +## Additional context + + +```` diff --git a/.claude/skills/cosmos3-inference/SKILL.md b/.claude/skills/cosmos3-inference/SKILL.md new file mode 100644 index 0000000..9419856 --- /dev/null +++ b/.claude/skills/cosmos3-inference/SKILL.md @@ -0,0 +1,59 @@ +--- +name: cosmos3-inference +description: > + Guide users through running Cosmos3 inference — offline batch generation, online + serving with Ray and Gradio, parallelism options, input formats, sampling parameters, + and prompt upsampling. Use when the user asks "how do I run inference", + "how do I generate a video", "how do I serve the model", "what parameters should I use", + or any question about running the model to produce outputs. +--- + +# Cosmos3 Inference + +## When to use this skill + +- Use when a user wants to generate images or videos with Cosmos3 +- Use when a user asks about inference parameters, input formats, or parallelism +- Use when a user wants to set up online serving (Ray Serve, Gradio) +- Use when a user asks about prompt engineering or upsampling +- For environment or import errors, hand off to **cosmos3-env-troubleshoot** + +## Path convention + +All paths below are relative to the cosmos3 package root (`../../../` from this skill file). All `uv run` / `python` commands should also be run from there. + +## Where to find answers + +| User question | Go to | +| ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | +| How do I run inference? (single-GPU, multi-GPU) | `README.md` § Inference | +| Which model should I use? (Nano vs Super, memory, shift) | `README.md` § Models | +| Which modality? (t2i, t2v, i2v, examples) | `README.md` § Modalities | +| What parallelism preset? (latency vs throughput) | `README.md` § Inference | +| What input fields are available? (prompt, vision_path, num_frames, ...) | `docs/inference.md` § Sample Arguments | +| What are the default parameter values? | `cosmos_framework/inference/defaults//sample_args.json` (per-modality JSON) | +| How do I use custom defaults? | `docs/inference.md` § Custom Defaults | +| How do I override a parameter? (precedence) | `docs/faq.md` § How do I override a default parameter? | +| What is the shift parameter? | `docs/faq.md` § What is the `shift` parameter? | +| How many frames can I generate? (resolution caps) | `docs/faq.md` § How many frames can I generate? | +| How do I start Ray Serve / Gradio / submit requests? | `docs/inference_online.md` | +| How do I write good prompts? | `docs/prompting.md` | +| How do I upsample short prompts? | `docs/prompting.md` § Upsampling | +| How do I use the low-level API? (examples/) | `examples/inference.py` (model API) / `examples/inference_pipeline.py` (pipeline API) | +| All CLI flags | `uv run --all-extras --group=cu130 python -m cosmos_framework.scripts.inference --help` | + +## Things not obvious from the docs + +- **Path resolution**: relative paths in input JSON files are resolved relative to the **JSON file's directory**, not the working directory. +- **Seed**: always pass `--seed` for reproducible results. Without it, a random seed is used each time. +- **Resume**: interrupted runs can be resumed by re-running the same command — existing outputs are skipped automatically. +- **`--keep-going`**: continues processing remaining samples after a per-sample failure (e.g. guardrail rejection). Used in online serving by default. +- **Unique names**: every sample in a run must have a unique `name` field, or the script will error. + +## Related skills + +| Skill | When to use | +| -------------------------------------- | ---------------------------------------------- | +| `../cosmos3-setup/SKILL.md` | Installation and environment setup | +| `../cosmos3-codebase-nav/SKILL.md` | Finding files, parameters, and configs in code | +| `../cosmos3-env-troubleshoot/SKILL.md` | Debugging environment and runtime errors | diff --git a/.claude/skills/cosmos3-post-training/SKILL.md b/.claude/skills/cosmos3-post-training/SKILL.md new file mode 100644 index 0000000..26dd167 --- /dev/null +++ b/.claude/skills/cosmos3-post-training/SKILL.md @@ -0,0 +1,103 @@ +--- +name: cosmos3-post-training +description: > + Guide users through Cosmos3 supervised fine-tuning (SFT) post-training: + preparing the example dataset and Wan2.2 VAE, converting the base + checkpoint to DCP, launching distributed training (paired launch shell + recommended, raw `torchrun` as an alternative), running T2V/I2V/V2V + inference with the trained DCP checkpoint, optionally exporting it to + Hugging Face safetensors, and running **action evaluation** + (`cosmos_framework.scripts.eval`) on action checkpoints (forward / inverse + dynamics, policy) for PSNR / action MSE. Use when the user asks how to + post-train Cosmos3, fine-tune on a custom video dataset, export a + trained checkpoint, evaluate an action checkpoint, run `eval.py`, or + invoke one of the recipe launch shells (`launch_sft_action_fdm_nano.sh`, + `launch_sft_action_policy_nano.sh`, `launch_sft_vision_nano.sh`, + `launch_sft_llava_ov.sh`, plus their `_super` LoRA variants) + — or any question about `cu130-train` / `cu128-train`, + `convert_model_to_dcp` / `export_model` / `train` / `eval`, action-eval + metrics, or SFT output paths. `eval.py` is action-only; for T2V/I2V/V2V + use inference. For dataset captioning / JSONL assembly, see + `docs/dataset_jsonl.md`. +--- + +# Cosmos3 Post-Training (SFT) + +## When to use this skill + +- User wants to fine-tune Cosmos3-Nano (or Cosmos3-Super via LoRA) on a Bridge / LIBERO / custom video dataset (SFT) +- User asks which fields in a recipe TOML to override (`[model.parallelism].data_parallel_shard_degree`, `[dataloader_train].max_samples_per_batch`, `[optimizer].lr`, `[trainer].max_iter`, `[checkpoint].load_path`, ...) or which experiment SKU to pick +- User wants to convert a base Hugging Face checkpoint to DCP, or convert a trained DCP back to safetensors +- User wants to score an **action** checkpoint (forward / inverse dynamics, policy) against a held-out dataset with `cosmos_framework.scripts.eval` — per-sample PSNR / action MSE plus an aggregate. Eval is action-only; do not invoke this skill's eval guidance for T2V/I2V/V2V checkpoints +- For installation, `--group=cu130-train` / `cu128-train`, or LD_LIBRARY_PATH issues, hand off to **cosmos3-setup** +- For inference parameters, parallelism presets, or online serving, hand off to **cosmos3-inference** +- For raw-video captioning or assembling a SFT JSONL, see `docs/dataset_jsonl.md` (the captioning flow has moved out of `docs/training.md`) +- For LIBERO closed-loop policy evaluation (simulator-in-the-loop), see `docs/action_policy_closed_loop_eval.md` — separate from `cosmos_framework.scripts.eval` + +## Path convention + +All paths below are relative to the cosmos3 package root (`../../../` from this skill file). All `uv run` / `python` / `torchrun` / `bash` commands should also be run from there. + +## Where to find answers + +The canonical reference is `docs/training.md`. Use this table to route questions: + +| User question | Go to | +| --------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | +| Full step-by-step SFT workflow | `docs/training.md` | +| Which install group? (`cu130-train` vs `cu128-train`) | `docs/setup.md` § CUDA Variants | +| Which recipes exist? (action FDM / Policy / Vision SFT / Reasoner) | `docs/training.md` § Step 1 - Prepare data and config | +| How do I download the example dataset / Wan VAE? | `docs/training.md` § Step 1 - Prepare data and config | +| How do I convert a base HF checkpoint to DCP? | `docs/training.md` § Step 2 — Prepare checkpoint | +| How do I launch training (paired shell, recommended)? | `docs/training.md` § Step 3 → Generator Post-Training → Option A | +| How do I launch training with raw `torchrun`? | `docs/training.md` § Step 3 → Generator Post-Training → Option B | +| How do I override `DATASET_PATH` / `BASE_CHECKPOINT_PATH` / `WAN_VAE_PATH`? | `docs/training.md` § Step 3 → Generator Post-Training → Overriding the defaults | +| Which TOML keys are commonly tuned? | `docs/training.md` § Config | +| LoRA knobs (`lora_enabled` / `lora_rank` / `lora_alpha`) | `docs/training.md` § Config — `[model]` block (VFM only) | +| How do I validate the config without actually training? | `cosmos_framework/scripts/train.py` `--dryrun` flag (not in docs) | +| How do I export the trained DCP back to safetensors? | `docs/training.md` § Export checkpoint to Hugging Face safetensors | +| How do I run inference with the trained checkpoint? | `cosmos3-inference` skill (point at `$RUN_DIR/checkpoints/iter_`) | +| How do I evaluate an action checkpoint (forward/inverse/policy)? | `docs/training.md` § Evaluation | +| How do I run `cosmos_framework.scripts.eval` / `eval.py`? | `docs/training.md` § Evaluation | +| What metrics does action eval report (PSNR, MSE)? | `docs/training.md` § Evaluation | +| Where do training and action-eval artifacts land? | `docs/training.md` § Outputs | +| How do I caption raw videos / build a SFT JSONL? | `docs/dataset_jsonl.md` | +| LIBERO closed-loop policy eval | `docs/action_policy_closed_loop_eval.md` | + +## Workflow at a glance + +1. **Setup** — install the training extras: `uv sync --all-extras --group=cu130-train` (or `cu128-train` on older drivers), then `source .venv/bin/activate && export LD_LIBRARY_PATH=`. +2. **Step 1 - Prepare data and config** — for the recipe you're running, download the HF dataset to `examples/data//` and the Wan2.2 VAE to `examples/checkpoints/wan22_vae/Wan2.2_VAE.pth` via `uvx hf@latest download …`. The Reasoner recipe streams its dataset from HF Hub at startup — no Step 1 download needed. +3. **Step 2 — Prepare checkpoint** — set `BASE_CHECKPOINT_NAME` (`Cosmos3-Nano` or `Cosmos3-Super`, matching the recipe) and run `python -m cosmos_framework.scripts.convert_model_to_dcp -o examples/checkpoints/$BASE_CHECKPOINT_NAME --checkpoint-path $BASE_CHECKPOINT_NAME`. Skip for the Reasoner recipe (the Qwen3-VL backbone is fetched from HF Hub at startup). +4. **Step 3 — Run training (Option A, recommended)** — from the repo root, `bash examples/launch_sft_.sh` (e.g. `launch_sft_action_fdm_nano.sh`). The launcher resolves `DATASET_PATH`, `BASE_CHECKPOINT_PATH`, `WAN_VAE_PATH` from the default `examples/` locations populated by Steps 1+2; export any of them in the shell first to override. +5. **Step 3 — Run training (Option B, raw `torchrun`)** — export the env vars yourself, then `IMAGINAIRE_OUTPUT_ROOT=outputs/train PYTHONPATH=. torchrun --nproc_per_node=8 -m cosmos_framework.scripts.train --sft-toml=examples/toml/sft_config/.toml`. Unlike Option A, raw `torchrun` does NOT auto-resolve the env-var trio from `examples/` — they must come from the shell, or you must hand-edit the TOML to inline literal paths. +6. **Outputs** — `$RUN_DIR = $IMAGINAIRE_OUTPUT_ROOT///`. DCP checkpoints land under `$RUN_DIR/checkpoints/iter_/`; the latest iter name is in `$RUN_DIR/checkpoints/latest_checkpoint.txt`. `$RUN_DIR/config.yaml` next to the checkpoints is what inference + eval consume. +7. **Inference** — point `cosmos_framework.scripts.inference` at `$RUN_DIR/checkpoints/iter_` together with `--config-file $RUN_DIR/config.yaml` (see `cosmos3-inference` skill for presets / input formats). +8. **Action evaluation (action checkpoints only)** — `torchrun --nproc-per-node=8 -m cosmos_framework.scripts.eval -o outputs/train_eval --checkpoint-path $RUN_DIR/model --dataset.config-file $RUN_DIR/config.yaml`. Generates each held-out sample through the inference engine and scores against GT — PSNR for vision modes, MSE for action modes. Per-sample `metrics.json` lives next to each `vision.mp4`; rank-0 aggregate is `outputs/train_eval/metrics_aggregate.json`. Skip for T2V/I2V/V2V checkpoints. +9. **Export (optional)** — `python -m cosmos_framework.scripts.export_model --checkpoint-path $RUN_DIR/checkpoints/$(cat $RUN_DIR/checkpoints/latest_checkpoint.txt) --config-file $RUN_DIR/config.yaml -o $RUN_DIR/model` writes a portable HF safetensors checkpoint to `$RUN_DIR/model`. + +## Things not obvious from the docs + +- **Training extras are a separate group**: SFT requires the `cu130-train` / `cu128-train` install group, not the inference-only `cu130` / `cu128`. Re-running `uv sync` with the wrong group silently leaves training deps uninstalled. +- **Recipe = paired `examples/launch_sft_.sh` + `examples/toml/sft_config/.toml`**: the `.sh` declares `TOML_FILE` directly (full repo-relative path) plus `: "${DATASET_PATH:=…}"` / `: "${BASE_CHECKPOINT_PATH:=…}"` defaults that line up with where Steps 1+2 land, then sources `examples/_sft_launcher_common.sh`. `export`ed values in the user's shell win over the defaults. The helper forwards into `cosmos_framework.scripts.train --sft-toml=$TOML_FILE`, with any `TAIL_OVERRIDES` bash-array entries appended after `--` as Hydra-style `key.path=value` overrides (applied last on top of the pydantic-validated TOML schema in `cosmos_framework/configs/toml_config/sft_config.py`). `MASTER_PORT` defaults to `50012` in the helper; set it in the launcher (or `export`) only if you need to co-launch multiple jobs on one node. +- **Option A (paired launch shell) vs Option B (raw `torchrun`)**: Option A resolves the env-var trio (`DATASET_PATH` / `BASE_CHECKPOINT_PATH` / `WAN_VAE_PATH`) from `examples/` defaults so unset env runs out of the box. Option B requires you to either export those env vars yourself or hand-edit the TOML to inline the paths; the TOMLs use `${ENV:DATASET_PATH}` interpolation that's resolved at TOML load time. +- **`IMAGINAIRE_OUTPUT_ROOT` controls the entire output tree**: setting `IMAGINAIRE_OUTPUT_ROOT=outputs/train` makes everything land under `outputs/train////` (logs, `config.yaml`, `checkpoints/iter_`, callback outputs). Unset, training falls back to `/tmp/imaginaire4-output/...`. +- **W&B is disabled by default**: every recipe TOML sets `[job].wandb_mode = "disabled"`. To log to W&B, flip it to `"online"` in the TOML and export `WANDB_API_KEY` before launching. +- **Inference uses the DCP checkpoint directly**: the standard flow points `cosmos_framework.scripts.inference` at `$RUN_DIR/checkpoints/iter_` together with `--config-file $RUN_DIR/config.yaml`. The Hugging Face safetensors export (`$RUN_DIR/model`) is optional — only needed if you want a portable single-file checkpoint. +- **Parallelism degree must match topology**: in the TOML `[model.parallelism]` block, `data_parallel_shard_degree × data_parallel_replicate_degree × context_parallel_shard_degree` must equal `WORLD_SIZE`. `-1` autoselects `data_parallel_shard_degree` from torchrun world size. Mismatch → FSDP init failure. +- **`--dryrun`**: `cosmos_framework.scripts.train` accepts `--dryrun` to validate the config end-to-end without launching training. Use it whenever iterating on TOML keys or Hydra overrides. +- **`eval.py` is action-only**: `cosmos_framework.scripts.eval` only scores action-mode generations (PSNR for predicted video, MSE for predicted action). It does *not* score T2V/I2V/V2V (vision SFT) checkpoints — those use `cosmos_framework.scripts.inference` (no GT scoring). Pointing `eval.py` at a non-action dataloader fails with "mode requires GT video/action but data_batch had none". +- **Eval flag names are nested under `--dataset.*`**: `--dataset.config-file`, `--dataset.model-mode`, `--dataset.num-samples`, `--dataset.sample-stride`, `--dataset.root-override`, `--dataset.gcs-root-override`, `--dataset.gcs-path-map`, `--dataset.dataset` (key-or-name override). The training docs only call out `--dataset.config-file`, `--dataset.model-mode`, `--dataset.num-samples`; the rest still work. +- **`--dataset.model-mode` defaults to `joint`**: every dataset entry is evaluated under all three action modes — total generation count = `len(modes) × ceil(len(val_split) / sample_stride)`, capped by `--dataset.num-samples`. Restrict during development with `--dataset.num-samples N`, `--dataset.sample-stride K`, or `--dataset.model-mode `. Mode is also encoded in each sample's name (`//`) and is what the metric dispatcher reads back when scoring. +- **Action eval reuses the training dataloader via `--dataset.config-file`**: pointing `eval.py` at `$RUN_DIR/config.yaml` resolves the same dataloader the model was trained against (`val` split by default; falls back to `dataloader_train` when there is no `dataloader_val`). Use `--dataset.root-override /path/to/eval/dataset` to swap in held-out data without editing the config; alternatives are `--dataset.gcs-root-override ` (downloads via `--dataset.cache-dir` / `-c`) and `--dataset.gcs-path-map`. +- **Throughput preset for full-dataset action eval**: `--parallelism-preset` defaults to `latency` (model sharded across all ranks, one sample at a time — required when the checkpoint is too large to fit on a single GPU). When the model fits on one GPU, pass `--parallelism-preset=throughput` so wall-clock scales as `N / num_gpus × per_sample_time` instead of `N × per_sample_time`. Not promoted in `docs/training.md` but still supported. +- **Eval `--checkpoint-path` expects the exported safetensors dir, not the DCP iter**: the command in `docs/training.md` (`--checkpoint-path outputs/train/model`) assumes you ran the export step first. To eval directly off a DCP iter, point `--checkpoint-path` at `$RUN_DIR/checkpoints/iter_` instead. + +## Related skills + +| Skill | When to use | +| -------------------------------------- | ---------------------------------------------------------------------------- | +| `../cosmos3-setup/SKILL.md` | Initial install, CUDA variant selection, container/`LD_LIBRARY_PATH` setup | +| `../cosmos3-inference/SKILL.md` | Inference parameters, parallelism presets, input JSON format, online serving | +| `../cosmos3-codebase-nav/SKILL.md` | Locating configs, scripts, and defaults inside the package | +| `../cosmos3-env-troubleshoot/SKILL.md` | Debugging environment / runtime errors during training | diff --git a/.claude/skills/cosmos3-setup/SKILL.md b/.claude/skills/cosmos3-setup/SKILL.md new file mode 100644 index 0000000..aae4ba9 --- /dev/null +++ b/.claude/skills/cosmos3-setup/SKILL.md @@ -0,0 +1,61 @@ +--- +name: cosmos3-setup +description: > + Guide users through Cosmos3 installation, environment setup, checkpoint downloading, + and verification. Use when the user asks "how do I install cosmos3", "how do I set up + the environment", "how do I download checkpoints", "how do I use Docker", or any + question about getting the package running for the first time. +--- + +# Cosmos3 Setup + +## When to use this skill + +- Use when a user wants to install Cosmos3 or set up a development environment +- Use when a user asks about system requirements, CUDA versions, or GPU compatibility +- Use when a user needs to download model checkpoints or configure HuggingFace auth +- Use when a user wants to run Cosmos3 inside a Docker container or NGC container +- For errors during setup, hand off to the **cosmos3-env-troubleshoot** skill + +## Path convention + +All paths below are relative to the cosmos3 package root (`../../../` from this skill file). All `uv run` / `python` commands should also be run from there. + +## Where to find answers + +The canonical setup reference is `docs/setup.md`. The README (`README.md` § Setup) has the shortest quickstart. + +| User question | Go to | +| ------------------------------------------------------ | -------------------------------------------------------------------- | +| What are the system requirements? | `docs/setup.md` § System Requirements | +| How do I install with uv? (sync, pip venv, pip system) | `docs/setup.md` § Virtual Environment | +| How do I install with Docker? | `docs/setup.md` § Docker Container | +| Custom torch/CUDA versions or attention backends? | `docs/setup.md` § Advanced | +| Which CUDA version? (cu130 vs cu128) | `docs/setup.md` § CUDA Variants, `docs/faq.md` § Which CUDA version? | +| How do I download checkpoints? | `docs/setup.md` § Downloading Base Checkpoints | +| NGC container issues? | `docs/setup.md` § PyTorch Import Issue | +| Any installation error | `../cosmos3-env-troubleshoot/SKILL.md` | + +## Setup steps at a glance + +1. **Clone** the repository and `cd` into the project root (the directory containing `pyproject.toml`) +2. **System deps**: `sudo apt-get install -y --no-install-recommends curl ffmpeg git-lfs libx11-dev tree wget` +3. **Install uv**: `curl -LsSf https://astral.sh/uv/install.sh | sh && source $HOME/.local/bin/env` +4. **Install package**: `uv sync --all-extras --group=cu130-train && source .venv/bin/activate && export LD_LIBRARY_PATH=` (use `cu128-train` on older drivers; the inference-only `cu130` / `cu128` groups omit the training extras) +5. **Checkpoints**: auto-downloaded during inference; requires HuggingFace auth (see docs) +6. **Verify**: `uv run --all-extras --group=cu130-train python -c "import cosmos_framework; print('ok')"` + +## Things not obvious from the docs + +- **NGC container caveat**: you must run `export LD_LIBRARY_PATH=''` *before* any Python imports when inside an NGC PyTorch container. Easy to miss. +- **CUDA version alignment**: the major CUDA version from `nvidia-smi` must match `torch.version.cuda`. Mismatches cause cryptic shared-library errors. +- **`HF_HOME`**: controls where checkpoints are cached (default: `~/.cache/huggingface`). Set this if disk space is tight or you want a shared cache. +- **Conflicting env vars**: stale `HF_TOKEN` or `HUGGING_FACE_HUB_TOKEN` env vars can silently override CLI auth. Check with `printenv | grep HF_`. + +## Related skills + +| Skill | When to use | +| -------------------------------------- | ---------------------------------------------- | +| `../cosmos3-inference/SKILL.md` | Running inference after setup is complete | +| `../cosmos3-codebase-nav/SKILL.md` | Finding files, parameters, and configs in code | +| `../cosmos3-env-troubleshoot/SKILL.md` | Debugging environment and runtime errors | diff --git a/.config/rumdl.toml b/.config/rumdl.toml new file mode 100644 index 0000000..96cee39 --- /dev/null +++ b/.config/rumdl.toml @@ -0,0 +1,31 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# https://rumdl.dev/global-settings/ +[global] +flavor = "standard" +exclude = [ + "ATTRIBUTIONS.md", + "_src", +] +disable = [ + "MD013", # line-length + "MD033", # inline-html + "MD040", # fenced-code-language +] + +# https://rumdl.dev/rules/ + +[per-file-ignores] +"README.md" = [ + "MD041" # first-line-heading +] + +# ul-style +[MD004] +style = "dash" + +# table-format +[MD060] +enabled = true +style = "aligned" diff --git a/.coveragerc b/.coveragerc new file mode 100644 index 0000000..93b0d6e --- /dev/null +++ b/.coveragerc @@ -0,0 +1,32 @@ +# https://coverage.readthedocs.io/en/latest/subprocess.html + +[run] +data_file = outputs/coverage/coverage +disable_warnings = + module-not-imported + no-data-collected +parallel = True +patch = subprocess + +[report] +exclude_lines = + @overload + def __repr__ + if __name__ == .__main__.: + if TYPE_CHECKING: + pragma: no cover + raise AssertionError + raise NotImplementedError +omit = + *_test.py +skip_empty = True +show_missing = True + +[html] +directory = outputs/coverage/html + +[json] +output = outputs/coverage/coverage.json + +[xml] +output = outputs/coverage/coverage.xml diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..0dfe444 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,8 @@ +.venv +.git +/checkpoints +/datasets +/output +/examples/**/checkpoints +/examples/**/output +/examples/**/datasets diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..dba17f5 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,25 @@ +*.lock linguist-generated=true +tests/data/** linguist-generated=true +ATTRIBUTIONS.md linguist-generated=true + +assets/** filter=lfs diff=lfs merge=lfs -text + +# Video files +*.mp4 filter=lfs diff=lfs merge=lfs -text +*.avi filter=lfs diff=lfs merge=lfs -text +*.mov filter=lfs diff=lfs merge=lfs -text +*.mkv filter=lfs diff=lfs merge=lfs -text +*.webm filter=lfs diff=lfs merge=lfs -text + +# Audio files +*.wav filter=lfs diff=lfs merge=lfs -text +*.mp3 filter=lfs diff=lfs merge=lfs -text +*.flac filter=lfs diff=lfs merge=lfs -text +*.aac filter=lfs diff=lfs merge=lfs -text + +# Image files +*.jpg filter=lfs diff=lfs merge=lfs -text +*.jpeg filter=lfs diff=lfs merge=lfs -text +*.png filter=lfs diff=lfs merge=lfs -text +*.tiff filter=lfs diff=lfs merge=lfs -text +*.bmp filter=lfs diff=lfs merge=lfs -text diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 0000000..2ee4d0e --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,65 @@ +--- +name: Bug Report +about: Report a reproducible bug or unexpected behavior +title: "[BUG] " +labels: 'bug' +assignees: + - spectralflight + - jeanachoi + +--- + +## Bug Description + + + +## Reproduction Steps + +```bash +# Minimal command or script to reproduce +``` + +**Reproducibility:** + +- [ ] Always +- [ ] Intermittently (~___% of the time) +- [ ] Only once + +## Expected vs. Actual Behavior + +| | Description | +| ------------ | --------------------------- | +| **Expected** | What you expected to happen | +| **Actual** | What actually happened | + +## Outputs + +
+Error / Stack Trace + + + +
+ +
+Log Files + + + +
+ +## System Information + +| Field | Value | +| ---------------------------- | ------------------------------------------- | +| **Environment** | | +| **Hardware** | | +| **OS** | | +| **GPU Driver** | | +| **CUDA Version** | | +| **Python Version** | | +| **Package Version / Commit** | | + +## Additional Context + + diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml new file mode 100644 index 0000000..e5794dd --- /dev/null +++ b/.github/workflows/pre-commit.yml @@ -0,0 +1,19 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +name: Pre-commit +on: + pull_request: + push: + branches: [main] +jobs: + pre-commit: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v6 + with: + lfs: true + - uses: actions/setup-python@v6 + - uses: astral-sh/setup-uv@v7 + - run: uvx pre-commit@4.5.1 run -a -c ci/.pre-commit-config-base.yaml + - run: uvx pre-commit@4.5.1 run -a diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..2592e48 --- /dev/null +++ b/.gitignore @@ -0,0 +1,220 @@ +/assets +/credentials +/datasets +/outputs +/tmp +.cuda-name +*.env + +# Default targets for docs/training.md Step 1 (data + VAE download) and +# Step 2 (DCP conversion). Multi-GB; not for tracking. +/examples/data/ +/examples/checkpoints/ + +# Local training output (generated, large). +training_output/ + +# Release-tool metadata (regenerated on every release run). +cosmos_training_meta/ + +# Editor / OS noise. +*.swp +.DS_Store + +# ------------------------ BELOW IS AUTO-GENERATED FOR PYTHON REPOS ------------------------ + +# Byte-compiled / optimized / DLL files +__pycache__/ +*.py[cod] +*$py.class + +# C extensions +*.so + +# Distribution / packaging +.Python +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +share/python-wheels/ +*.egg-info/ +.installed.cfg +*.egg +MANIFEST + +# PyInstaller +# Usually these files are written by a python script from a template +# before PyInstaller builds the exe, so as to inject date/other infos into it. +*.manifest +*.spec + +# Installer logs +pip-log.txt +pip-delete-this-directory.txt + +# Unit test / coverage reports +htmlcov/ +.tox/ +.nox/ +.coverage +.coverage.* +.cache +nosetests.xml +coverage.xml +*.cover +*.py,cover +.hypothesis/ +.pytest_cache/ +cover/ + +# Translations +*.mo +*.pot + +# Django stuff: +*.log +local_settings.py +db.sqlite3 +db.sqlite3-journal + +# Flask stuff: +instance/ +.webassets-cache + +# Scrapy stuff: +.scrapy + +# Sphinx documentation +docs/_build/ + +# PyBuilder +.pybuilder/ +target/ + +# Jupyter Notebook +.ipynb_checkpoints + +# IPython +profile_default/ +ipython_config.py + +# pyenv +# For a library or package, you might want to ignore these files since the code is +# intended to run in multiple environments; otherwise, check them in: +# .python-version + +# pipenv +# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. +# However, in case of collaboration, if having platform-specific dependencies or dependencies +# having no cross-platform support, pipenv may install dependencies that don't work, or not +# install all needed dependencies. +#Pipfile.lock + +# UV +# Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control. +# This is especially recommended for binary packages to ensure reproducibility, and is more +# commonly ignored for libraries. +#uv.lock + +# poetry +# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. +# This is especially recommended for binary packages to ensure reproducibility, and is more +# commonly ignored for libraries. +# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control +#poetry.lock +#poetry.toml + +# pdm +# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. +#pdm.lock +# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it +# in version control. +# https://pdm.fming.dev/latest/usage/project/#working-with-version-control +.pdm.toml +.pdm-python +.pdm-build/ + +# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm +__pypackages__/ + +# Celery stuff +celerybeat-schedule +celerybeat.pid + +# SageMath parsed files +*.sage.py + +# Environments +.env +.venv +env/ +venv/ +ENV/ +env.bak/ +venv.bak/ + +# Spyder project settings +.spyderproject +.spyproject + +# Rope project settings +.ropeproject + +# mkdocs documentation +/site + +# mypy +.mypy_cache/ +.dmypy.json +dmypy.json + +# Pyre type checker +.pyre/ + +# pytype static type analyzer +.pytype/ + +# Cython debug symbols +cython_debug/ + +# PyCharm +# JetBrains specific template is maintained in a separate JetBrains.gitignore that can +# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore +# and can be added to the global gitignore or merged into this file. For a more nuclear +# option (not recommended) you can uncomment the following to ignore the entire idea folder. +#.idea/ + +# Abstra +# Abstra is an AI-powered process automation framework. +# Ignore directories containing user credentials, local state, and settings. +# Learn more at https://abstra.io/docs +.abstra/ + +# Visual Studio Code +# Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore +# that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore +# and can be added to the global gitignore or merged into this file. However, if you prefer, +# you could uncomment the following to ignore the entire vscode folder +# .vscode/ + +# Ruff stuff: +.ruff_cache/ + +# PyPI configuration file +.pypirc + +# Cursor +# Cursor is an AI-powered code editor. `.cursorignore` specifies files/directories to +# exclude from AI features like autocomplete and code analysis. Recommended for sensitive data +# refer to https://docs.cursor.com/context/ignore-files +.cursorignore +.cursorindexingignore diff --git a/.gitleaks.toml b/.gitleaks.toml new file mode 100644 index 0000000..59befff --- /dev/null +++ b/.gitleaks.toml @@ -0,0 +1,7 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +[[allowlists]] +regexes = [ + '''Qwen3MoeForCausalLM''' +] diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..31460cc --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,55 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 +default_language_version: + node: 25.2.1 + python: python3.13 +exclude: (?x)( + ^tests/data/ + ) +repos: + - repo: https://github.com/google/addlicense + rev: v1.2.0 + hooks: + - id: addlicense + args: ["-f", "ci/license.txt"] + - repo: https://github.com/jsh9/markdown-toc-creator + rev: 0.1.3 + hooks: + - id: markdown-toc-creator + args: ["--config=ci/.markdown-toc-creator.toml"] + - repo: https://github.com/rvben/rumdl-pre-commit + rev: v0.1.62 + hooks: + - id: rumdl-fmt + - repo: https://github.com/pre-commit/pre-commit-hooks + rev: v6.0.0 + hooks: + - id: check-symlinks + - id: check-executables-have-shebangs + exclude: /_src/ + - id: check-shebang-scripts-are-executable + exclude: /_src/ + - repo: local + hooks: + - id: uv-lock + name: Generate uv lock files for projects + entry: ./ci/uv_lock.sh + language: script + files: pyproject\.toml$ + - id: uv-lock-script + name: Generate uv lock files for scripts + entry: ./ci/uv_lock_script.sh + language: script + types: [python] + - repo: https://github.com/tcort/markdown-link-check + rev: v3.14.2 + hooks: + - alias: link-check + name: link check + id: markdown-link-check + args: [--config, "ci/.link-check.json", --quiet] + stages: [manual] + exclude: (?x)( + \bATTRIBUTIONS\b| + /_src/ + ) diff --git a/.pytest.toml b/.pytest.toml new file mode 100644 index 0000000..0b32661 --- /dev/null +++ b/.pytest.toml @@ -0,0 +1,35 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +[pytest] +python_files = [ + "*_test.py", +] +norecursedirs = [ + "_src", + "imaginaire", + "packages", + "projects", +] +addopts = [ + "--suppress-no-test-exit-code", +] +filterwarnings = [ + "ignore::DeprecationWarning", + "ignore::FutureWarning", +] +markers = [ + "manual: Test requires --manual.", + "level(l): Test level in [0, 1, 2].", + "gpus(n): Test requires GPUs.", +] + +[pytest_env] +COSMOS_VERBOSE = { value = "0", skip_if_set = true } +CUDA_VISIBLE_DEVICES = { unset = true } +PYTORCH_CUDA_ALLOC_CONF = "expandable_segments:True" # Reduce chance of OOM errors +# Limit threading to reduce contention +MKL_NUM_THREADS = "1" +NUMEXPR_NUM_THREADS = "1" +OMP_NUM_THREADS = "1" +OPENBLAS_NUM_THREADS = "1" diff --git a/.python-version b/.python-version new file mode 100644 index 0000000..24ee5b1 --- /dev/null +++ b/.python-version @@ -0,0 +1 @@ +3.13 diff --git a/.ruff.toml b/.ruff.toml new file mode 100644 index 0000000..a2f7681 --- /dev/null +++ b/.ruff.toml @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +line-length = 120 +target-version = "py310" + +[lint] +select = [ + "E", # pycodestyle errors + "F", # pyflakes + "I", # isort + "TID252", # relative-imports + "T10", # debugger +] +ignore = [ + "E402", # module-import-not-at-top-of-file + "E501", # line-too-long + "E721", # type-comparison + "E741", # ambiguous-variable-name + "F541", # f-string-missing-placeholders + "F811", # redefined-while-unused + "F841", # unused-variable +] +fixable = ["ALL"] diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..1fd13e7 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,108 @@ +# AGENTS.md — Cosmos-Framework + +Read this file first — it is the canonical map for navigating the Cosmos repository and stays up to date. + +**Cosmos** is a framework for training and serving world foundation models. Everything lives in a single top-level `cosmos_framework/` Python package: + +- **Training infrastructure** — top-level subpackages under `cosmos_framework/` (data, model, trainer, callbacks, checkpoint, …). +- **Inference infrastructure** — `cosmos_framework/inference/` (Diffusers / Transformers / vLLM-friendly inference core, online serving via Ray + Gradio). +- **Backend packages** — `packages/{diffusers,transformers,vllm}-cosmos3/` provide library-style shims that load Cosmos3 checkpoints into the respective ecosystems. +- **Entry-point scripts** — `cosmos_framework/scripts/` (`train.py`, `inference.py`, `eval.py`, `export_model.py`, …) invoked as `python -m cosmos_framework.scripts.`. Primary training entry point: `cosmos_framework.scripts.train` driven by a structured, pydantic-validated TOML interface (`--sft-toml=`); the schema lives at [`cosmos_framework/configs/toml_config/sft_config.py`](./cosmos_framework/configs/toml_config/sft_config.py) and the canonical recipe pattern is documented in [`examples/README.md`](./examples/README.md). + +> All paths below are relative to the repository root (the directory containing `pyproject.toml`, the `cosmos_framework/` Python package, and `packages/`). + +## Commands + +| Task | Command | +| ---------------------- | --------------------------------------------------- | +| Lint | `uv run ruff check .` | +| Format check | `uv run ruff format --check .` | +| Auto-fix lint + format | `uv run ruff check --fix . && uv run ruff format .` | +| Type-check | `uv run pyrefly check` | +| Test (all) | `uv run pytest` | +| Test (single file) | `uv run pytest --capture=no ` | + +Config files: `.ruff.toml` (ruff), `pyrefly.toml` (pyrefly), `.pytest.toml` (pytest), `conftest.py` (pytest fixtures). + +A `justfile` is provided at the root with longer recipes (`just install`, `just lint`, `just test`, `just docker-cu130`). + +## Rules + +- Always answer questions with references to code or documentation in `file:line` format. +- When unsure, point the user to the closest doc rather than guessing. +- Keep this file short. Link out to skills and docs for detail — this file is included in every prompt. +- Inference code belongs under `cosmos_framework/inference/`; training infrastructure belongs under the other `cosmos_framework/` subpackages. Don't blur the two — if you find yourself adding training-time imports inside `cosmos_framework/inference/` (or vice versa), reconsider. + +## Key File Locations + +### Training (`cosmos_framework/`) + +| What | Where | +| ---------------------------------------------------- | ----------------------------------------------- | +| Algorithms (losses, RL, reward) | `cosmos_framework/algorithm/{loss,reward,rl}` | +| Training loop | `cosmos_framework/trainer/` | +| Models + parallelism | `cosmos_framework/model/` | +| Datasets / data loading | `cosmos_framework/data/` | +| Checkpoint I/O | `cosmos_framework/checkpoint/` | +| Callbacks (logging, eval) | `cosmos_framework/callbacks/` | +| RL workers (rollout, reward, reference, simulations) | `cosmos_framework/workers/` | +| Controller / orchestrator | `cosmos_framework/controller/` | +| Launchers (Slurm, torchrun, k8s) | `cosmos_framework/launcher/` | +| Evaluation harness | `cosmos_framework/evaluation/` | +| CLI tools | `cosmos_framework/tools/`, `tools/` (repo root) | + +For a per-subpackage tour with descriptions, see [`docs/code_structure.md`](./docs/code_structure.md). + +### Inference (`cosmos_framework/inference/`) + +| What | Where | +| ------------------------ | -------------------------------------------------------------------------------- | +| CLI entry point | `cosmos_framework/scripts/inference.py` | +| Args / param definitions | `cosmos_framework/inference/args.py` | +| Per-modality defaults | `cosmos_framework/inference/defaults//sample_args.json` | +| Model / inference core | `cosmos_framework/inference/model.py`, `cosmos_framework/inference/inference.py` | +| Ray serving | `cosmos_framework/inference/ray/` | +| Backend packages | `packages/{diffusers,transformers,vllm}-cosmos3/` | +| Example inputs | `inputs/omni/*.json`, `inputs/autoregressive/*.{json,jsonl}` | + +## Documentation + +| Doc | What it covers | +| -------------------------------------------------- | ----------------------------------------------------------------- | +| [docs/setup.md](./docs/setup.md) | Install, NGC base image, CUDA variants, base-checkpoint download. | +| [docs/code_structure.md](./docs/code_structure.md) | Repo layout and per-subpackage tour of `cosmos_framework/`. | +| [docs/training.md](./docs/training.md) | Single- and multi-node launches, parallelism, mixed precision. | +| [docs/inference.md](./docs/inference.md) | Sample arguments, parallelism, schemas, troubleshooting. | +| [docs/faq.md](./docs/faq.md) | Troubleshooting (OOM, NCCL, slow training) + env vars. | + +Agent skills (codebase navigation, env troubleshooting, inference, post-training, setup) live in [`.agents/skills/`](./.agents/skills) and [`.claude/skills/`](./.claude/skills). + +## Common Tasks + +### Training + +| Task | Command | +| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Single-GPU train (smoke) | `python -m cosmos_framework.scripts.train --sft-toml=examples/toml/sft_config/.toml` | +| Multi-GPU train | `IMAGINAIRE_OUTPUT_ROOT=outputs/train torchrun --nproc-per-node=8 -m cosmos_framework.scripts.train --sft-toml=examples/toml/sft_config/.toml` | +| Resume from checkpoint | Re-run the same `train --sft-toml=.toml` against the same `IMAGINAIRE_OUTPUT_ROOT` (auto-resume from latest DCP). | +| Export DCP → HF | `python -m cosmos_framework.scripts.export_model --src --dst ` | +| Run a config sweep | `just run python -m cosmos_framework.scripts.train --sft-toml=examples/toml/sft_config/.toml -- key.path=value ...` | + +### Inference + +| Task | Command | +| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------- | +| Single-GPU inference | `python -m cosmos_framework.scripts.inference -i inputs/omni/t2v.json -o outputs/ --checkpoint-path Cosmos3-Nano` | +| Multi-GPU inference | `torchrun --nproc-per-node=4 -m cosmos_framework.scripts.inference --parallelism-preset=latency -i ... -o outputs/ ...` | +| Start online Ray server | `python -m cosmos_framework.inference.ray.serve --parallelism-preset=latency -o outputs/ray_serve --checkpoint-path Cosmos3-Nano` | +| Launch Gradio UI | `python -m cosmos_framework.inference.ray.gradio --port=8080` | +| See all CLI flags | `python -m cosmos_framework.scripts.inference --help` | + +## Gotchas + +- **NGC / PyTorch containers**: run `export LD_LIBRARY_PATH=''` before any `python` call or you'll hit a `torch._C` import error. See [`docs/setup.md`](./docs/setup.md#pytorch-import-issue). +- **Reproducibility**: always pass `--seed `. Without it a random seed is used each run. +- **JSON paths**: relative paths inside input JSON files resolve relative to the JSON file's directory, not the working directory. +- **Resume**: re-running the same inference command skips already-generated outputs automatically. +- **Separation of concerns**: keep training-time imports out of `cosmos_framework/inference/`, and keep heavyweight inference-only deps (vLLM, Ray Serve, Gradio) gated behind optional extras so plain training installs stay slim. diff --git a/ATTRIBUTIONS.md b/ATTRIBUTIONS.md new file mode 100644 index 0000000..7567787 --- /dev/null +++ b/ATTRIBUTIONS.md @@ -0,0 +1,57534 @@ +DISTS-pytorch +MIT +https://github.com/dingkeyan93/DISTS +MIT License + +Copyright (c) 2020 Keyan Ding + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +Deprecated +MIT License +https://github.com/laurent-laporte-pro/deprecated +The MIT License (MIT) + +Copyright (c) 2017 Laurent LAPORTE + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +Flask +BSD-3-Clause +https://github.com/pallets/flask/ +Copyright 2010 Pallets + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +GitPython +BSD-3-Clause +https://github.com/gitpython-developers/GitPython +Copyright (C) 2008, 2009 Michael Trier and contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +* Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +* Neither the name of the GitPython project nor the names of +its contributors may be used to endorse or promote products derived +from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +ImageIO +BSD-2-Clause +https://github.com/imageio/imageio +Copyright (c) 2014-2025, imageio developers +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + + +Jinja2 +BSD License +https://github.com/pallets/jinja/ +Copyright 2007 Pallets + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +Markdown +BSD-3-Clause +https://Python-Markdown.github.io/ +BSD 3-Clause License + +Copyright 2007, 2008 The Python Markdown Project (v. 1.7 and later) +Copyright 2004, 2005, 2006 Yuri Takhteyev (v. 0.2-1.6b) +Copyright 2004 Manfred Stienstra (the original version) + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +MarkupSafe +BSD-3-Clause +https://github.com/pallets/markupsafe/ +Copyright 2010 Pallets + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +NATTEN +UNKNOWN +https://natten.org +MIT License + +Copyright (c) 2022 - 2026 Ali Hassani. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +PyDispatcher +BSD License +https://github.com/mcfletch/pydispatcher +UNKNOWN + +PyOpenGL +BSD License +https://mcfletch.github.io/pyopengl/ +UNKNOWN + +PyYAML +MIT License +https://pyyaml.org/ +Copyright (c) 2017-2021 Ingy döt Net +Copyright (c) 2006-2016 Kirill Simonov + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +Pygments +BSD License +https://pygments.org +Copyright (c) 2006-2022 by the respective authors (see AUTHORS file). +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +* Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +SecretStorage +BSD-3-Clause +https://github.com/mitya57/secretstorage +Copyright 2012-2025 Dmitry Shachnev +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. +3. Neither the name of the University nor the names of its contributors may be + used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR ANY +DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +Send2Trash +BSD-3-Clause +https://github.com/arsenetar/send2trash +Copyright (c) 2017, Virgil Dupras +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. + * Neither the name of Hardcoded Software Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +Werkzeug +BSD-3-Clause +https://github.com/pallets/werkzeug/ +Copyright 2007 Pallets + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +absl-py +Apache-2.0 +https://github.com/abseil/abseil-py + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +accelerate +Apache Software License +https://github.com/huggingface/accelerate + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +aioboto3 +Apache Software License +https://github.com/terricain/aioboto3 +Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2015-2016 Nikolai Novik + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +aiobotocore +Apache-2.0 +https://github.com/aio-libs/aiobotocore +Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2015-2016 Nikolai Novik + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +aiofiles +Apache Software License +https://github.com/Tinche/aiofiles +Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +aiohappyeyeballs +Python Software Foundation License +https://github.com/aio-libs/aiohappyeyeballs +A. HISTORY OF THE SOFTWARE +========================== + +Python was created in the early 1990s by Guido van Rossum at Stichting +Mathematisch Centrum (CWI, see https://www.cwi.nl) in the Netherlands +as a successor of a language called ABC. Guido remains Python's +principal author, although it includes many contributions from others. + +In 1995, Guido continued his work on Python at the Corporation for +National Research Initiatives (CNRI, see https://www.cnri.reston.va.us) +in Reston, Virginia where he released several versions of the +software. + +In May 2000, Guido and the Python core development team moved to +BeOpen.com to form the BeOpen PythonLabs team. In October of the same +year, the PythonLabs team moved to Digital Creations, which became +Zope Corporation. In 2001, the Python Software Foundation (PSF, see +https://www.python.org/psf/) was formed, a non-profit organization +created specifically to own Python-related Intellectual Property. +Zope Corporation was a sponsoring member of the PSF. + +All Python releases are Open Source (see https://opensource.org for +the Open Source Definition). Historically, most, but not all, Python +releases have also been GPL-compatible; the table below summarizes +the various releases. + + Release Derived Year Owner GPL- + from compatible? (1) + + 0.9.0 thru 1.2 1991-1995 CWI yes + 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes + 1.6 1.5.2 2000 CNRI no + 2.0 1.6 2000 BeOpen.com no + 1.6.1 1.6 2001 CNRI yes (2) + 2.1 2.0+1.6.1 2001 PSF no + 2.0.1 2.0+1.6.1 2001 PSF yes + 2.1.1 2.1+2.0.1 2001 PSF yes + 2.1.2 2.1.1 2002 PSF yes + 2.1.3 2.1.2 2002 PSF yes + 2.2 and above 2.1.1 2001-now PSF yes + +Footnotes: + +(1) GPL-compatible doesn't mean that we're distributing Python under + the GPL. All Python licenses, unlike the GPL, let you distribute + a modified version without making your changes open source. The + GPL-compatible licenses make it possible to combine Python with + other software that is released under the GPL; the others don't. + +(2) According to Richard Stallman, 1.6.1 is not GPL-compatible, + because its license has a choice of law clause. According to + CNRI, however, Stallman's lawyer has told CNRI's lawyer that 1.6.1 + is "not incompatible" with the GPL. + +Thanks to the many outside volunteers who have worked under Guido's +direction to make these releases possible. + + +B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON +=============================================================== + +Python software and documentation are licensed under the +Python Software Foundation License Version 2. + +Starting with Python 3.8.6, examples, recipes, and other code in +the documentation are dual licensed under the PSF License Version 2 +and the Zero-Clause BSD license. + +Some software incorporated into Python is under different licenses. +The licenses are listed with code falling under that license. + + +PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 +-------------------------------------------- + +1. This LICENSE AGREEMENT is between the Python Software Foundation +("PSF"), and the Individual or Organization ("Licensee") accessing and +otherwise using this software ("Python") in source or binary form and +its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, PSF hereby +grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, +analyze, test, perform and/or display publicly, prepare derivative works, +distribute, and otherwise use Python alone or in any derivative version, +provided, however, that PSF's License Agreement and PSF's notice of copyright, +i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, +2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023 Python Software Foundation; +All Rights Reserved" are retained in Python alone or in any derivative version +prepared by Licensee. + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python. + +4. PSF is making Python available to Licensee on an "AS IS" +basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. Nothing in this License Agreement shall be deemed to create any +relationship of agency, partnership, or joint venture between PSF and +Licensee. This License Agreement does not grant permission to use PSF +trademarks or trade name in a trademark sense to endorse or promote +products or services of Licensee, or any third party. + +8. By copying, installing or otherwise using Python, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0 +------------------------------------------- + +BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1 + +1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an +office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the +Individual or Organization ("Licensee") accessing and otherwise using +this software in source or binary form and its associated +documentation ("the Software"). + +2. Subject to the terms and conditions of this BeOpen Python License +Agreement, BeOpen hereby grants Licensee a non-exclusive, +royalty-free, world-wide license to reproduce, analyze, test, perform +and/or display publicly, prepare derivative works, distribute, and +otherwise use the Software alone or in any derivative version, +provided, however, that the BeOpen Python License is retained in the +Software, alone or in any derivative version prepared by Licensee. + +3. BeOpen is making the Software available to Licensee on an "AS IS" +basis. BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE +SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS +AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY +DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +5. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +6. This License Agreement shall be governed by and interpreted in all +respects by the law of the State of California, excluding conflict of +law provisions. Nothing in this License Agreement shall be deemed to +create any relationship of agency, partnership, or joint venture +between BeOpen and Licensee. This License Agreement does not grant +permission to use BeOpen trademarks or trade names in a trademark +sense to endorse or promote products or services of Licensee, or any +third party. As an exception, the "BeOpen Python" logos available at +http://www.pythonlabs.com/logos.html may be used according to the +permissions granted on that web page. + +7. By copying, installing or otherwise using the software, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1 +--------------------------------------- + +1. This LICENSE AGREEMENT is between the Corporation for National +Research Initiatives, having an office at 1895 Preston White Drive, +Reston, VA 20191 ("CNRI"), and the Individual or Organization +("Licensee") accessing and otherwise using Python 1.6.1 software in +source or binary form and its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, CNRI +hereby grants Licensee a nonexclusive, royalty-free, world-wide +license to reproduce, analyze, test, perform and/or display publicly, +prepare derivative works, distribute, and otherwise use Python 1.6.1 +alone or in any derivative version, provided, however, that CNRI's +License Agreement and CNRI's notice of copyright, i.e., "Copyright (c) +1995-2001 Corporation for National Research Initiatives; All Rights +Reserved" are retained in Python 1.6.1 alone or in any derivative +version prepared by Licensee. Alternately, in lieu of CNRI's License +Agreement, Licensee may substitute the following text (omitting the +quotes): "Python 1.6.1 is made available subject to the terms and +conditions in CNRI's License Agreement. This Agreement together with +Python 1.6.1 may be located on the internet using the following +unique, persistent identifier (known as a handle): 1895.22/1013. This +Agreement may also be obtained from a proxy server on the internet +using the following URL: http://hdl.handle.net/1895.22/1013". + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python 1.6.1 or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python 1.6.1. + +4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS" +basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1 WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. This License Agreement shall be governed by the federal +intellectual property law of the United States, including without +limitation the federal copyright law, and, to the extent such +U.S. federal law does not apply, by the law of the Commonwealth of +Virginia, excluding Virginia's conflict of law provisions. +Notwithstanding the foregoing, with regard to derivative works based +on Python 1.6.1 that incorporate non-separable material that was +previously distributed under the GNU General Public License (GPL), the +law of the Commonwealth of Virginia shall govern this License +Agreement only as to issues arising under or with respect to +Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this +License Agreement shall be deemed to create any relationship of +agency, partnership, or joint venture between CNRI and Licensee. This +License Agreement does not grant permission to use CNRI trademarks or +trade name in a trademark sense to endorse or promote products or +services of Licensee, or any third party. + +8. By clicking on the "ACCEPT" button where indicated, or by copying, +installing or otherwise using Python 1.6.1, Licensee agrees to be +bound by the terms and conditions of this License Agreement. + + ACCEPT + + +CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2 +-------------------------------------------------- + +Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, +The Netherlands. All rights reserved. + +Permission to use, copy, modify, and distribute this software and its +documentation for any purpose and without fee is hereby granted, +provided that the above copyright notice appear in all copies and that +both that copyright notice and this permission notice appear in +supporting documentation, and that the name of Stichting Mathematisch +Centrum or CWI not be used in advertising or publicity pertaining to +distribution of the software without specific, written prior +permission. + +STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO +THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND +FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE +FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT +OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + +ZERO-CLAUSE BSD LICENSE FOR CODE IN THE PYTHON DOCUMENTATION +---------------------------------------------------------------------- + +Permission to use, copy, modify, and/or distribute this software for any +purpose with or without fee is hereby granted. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH +REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY +AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, +INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM +LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR +OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR +PERFORMANCE OF THIS SOFTWARE. + + +aiohttp +Apache-2.0 AND MIT +https://github.com/aio-libs/aiohttp + Copyright aio-libs contributors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +aiohttp-cors +Apache Software License +https://github.com/aio-libs/aiohttp-cors + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2015-2018 Vladimir Rutsky and aio-libs team + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +aioitertools +MIT +https://aioitertools.omnilib.dev/en/latest/changelog.html +MIT License + +Copyright (c) 2022 Amethyst Reese + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +aiosignal +Apache Software License +https://github.com/aio-libs/aiosignal +Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2013-2019 Nikolay Kim and Andrew Svetlov + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +annotated-doc +MIT +https://github.com/fastapi/annotated-doc +The MIT License (MIT) + +Copyright (c) 2025 Sebastián Ramírez + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +annotated-types +MIT License +https://github.com/annotated-types/annotated-types +The MIT License (MIT) + +Copyright (c) 2022 the contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +antlr4-python3-runtime +BSD +http://www.antlr.org +UNKNOWN + +anyio +MIT License +https://anyio.readthedocs.io/en/stable/versionhistory.html +The MIT License (MIT) + +Copyright (c) 2018 Alex Grönholm + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +argon2-cffi +MIT +https://github.com/hynek/argon2-cffi/blob/main/CHANGELOG.md +The MIT License (MIT) + +Copyright (c) 2015 Hynek Schlawack and the argon2-cffi contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +argon2-cffi-bindings +MIT +https://github.com/hynek/argon2-cffi-bindings/blob/main/CHANGELOG.md +The MIT License (MIT) + +Copyright (c) 2021 Hynek Schlawack + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +arrgh +MIT License + +Copyright (c) 2023 Nicholas Sharp + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +https://github.com/nmwsharp/arrgh +MIT License + +Copyright (c) 2023 Nicholas Sharp + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +arrow +Apache Software License +https://github.com/arrow-py/arrow + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2023 Chris Smith + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +asttokens +Apache 2.0 +https://github.com/gristlabs/asttokens + Apache License + Version 2.0, January 2004 + https://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + https://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +astunparse +BSD License +https://github.com/simonpercivall/astunparse +LICENSE +======= + +Copyright (c) 2014, Simon Percivall +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. + +* Neither the name of AST Unparser nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 +-------------------------------------------- + +1. This LICENSE AGREEMENT is between the Python Software Foundation +("PSF"), and the Individual or Organization ("Licensee") accessing and +otherwise using this software ("Python") in source or binary form and +its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, PSF hereby +grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, +analyze, test, perform and/or display publicly, prepare derivative works, +distribute, and otherwise use Python alone or in any derivative version, +provided, however, that PSF's License Agreement and PSF's notice of copyright, +i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, +2011, 2012, 2013, 2014 Python Software Foundation; All Rights Reserved" are retained +in Python alone or in any derivative version prepared by Licensee. + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python. + +4. PSF is making Python available to Licensee on an "AS IS" +basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. Nothing in this License Agreement shall be deemed to create any +relationship of agency, partnership, or joint venture between PSF and +Licensee. This License Agreement does not grant permission to use PSF +trademarks or trade name in a trademark sense to endorse or promote +products or services of Licensee, or any third party. + +8. By copying, installing or otherwise using Python, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +async-lru +MIT License +https://github.com/aio-libs/async-lru +The MIT License + +Copyright (c) 2018 aio-libs team https://github.com/aio-libs/ +Copyright (c) 2017 Ocean S. A. https://ocean.io/ +Copyright (c) 2016-2017 WikiBusiness Corporation http://wikibusiness.org/ + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +attrs +MIT +https://www.attrs.org/en/stable/changelog.html +The MIT License (MIT) + +Copyright (c) 2015 Hynek Schlawack and the attrs contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +audioop-lts +PSF-2.0 +https://github.com/AbstractUmbra/audioop +A. HISTORY OF THE SOFTWARE +========================== + +Python was created in the early 1990s by Guido van Rossum at Stichting +Mathematisch Centrum (CWI, see https://www.cwi.nl) in the Netherlands +as a successor of a language called ABC. Guido remains Python's +principal author, although it includes many contributions from others. + +In 1995, Guido continued his work on Python at the Corporation for +National Research Initiatives (CNRI, see https://www.cnri.reston.va.us) +in Reston, Virginia where he released several versions of the +software. + +In May 2000, Guido and the Python core development team moved to +BeOpen.com to form the BeOpen PythonLabs team. In October of the same +year, the PythonLabs team moved to Digital Creations, which became +Zope Corporation. In 2001, the Python Software Foundation (PSF, see +https://www.python.org/psf/) was formed, a non-profit organization +created specifically to own Python-related Intellectual Property. +Zope Corporation was a sponsoring member of the PSF. + +All Python releases are Open Source (see https://opensource.org for +the Open Source Definition). Historically, most, but not all, Python +releases have also been GPL-compatible; the table below summarizes +the various releases. + + Release Derived Year Owner GPL- + from compatible? (1) + + 0.9.0 thru 1.2 1991-1995 CWI yes + 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes + 1.6 1.5.2 2000 CNRI no + 2.0 1.6 2000 BeOpen.com no + 1.6.1 1.6 2001 CNRI yes (2) + 2.1 2.0+1.6.1 2001 PSF no + 2.0.1 2.0+1.6.1 2001 PSF yes + 2.1.1 2.1+2.0.1 2001 PSF yes + 2.1.2 2.1.1 2002 PSF yes + 2.1.3 2.1.2 2002 PSF yes + 2.2 and above 2.1.1 2001-now PSF yes + +Footnotes: + +(1) GPL-compatible doesn't mean that we're distributing Python under + the GPL. All Python licenses, unlike the GPL, let you distribute + a modified version without making your changes open source. The + GPL-compatible licenses make it possible to combine Python with + other software that is released under the GPL; the others don't. + +(2) According to Richard Stallman, 1.6.1 is not GPL-compatible, + because its license has a choice of law clause. According to + CNRI, however, Stallman's lawyer has told CNRI's lawyer that 1.6.1 + is "not incompatible" with the GPL. + +Thanks to the many outside volunteers who have worked under Guido's +direction to make these releases possible. + + +B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON +=============================================================== + +Python software and documentation are licensed under the +Python Software Foundation License Version 2. + +Starting with Python 3.8.6, examples, recipes, and other code in +the documentation are dual licensed under the PSF License Version 2 +and the Zero-Clause BSD license. + +Some software incorporated into Python is under different licenses. +The licenses are listed with code falling under that license. + + +PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 +-------------------------------------------- + +1. This LICENSE AGREEMENT is between the Python Software Foundation +("PSF"), and the Individual or Organization ("Licensee") accessing and +otherwise using this software ("Python") in source or binary form and +its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, PSF hereby +grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, +analyze, test, perform and/or display publicly, prepare derivative works, +distribute, and otherwise use Python alone or in any derivative version, +provided, however, that PSF's License Agreement and PSF's notice of copyright, +i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, +2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023 Python Software Foundation; +All Rights Reserved" are retained in Python alone or in any derivative version +prepared by Licensee. + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python. + +4. PSF is making Python available to Licensee on an "AS IS" +basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. Nothing in this License Agreement shall be deemed to create any +relationship of agency, partnership, or joint venture between PSF and +Licensee. This License Agreement does not grant permission to use PSF +trademarks or trade name in a trademark sense to endorse or promote +products or services of Licensee, or any third party. + +8. By copying, installing or otherwise using Python, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0 +------------------------------------------- + +BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1 + +1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an +office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the +Individual or Organization ("Licensee") accessing and otherwise using +this software in source or binary form and its associated +documentation ("the Software"). + +2. Subject to the terms and conditions of this BeOpen Python License +Agreement, BeOpen hereby grants Licensee a non-exclusive, +royalty-free, world-wide license to reproduce, analyze, test, perform +and/or display publicly, prepare derivative works, distribute, and +otherwise use the Software alone or in any derivative version, +provided, however, that the BeOpen Python License is retained in the +Software, alone or in any derivative version prepared by Licensee. + +3. BeOpen is making the Software available to Licensee on an "AS IS" +basis. BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE +SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS +AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY +DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +5. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +6. This License Agreement shall be governed by and interpreted in all +respects by the law of the State of California, excluding conflict of +law provisions. Nothing in this License Agreement shall be deemed to +create any relationship of agency, partnership, or joint venture +between BeOpen and Licensee. This License Agreement does not grant +permission to use BeOpen trademarks or trade names in a trademark +sense to endorse or promote products or services of Licensee, or any +third party. As an exception, the "BeOpen Python" logos available at +http://www.pythonlabs.com/logos.html may be used according to the +permissions granted on that web page. + +7. By copying, installing or otherwise using the software, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1 +--------------------------------------- + +1. This LICENSE AGREEMENT is between the Corporation for National +Research Initiatives, having an office at 1895 Preston White Drive, +Reston, VA 20191 ("CNRI"), and the Individual or Organization +("Licensee") accessing and otherwise using Python 1.6.1 software in +source or binary form and its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, CNRI +hereby grants Licensee a nonexclusive, royalty-free, world-wide +license to reproduce, analyze, test, perform and/or display publicly, +prepare derivative works, distribute, and otherwise use Python 1.6.1 +alone or in any derivative version, provided, however, that CNRI's +License Agreement and CNRI's notice of copyright, i.e., "Copyright (c) +1995-2001 Corporation for National Research Initiatives; All Rights +Reserved" are retained in Python 1.6.1 alone or in any derivative +version prepared by Licensee. Alternately, in lieu of CNRI's License +Agreement, Licensee may substitute the following text (omitting the +quotes): "Python 1.6.1 is made available subject to the terms and +conditions in CNRI's License Agreement. This Agreement together with +Python 1.6.1 may be located on the internet using the following +unique, persistent identifier (known as a handle): 1895.22/1013. This +Agreement may also be obtained from a proxy server on the internet +using the following URL: http://hdl.handle.net/1895.22/1013". + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python 1.6.1 or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python 1.6.1. + +4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS" +basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1 WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. This License Agreement shall be governed by the federal +intellectual property law of the United States, including without +limitation the federal copyright law, and, to the extent such +U.S. federal law does not apply, by the law of the Commonwealth of +Virginia, excluding Virginia's conflict of law provisions. +Notwithstanding the foregoing, with regard to derivative works based +on Python 1.6.1 that incorporate non-separable material that was +previously distributed under the GNU General Public License (GPL), the +law of the Commonwealth of Virginia shall govern this License +Agreement only as to issues arising under or with respect to +Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this +License Agreement shall be deemed to create any relationship of +agency, partnership, or joint venture between CNRI and Licensee. This +License Agreement does not grant permission to use CNRI trademarks or +trade name in a trademark sense to endorse or promote products or +services of Licensee, or any third party. + +8. By clicking on the "ACCEPT" button where indicated, or by copying, +installing or otherwise using Python 1.6.1, Licensee agrees to be +bound by the terms and conditions of this License Agreement. + + ACCEPT + + +CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2 +-------------------------------------------------- + +Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, +The Netherlands. All rights reserved. + +Permission to use, copy, modify, and distribute this software and its +documentation for any purpose and without fee is hereby granted, +provided that the above copyright notice appear in all copies and that +both that copyright notice and this permission notice appear in +supporting documentation, and that the name of Stichting Mathematisch +Centrum or CWI not be used in advertising or publicity pertaining to +distribution of the software without specific, written prior +permission. + +STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO +THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND +FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE +FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT +OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + +ZERO-CLAUSE BSD LICENSE FOR CODE IN THE PYTHON DOCUMENTATION +---------------------------------------------------------------------- + +Permission to use, copy, modify, and/or distribute this software for any +purpose with or without fee is hereby granted. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH +REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY +AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, +INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM +LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR +OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR +PERFORMANCE OF THIS SOFTWARE. + + +av +BSD-3-Clause +https://pyav.basswood-io.com +Copyright retained by original committers. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + * Neither the name of the project nor the names of its contributors may be + used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS BE LIABLE FOR ANY DIRECT, +INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, +BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY +OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, +EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +babel +BSD License +https://babel.pocoo.org/ +Copyright (c) 2013-2026 by the Babel Team, see AUTHORS for more information. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + + 1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + 3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +backports.zstd +PSF-2.0 +https://github.com/rogdham/backports.zstd +A. HISTORY OF THE SOFTWARE +========================== + +Python was created in the early 1990s by Guido van Rossum at Stichting +Mathematisch Centrum (CWI, see https://www.cwi.nl) in the Netherlands +as a successor of a language called ABC. Guido remains Python's +principal author, although it includes many contributions from others. + +In 1995, Guido continued his work on Python at the Corporation for +National Research Initiatives (CNRI, see https://www.cnri.reston.va.us) +in Reston, Virginia where he released several versions of the +software. + +In May 2000, Guido and the Python core development team moved to +BeOpen.com to form the BeOpen PythonLabs team. In October of the same +year, the PythonLabs team moved to Digital Creations, which became +Zope Corporation. In 2001, the Python Software Foundation (PSF, see +https://www.python.org/psf/) was formed, a non-profit organization +created specifically to own Python-related Intellectual Property. +Zope Corporation was a sponsoring member of the PSF. + +All Python releases are Open Source (see https://opensource.org for +the Open Source Definition). Historically, most, but not all, Python +releases have also been GPL-compatible; the table below summarizes +the various releases. + + Release Derived Year Owner GPL- + from compatible? (1) + + 0.9.0 thru 1.2 1991-1995 CWI yes + 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes + 1.6 1.5.2 2000 CNRI no + 2.0 1.6 2000 BeOpen.com no + 1.6.1 1.6 2001 CNRI yes (2) + 2.1 2.0+1.6.1 2001 PSF no + 2.0.1 2.0+1.6.1 2001 PSF yes + 2.1.1 2.1+2.0.1 2001 PSF yes + 2.1.2 2.1.1 2002 PSF yes + 2.1.3 2.1.2 2002 PSF yes + 2.2 and above 2.1.1 2001-now PSF yes + +Footnotes: + +(1) GPL-compatible doesn't mean that we're distributing Python under + the GPL. All Python licenses, unlike the GPL, let you distribute + a modified version without making your changes open source. The + GPL-compatible licenses make it possible to combine Python with + other software that is released under the GPL; the others don't. + +(2) According to Richard Stallman, 1.6.1 is not GPL-compatible, + because its license has a choice of law clause. According to + CNRI, however, Stallman's lawyer has told CNRI's lawyer that 1.6.1 + is "not incompatible" with the GPL. + +Thanks to the many outside volunteers who have worked under Guido's +direction to make these releases possible. + + +B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON +=============================================================== + +Python software and documentation are licensed under the +Python Software Foundation License Version 2. + +Starting with Python 3.8.6, examples, recipes, and other code in +the documentation are dual licensed under the PSF License Version 2 +and the Zero-Clause BSD license. + +Some software incorporated into Python is under different licenses. +The licenses are listed with code falling under that license. + + +PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 +-------------------------------------------- + +1. This LICENSE AGREEMENT is between the Python Software Foundation +("PSF"), and the Individual or Organization ("Licensee") accessing and +otherwise using this software ("Python") in source or binary form and +its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, PSF hereby +grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, +analyze, test, perform and/or display publicly, prepare derivative works, +distribute, and otherwise use Python alone or in any derivative version, +provided, however, that PSF's License Agreement and PSF's notice of copyright, +i.e., "Copyright (c) 2001 Python Software Foundation; All Rights Reserved" +are retained in Python alone or in any derivative version prepared by Licensee. + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python. + +4. PSF is making Python available to Licensee on an "AS IS" +basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. Nothing in this License Agreement shall be deemed to create any +relationship of agency, partnership, or joint venture between PSF and +Licensee. This License Agreement does not grant permission to use PSF +trademarks or trade name in a trademark sense to endorse or promote +products or services of Licensee, or any third party. + +8. By copying, installing or otherwise using Python, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0 +------------------------------------------- + +BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1 + +1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an +office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the +Individual or Organization ("Licensee") accessing and otherwise using +this software in source or binary form and its associated +documentation ("the Software"). + +2. Subject to the terms and conditions of this BeOpen Python License +Agreement, BeOpen hereby grants Licensee a non-exclusive, +royalty-free, world-wide license to reproduce, analyze, test, perform +and/or display publicly, prepare derivative works, distribute, and +otherwise use the Software alone or in any derivative version, +provided, however, that the BeOpen Python License is retained in the +Software, alone or in any derivative version prepared by Licensee. + +3. BeOpen is making the Software available to Licensee on an "AS IS" +basis. BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE +SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS +AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY +DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +5. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +6. This License Agreement shall be governed by and interpreted in all +respects by the law of the State of California, excluding conflict of +law provisions. Nothing in this License Agreement shall be deemed to +create any relationship of agency, partnership, or joint venture +between BeOpen and Licensee. This License Agreement does not grant +permission to use BeOpen trademarks or trade names in a trademark +sense to endorse or promote products or services of Licensee, or any +third party. As an exception, the "BeOpen Python" logos available at +http://www.pythonlabs.com/logos.html may be used according to the +permissions granted on that web page. + +7. By copying, installing or otherwise using the software, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1 +--------------------------------------- + +1. This LICENSE AGREEMENT is between the Corporation for National +Research Initiatives, having an office at 1895 Preston White Drive, +Reston, VA 20191 ("CNRI"), and the Individual or Organization +("Licensee") accessing and otherwise using Python 1.6.1 software in +source or binary form and its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, CNRI +hereby grants Licensee a nonexclusive, royalty-free, world-wide +license to reproduce, analyze, test, perform and/or display publicly, +prepare derivative works, distribute, and otherwise use Python 1.6.1 +alone or in any derivative version, provided, however, that CNRI's +License Agreement and CNRI's notice of copyright, i.e., "Copyright (c) +1995-2001 Corporation for National Research Initiatives; All Rights +Reserved" are retained in Python 1.6.1 alone or in any derivative +version prepared by Licensee. Alternately, in lieu of CNRI's License +Agreement, Licensee may substitute the following text (omitting the +quotes): "Python 1.6.1 is made available subject to the terms and +conditions in CNRI's License Agreement. This Agreement together with +Python 1.6.1 may be located on the internet using the following +unique, persistent identifier (known as a handle): 1895.22/1013. This +Agreement may also be obtained from a proxy server on the internet +using the following URL: http://hdl.handle.net/1895.22/1013". + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python 1.6.1 or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python 1.6.1. + +4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS" +basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1 WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. This License Agreement shall be governed by the federal +intellectual property law of the United States, including without +limitation the federal copyright law, and, to the extent such +U.S. federal law does not apply, by the law of the Commonwealth of +Virginia, excluding Virginia's conflict of law provisions. +Notwithstanding the foregoing, with regard to derivative works based +on Python 1.6.1 that incorporate non-separable material that was +previously distributed under the GNU General Public License (GPL), the +law of the Commonwealth of Virginia shall govern this License +Agreement only as to issues arising under or with respect to +Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this +License Agreement shall be deemed to create any relationship of +agency, partnership, or joint venture between CNRI and Licensee. This +License Agreement does not grant permission to use CNRI trademarks or +trade name in a trademark sense to endorse or promote products or +services of Licensee, or any third party. + +8. By clicking on the "ACCEPT" button where indicated, or by copying, +installing or otherwise using Python 1.6.1, Licensee agrees to be +bound by the terms and conditions of this License Agreement. + + ACCEPT + + +CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2 +-------------------------------------------------- + +Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, +The Netherlands. All rights reserved. + +Permission to use, copy, modify, and distribute this software and its +documentation for any purpose and without fee is hereby granted, +provided that the above copyright notice appear in all copies and that +both that copyright notice and this permission notice appear in +supporting documentation, and that the name of Stichting Mathematisch +Centrum or CWI not be used in advertising or publicity pertaining to +distribution of the software without specific, written prior +permission. + +STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO +THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND +FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE +FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT +OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + +ZERO-CLAUSE BSD LICENSE FOR CODE IN THE PYTHON DOCUMENTATION +---------------------------------------------------------------------- + +Permission to use, copy, modify, and/or distribute this software for any +purpose with or without fee is hereby granted. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH +REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY +AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, +INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM +LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR +OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR +PERFORMANCE OF THIS SOFTWARE. + + +beautifulsoup4 +MIT License +https://www.crummy.com/software/BeautifulSoup/bs4/ +Beautiful Soup is made available under the MIT license: + + Copyright (c) Leonard Richardson + + Permission is hereby granted, free of charge, to any person obtaining + a copy of this software and associated documentation files (the + "Software"), to deal in the Software without restriction, including + without limitation the rights to use, copy, modify, merge, publish, + distribute, sublicense, and/or sell copies of the Software, and to + permit persons to whom the Software is furnished to do so, subject to + the following conditions: + + The above copyright notice and this permission notice shall be + included in all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + +Beautiful Soup incorporates code from the html5lib library, which is +also made available under the MIT license. Copyright (c) James Graham +and other contributors + +Beautiful Soup has an optional dependency on the soupsieve library, +which is also made available under the MIT license. Copyright (c) +Isaac Muse + + +better-profanity +MIT License +https://github.com/snguyenthanh/better_profanity +Copyright (c) 2018 The Python Packaging Authority + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +bleach +Apache Software License +https://github.com/mozilla/bleach +Copyright (c) 2014-2017, Mozilla Foundation + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + + +blinker +MIT License +https://github.com/pallets-eco/blinker/ +Copyright 2010 Jason Kirtland + +Permission is hereby granted, free of charge, to any person obtaining a +copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be included +in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS +OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, +TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +blobfile +Public Domain +https://github.com/blobfile/blobfile +This is free and unencumbered software released into the public domain. + +Anyone is free to copy, modify, publish, use, compile, sell, or +distribute this software, either in source code form or as a compiled +binary, for any purpose, commercial or non-commercial, and by any +means. + +In jurisdictions that recognize copyright laws, the author or authors +of this software dedicate any and all copyright interest in the +software to the public domain. We make this dedication for the benefit +of the public at large and to the detriment of our heirs and +successors. We intend this dedication to be an overt act of +relinquishment in perpetuity of all present and future rights to this +software under copyright law. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +OTHER DEALINGS IN THE SOFTWARE. + +For more information, please refer to + +boto3 +Apache-2.0 +https://github.com/boto/boto3 + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + +botocore +Apache-2.0 +https://github.com/boto/botocore + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + +braceexpand +MIT License +https://github.com/trendels/braceexpand +The MIT License (MIT) + +Copyright (c) 2015 Stanis Trendelenburg + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +bracex +MIT +https://github.com/facelessuser/bracex +MIT License + +Copyright (c) 2018 - 2025 Isaac Muse + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +brotli +MIT +https://github.com/google/brotli +Copyright (c) 2009, 2010, 2013-2016 by the Brotli Authors. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +cattrs +MIT License +https://catt.rs + +MIT License + +Copyright (c) 2016, Tin Tvrtković + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + + +certifi +Mozilla Public License 2.0 (MPL 2.0) +https://github.com/certifi/python-certifi +This package contains a modified version of ca-bundle.crt: + +ca-bundle.crt -- Bundle of CA Root Certificates + +This is a bundle of X.509 certificates of public Certificate Authorities +(CA). These were automatically extracted from Mozilla's root certificates +file (certdata.txt). This file can be found in the mozilla source tree: +https://hg.mozilla.org/mozilla-central/file/tip/security/nss/lib/ckfw/builtins/certdata.txt +It contains the certificates in PEM format and therefore +can be directly used with curl / libcurl / php_curl, or with +an Apache+mod_ssl webserver for SSL client authentication. +Just configure this file as the SSLCACertificateFile.# + +***** BEGIN LICENSE BLOCK ***** +This Source Code Form is subject to the terms of the Mozilla Public License, +v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain +one at http://mozilla.org/MPL/2.0/. + +***** END LICENSE BLOCK ***** +@(#) $RCSfile: certdata.txt,v $ $Revision: 1.80 $ $Date: 2011/11/03 15:11:58 $ + + +cffi +MIT +https://cffi.readthedocs.io/en/latest/whatsnew.html + +Except when otherwise stated (look for LICENSE files in directories or +information at the beginning of each file) all software and +documentation is licensed as follows: + + MIT No Attribution + + Permission is hereby granted, free of charge, to any person + obtaining a copy of this software and associated documentation + files (the "Software"), to deal in the Software without + restriction, including without limitation the rights to use, + copy, modify, merge, publish, distribute, sublicense, and/or + sell copies of the Software, and to permit persons to whom the + Software is furnished to do so. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS + OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER + DEALINGS IN THE SOFTWARE. + + + +charset-normalizer +MIT +https://github.com/jawah/charset_normalizer/blob/master/CHANGELOG.md +MIT License + +Copyright (c) 2025 TAHRI Ahmed R. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +click +BSD-3-Clause +https://github.com/pallets/click/ +Copyright 2014 Pallets + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +cmake +Apache Software License; BSD License +https://cmake.org +Apache License +Version 2.0, January 2004 +http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + +"License" shall mean the terms and conditions for use, reproduction, and +distribution as defined by Sections 1 through 9 of this document. + +"Licensor" shall mean the copyright owner or entity authorized by the copyright +owner that is granting the License. + +"Legal Entity" shall mean the union of the acting entity and all other entities +that control, are controlled by, or are under common control with that entity. +For the purposes of this definition, "control" means (i) the power, direct or +indirect, to cause the direction or management of such entity, whether by +contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the +outstanding shares, or (iii) beneficial ownership of such entity. + +"You" (or "Your") shall mean an individual or Legal Entity exercising +permissions granted by this License. + +"Source" form shall mean the preferred form for making modifications, including +but not limited to software source code, documentation source, and configuration +files. + +"Object" form shall mean any form resulting from mechanical transformation or +translation of a Source form, including but not limited to compiled object code, +generated documentation, and conversions to other media types. + +"Work" shall mean the work of authorship, whether in Source or Object form, made +available under the License, as indicated by a copyright notice that is included +in or attached to the work (an example is provided in the Appendix below). + +"Derivative Works" shall mean any work, whether in Source or Object form, that +is based on (or derived from) the Work and for which the editorial revisions, +annotations, elaborations, or other modifications represent, as a whole, an +original work of authorship. For the purposes of this License, Derivative Works +shall not include works that remain separable from, or merely link (or bind by +name) to the interfaces of, the Work and Derivative Works thereof. + +"Contribution" shall mean any work of authorship, including the original version +of the Work and any modifications or additions to that Work or Derivative Works +thereof, that is intentionally submitted to Licensor for inclusion in the Work +by the copyright owner or by an individual or Legal Entity authorized to submit +on behalf of the copyright owner. For the purposes of this definition, +"submitted" means any form of electronic, verbal, or written communication sent +to the Licensor or its representatives, including but not limited to +communication on electronic mailing lists, source code control systems, and +issue tracking systems that are managed by, or on behalf of, the Licensor for +the purpose of discussing and improving the Work, but excluding communication +that is conspicuously marked or otherwise designated in writing by the copyright +owner as "Not a Contribution." + +"Contributor" shall mean Licensor and any individual or Legal Entity on behalf +of whom a Contribution has been received by Licensor and subsequently +incorporated within the Work. + +2. Grant of Copyright License. + +Subject to the terms and conditions of this License, each Contributor hereby +grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, +irrevocable copyright license to reproduce, prepare Derivative Works of, +publicly display, publicly perform, sublicense, and distribute the Work and such +Derivative Works in Source or Object form. + +3. Grant of Patent License. + +Subject to the terms and conditions of this License, each Contributor hereby +grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, +irrevocable (except as stated in this section) patent license to make, have +made, use, offer to sell, sell, import, and otherwise transfer the Work, where +such license applies only to those patent claims licensable by such Contributor +that are necessarily infringed by their Contribution(s) alone or by combination +of their Contribution(s) with the Work to which such Contribution(s) was +submitted. If You institute patent litigation against any entity (including a +cross-claim or counterclaim in a lawsuit) alleging that the Work or a +Contribution incorporated within the Work constitutes direct or contributory +patent infringement, then any patent licenses granted to You under this License +for that Work shall terminate as of the date such litigation is filed. + +4. Redistribution. + +You may reproduce and distribute copies of the Work or Derivative Works thereof +in any medium, with or without modifications, and in Source or Object form, +provided that You meet the following conditions: + +You must give any other recipients of the Work or Derivative Works a copy of +this License; and +You must cause any modified files to carry prominent notices stating that You +changed the files; and +You must retain, in the Source form of any Derivative Works that You distribute, +all copyright, patent, trademark, and attribution notices from the Source form +of the Work, excluding those notices that do not pertain to any part of the +Derivative Works; and +If the Work includes a "NOTICE" text file as part of its distribution, then any +Derivative Works that You distribute must include a readable copy of the +attribution notices contained within such NOTICE file, excluding those notices +that do not pertain to any part of the Derivative Works, in at least one of the +following places: within a NOTICE text file distributed as part of the +Derivative Works; within the Source form or documentation, if provided along +with the Derivative Works; or, within a display generated by the Derivative +Works, if and wherever such third-party notices normally appear. The contents of +the NOTICE file are for informational purposes only and do not modify the +License. You may add Your own attribution notices within Derivative Works that +You distribute, alongside or as an addendum to the NOTICE text from the Work, +provided that such additional attribution notices cannot be construed as +modifying the License. +You may add Your own copyright statement to Your modifications and may provide +additional or different license terms and conditions for use, reproduction, or +distribution of Your modifications, or for any such Derivative Works as a whole, +provided Your use, reproduction, and distribution of the Work otherwise complies +with the conditions stated in this License. + +5. Submission of Contributions. + +Unless You explicitly state otherwise, any Contribution intentionally submitted +for inclusion in the Work by You to the Licensor shall be under the terms and +conditions of this License, without any additional terms or conditions. +Notwithstanding the above, nothing herein shall supersede or modify the terms of +any separate license agreement you may have executed with Licensor regarding +such Contributions. + +6. Trademarks. + +This License does not grant permission to use the trade names, trademarks, +service marks, or product names of the Licensor, except as required for +reasonable and customary use in describing the origin of the Work and +reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. + +Unless required by applicable law or agreed to in writing, Licensor provides the +Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, +including, without limitation, any warranties or conditions of TITLE, +NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are +solely responsible for determining the appropriateness of using or +redistributing the Work and assume any risks associated with Your exercise of +permissions under this License. + +8. Limitation of Liability. + +In no event and under no legal theory, whether in tort (including negligence), +contract, or otherwise, unless required by applicable law (such as deliberate +and grossly negligent acts) or agreed to in writing, shall any Contributor be +liable to You for damages, including any direct, indirect, special, incidental, +or consequential damages of any character arising as a result of this License or +out of the use or inability to use the Work (including but not limited to +damages for loss of goodwill, work stoppage, computer failure or malfunction, or +any and all other commercial damages or losses), even if such Contributor has +been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. + +While redistributing the Work or Derivative Works thereof, You may choose to +offer, and charge a fee for, acceptance of support, warranty, indemnity, or +other liability obligations and/or rights consistent with this License. However, +in accepting such obligations, You may act only on Your own behalf and on Your +sole responsibility, not on behalf of any other Contributor, and only if You +agree to indemnify, defend, and hold each Contributor harmless for any liability +incurred by, or claims asserted against, such Contributor by reason of your +accepting any such warranty or additional liability. + +END OF TERMS AND CONDITIONS + +APPENDIX: How to apply the Apache License to your work + +To apply the Apache License to your work, attach the following boilerplate +notice, with the fields enclosed by brackets "[]" replaced with your own +identifying information. (Don't include the brackets!) The text should be +enclosed in the appropriate comment syntax for the file format. We also +recommend that a file or class name and description of purpose be included on +the same "printed page" as the copyright notice for easier identification within +third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +colorful +MIT License +http://github.com/timofurrer/colorful +The MIT License (MIT) + +Copyright (c) 2017 Timo Furrer + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +comm +BSD License +https://github.com/ipython/comm +BSD 3-Clause License + +Copyright (c) 2022, Jupyter +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +contourpy +BSD License +https://github.com/contourpy/contourpy +BSD 3-Clause License + +Copyright (c) 2021-2025, ContourPy Developers. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +cosmos3 +Apache Software License +https://research.nvidia.com/labs/dir/cosmos3 + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +coverage +Apache-2.0 +https://github.com/coveragepy/coveragepy + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + +cramjam +MIT +https://github.com/milesgranger/pyrus-cramjam +MIT License + +Copyright (c) 2020 Miles Granger + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +cryptography +Apache-2.0 OR BSD-3-Clause +https://github.com/pyca/cryptography +This software is made available under the terms of *either* of the licenses +found in LICENSE.APACHE or LICENSE.BSD. Contributions to cryptography are made +under the terms of *both* these licenses. + + +cuda-bindings +LicenseRef-NVIDIA-SOFTWARE-LICENSE +https://github.com/NVIDIA/cuda-python +NVIDIA SOFTWARE LICENSE + +This license is a legal agreement between you and NVIDIA Corporation ("NVIDIA") and governs your use of the NVIDIA CUDA Python software and materials provided hereunder ("SOFTWARE"). + +This license can be accepted only by an adult of legal age of majority in the country in which the SOFTWARE is used. If you are under the legal age of majority, you must ask your parent or legal guardian to consent to this license. By taking delivery of the SOFTWARE, you affirm that you have reached the legal age of majority, you accept the terms of this license, and you take legal and financial responsibility for the actions of your permitted users. + +You agree to use the SOFTWARE only for purposes that are permitted by (a) this license, and (b) any applicable law, regulation or generally accepted practices or guidelines in the relevant jurisdictions. + +1. LICENSE. Subject to the terms of this license, NVIDIA grants you a non-exclusive limited license to: (a) install and use the SOFTWARE, and (b) distribute the SOFTWARE subject to the distribution requirements described in this license. NVIDIA reserves all rights, title and interest in and to the SOFTWARE not expressly granted to you under this license. + +2. DISTRIBUTION REQUIREMENTS. These are the distribution requirements for you to exercise the distribution grant: +a. The terms under which you distribute the SOFTWARE must be consistent with the terms of this license, including (without limitation) terms relating to the license grant and license restrictions and protection of NVIDIA's intellectual property rights. +b. You agree to notify NVIDIA in writing of any known or suspected distribution or use of the SOFTWARE not in compliance with the requirements of this license, and to enforce the terms of your agreements with respect to distributed SOFTWARE. + +3. LIMITATIONS. Your license to use the SOFTWARE is restricted as follows: +a. The SOFTWARE is licensed for you to develop applications only for use in systems with NVIDIA GPUs. +b. You may not reverse engineer, decompile or disassemble, or remove copyright or other proprietary notices from any portion of the SOFTWARE or copies of the SOFTWARE. +c. You may not modify or create derivative works of any portion of the SOFTWARE. +d. You may not bypass, disable, or circumvent any technical measure, encryption, security, digital rights management or authentication mechanism in the SOFTWARE. +e. You may not use the SOFTWARE in any manner that would cause it to become subject to an open source software license. As examples, licenses that require as a condition of use, modification, and/or distribution that the SOFTWARE be (i) disclosed or distributed in source code form; (ii) licensed for the purpose of making derivative works; or (iii) redistributable at no charge. +f. Unless you have an agreement with NVIDIA for this purpose, you may not use the SOFTWARE with any system or application where the use or failure of the system or application can reasonably be expected to threaten or result in personal injury, death, or catastrophic loss. Examples include use in avionics, navigation, military, medical, life support or other life critical applications. NVIDIA does not design, test or manufacture the SOFTWARE for these critical uses and NVIDIA shall not be liable to you or any third party, in whole or in part, for any claims or damages arising from such uses. +g. You agree to defend, indemnify and hold harmless NVIDIA and its affiliates, and their respective employees, contractors, agents, officers and directors, from and against any and all claims, damages, obligations, losses, liabilities, costs or debt, fines, restitutions and expenses (including but not limited to attorney's fees and costs incident to establishing the right of indemnification) arising out of or related to use of the SOFTWARE outside of the scope of this Agreement, or not in compliance with its terms. + +4. PRE-RELEASE. SOFTWARE versions identified as alpha, beta, preview, early access or otherwise as pre-release may not be fully functional, may contain errors or design flaws, and may have reduced or different security, privacy, availability, and reliability standards relative to commercial versions of NVIDIA software and materials. You may use a pre-release SOFTWARE version at your own risk, understanding that these versions are not intended for use in production or business-critical systems. + +5. OWNERSHIP. The SOFTWARE and the related intellectual property rights therein are and will remain the sole and exclusive property of NVIDIA or its licensors. The SOFTWARE is copyrighted and protected by the laws of the United States and other countries, and international treaty provisions. NVIDIA may make changes to the SOFTWARE, at any time without notice, but is not obligated to support or update the SOFTWARE. + +6. COMPONENTS UNDER OTHER LICENSES. The SOFTWARE may include NVIDIA or third-party components with separate legal notices or terms as may be described in proprietary notices accompanying the SOFTWARE. If and to the extent there is a conflict between the terms in this license and the license terms associated with a component, the license terms associated with the components control only to the extent necessary to resolve the conflict. + +7. FEEDBACK. You may, but don't have to, provide to NVIDIA any Feedback. "Feedback" means any suggestions, bug fixes, enhancements, modifications, feature requests or other feedback regarding the SOFTWARE. For any Feedback that you voluntarily provide, you hereby grant NVIDIA and its affiliates a perpetual, non-exclusive, worldwide, irrevocable license to use, reproduce, modify, license, sublicense (through multiple tiers of sublicensees), and distribute (through multiple tiers of distributors) the Feedback without the payment of any royalties or fees to you. NVIDIA will use Feedback at its choice. + +8. NO WARRANTIES. THE SOFTWARE IS PROVIDED "AS IS" WITHOUT ANY EXPRESS OR IMPLIED WARRANTY OF ANY KIND INCLUDING, BUT NOT LIMITED TO, WARRANTIES OF MERCHANTABILITY, NONINFRINGEMENT, OR FITNESS FOR A PARTICULAR PURPOSE. NVIDIA DOES NOT WARRANT THAT THE SOFTWARE WILL MEET YOUR REQUIREMENTS OR THAT THE OPERATION THEREOF WILL BE UNINTERRUPTED OR ERROR-FREE, OR THAT ALL ERRORS WILL BE CORRECTED. + +9. LIMITATIONS OF LIABILITY. TO THE MAXIMUM EXTENT PERMITTED BY LAW, NVIDIA AND ITS AFFILIATES SHALL NOT BE LIABLE FOR ANY SPECIAL, INCIDENTAL, PUNITIVE OR CONSEQUENTIAL DAMAGES, OR ANY LOST PROFITS, PROJECT DELAYS, LOSS OF USE, LOSS OF DATA OR LOSS OF GOODWILL, OR THE COSTS OF PROCURING SUBSTITUTE PRODUCTS, ARISING OUT OF OR IN CONNECTION WITH THIS LICENSE OR THE USE OR PERFORMANCE OF THE SOFTWARE, WHETHER SUCH LIABILITY ARISES FROM ANY CLAIM BASED UPON BREACH OF CONTRACT, BREACH OF WARRANTY, TORT (INCLUDING NEGLIGENCE), PRODUCT LIABILITY OR ANY OTHER CAUSE OF ACTION OR THEORY OF LIABILITY, EVEN IF NVIDIA HAS PREVIOUSLY BEEN ADVISED OF, OR COULD REASONABLY HAVE FORESEEN, THE POSSIBILITY OF SUCH DAMAGES. IN NO EVENT WILL NVIDIA'S AND ITS AFFILIATES TOTAL CUMULATIVE LIABILITY UNDER OR ARISING OUT OF THIS LICENSE EXCEED US$10.00. THE NATURE OF THE LIABILITY OR THE NUMBER OF CLAIMS OR SUITS SHALL NOT ENLARGE OR EXTEND THIS LIMIT. + +10. TERMINATION. Your rights under this license will terminate automatically without notice from NVIDIA if you fail to comply with any term and condition of this license or if you commence or participate in any legal proceeding against NVIDIA with respect to the SOFTWARE. NVIDIA may terminate this license with advance written notice to you if NVIDIA decides to no longer provide the SOFTWARE in a country or, in NVIDIA's sole discretion, the continued use of it is no longer commercially viable. Upon any termination of this license, you agree to promptly discontinue use of the SOFTWARE and destroy all copies in your possession or control. Your prior distributions in accordance with this license are not affected by the termination of this license. All provisions of this license will survive termination, except for the license granted to you. + +11. APPLICABLE LAW. This license will be governed in all respects by the laws of the United States and of the State of Delaware as those laws are applied to contracts entered into and performed entirely within Delaware by Delaware residents, without regard to the conflicts of laws principles. The United Nations Convention on Contracts for the International Sale of Goods is specifically disclaimed. You agree to all terms of this Agreement in the English language. The state or federal courts residing in Santa Clara County, California shall have exclusive jurisdiction over any dispute or claim arising out of this license. Notwithstanding this, you agree that NVIDIA shall still be allowed to apply for injunctive remedies or an equivalent type of urgent legal relief in any jurisdiction. + +12. NO ASSIGNMENT. This license and your rights and obligations thereunder may not be assigned by you by any means or operation of law without NVIDIA's permission. Any attempted assignment not approved by NVIDIA in writing shall be void and of no effect. + +13. EXPORT. The SOFTWARE is subject to United States export laws and regulations. You agree that you will not ship, transfer or export the SOFTWARE into any country, or use the SOFTWARE in any manner, prohibited by the United States Bureau of Industry and Security or economic sanctions regulations administered by the U.S. Department of Treasury's Office of Foreign Assets Control (OFAC), or any applicable export laws, restrictions or regulations. These laws include restrictions on destinations, end users and end use. By accepting this license, you confirm that you are not a resident or citizen of any country currently embargoed by the U.S. and that you are not otherwise prohibited from receiving the SOFTWARE. + +14. GOVERNMENT USE. The SOFTWARE has been developed entirely at private expense and is "commercial items" consisting of "commercial computer software" and "commercial computer software documentation" provided with RESTRICTED RIGHTS. Use, duplication or disclosure by the U.S. Government or a U.S. Government subcontractor is subject to the restrictions in this license pursuant to DFARS 227.7202-3(a) or as set forth in subparagraphs (b)(1) and (2) of the Commercial Computer Software - Restricted Rights clause at FAR 52.227-19, as applicable. Contractor/manufacturer is NVIDIA, 2788 San Tomas Expressway, Santa Clara, CA 95051. + +15. ENTIRE AGREEMENT. This license is the final, complete and exclusive agreement between the parties relating to the subject matter of this license and supersedes all prior or contemporaneous understandings and agreements relating to this subject matter, whether oral or written. If any court of competent jurisdiction determines that any provision of this license is illegal, invalid or unenforceable, the remaining provisions will remain in full force and effect. This license may only be modified in a writing signed by an authorized representative of each party. + +(v. May 12, 2021) + + +cuda-pathfinder +Apache-2.0 +https://github.com/NVIDIA/cuda-python + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + +cycler +BSD License +https://matplotlib.org/cycler/ +Copyright (c) 2015, matplotlib project +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the matplotlib project nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +dataclasses-json +MIT License +https://github.com/lidatong/dataclasses-json +MIT License + +Copyright (c) 2019 Charles Li and contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +datasets +Apache Software License +https://github.com/huggingface/datasets + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +debugpy +MIT License +https://aka.ms/debugpy + debugpy + + Copyright (c) Microsoft Corporation + All rights reserved. + + MIT License + + Permission is hereby granted, free of charge, to any person obtaining a copy of + this software and associated documentation files (the "Software"), to deal in + the Software without restriction, including without limitation the rights to + use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of + the Software, and to permit persons to whom the Software is furnished to do so, + subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER + IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + + +decorator +BSD License +UNKNOWN +Copyright (c) 2005-2025, Michele Simionato +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +* Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, +BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS +OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR +TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE +USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH +DAMAGE. + + +deepdiff +MIT License +https://zepworks.com/deepdiff/ +The MIT License (MIT) + +Copyright (c) 2014 - 2021 Sep Dehpour (Seperman) and contributors +www.zepworks.com + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + + +defusedxml +Python Software Foundation License +https://github.com/tiran/defusedxml +PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 +-------------------------------------------- + +1. This LICENSE AGREEMENT is between the Python Software Foundation +("PSF"), and the Individual or Organization ("Licensee") accessing and +otherwise using this software ("Python") in source or binary form and +its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, PSF +hereby grants Licensee a nonexclusive, royalty-free, world-wide +license to reproduce, analyze, test, perform and/or display publicly, +prepare derivative works, distribute, and otherwise use Python +alone or in any derivative version, provided, however, that PSF's +License Agreement and PSF's notice of copyright, i.e., "Copyright (c) +2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008 Python Software Foundation; +All Rights Reserved" are retained in Python alone or in any derivative +version prepared by Licensee. + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python. + +4. PSF is making Python available to Licensee on an "AS IS" +basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. Nothing in this License Agreement shall be deemed to create any +relationship of agency, partnership, or joint venture between PSF and +Licensee. This License Agreement does not grant permission to use PSF +trademarks or trade name in a trademark sense to endorse or promote +products or services of Licensee, or any third party. + +8. By copying, installing or otherwise using Python, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + + +diffusers +Apache Software License +https://github.com/huggingface/diffusers + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, Any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +dill +BSD License +https://github.com/uqfoundation/dill +Copyright (c) 2004-2016 California Institute of Technology. +Copyright (c) 2016-2025 The Uncertainty Quantification Foundation. +All rights reserved. + +This software is available subject to the conditions and terms laid +out below. By downloading and using this software you are agreeing +to the following conditions. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + + - Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + - Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + + - Neither the names of the copyright holders nor the names of any of + the contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED +TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; +OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, +WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR +OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF +ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + + +distlib +Python Software Foundation License +https://github.com/pypa/distlib +A. HISTORY OF THE SOFTWARE +========================== + +Python was created in the early 1990s by Guido van Rossum at Stichting +Mathematisch Centrum (CWI, see http://www.cwi.nl) in the Netherlands +as a successor of a language called ABC. Guido remains Python's +principal author, although it includes many contributions from others. + +In 1995, Guido continued his work on Python at the Corporation for +National Research Initiatives (CNRI, see http://www.cnri.reston.va.us) +in Reston, Virginia where he released several versions of the +software. + +In May 2000, Guido and the Python core development team moved to +BeOpen.com to form the BeOpen PythonLabs team. In October of the same +year, the PythonLabs team moved to Digital Creations (now Zope +Corporation, see http://www.zope.com). In 2001, the Python Software +Foundation (PSF, see http://www.python.org/psf/) was formed, a +non-profit organization created specifically to own Python-related +Intellectual Property. Zope Corporation is a sponsoring member of +the PSF. + +All Python releases are Open Source (see http://www.opensource.org for +the Open Source Definition). Historically, most, but not all, Python +releases have also been GPL-compatible; the table below summarizes +the various releases. + + Release Derived Year Owner GPL- + from compatible? (1) + + 0.9.0 thru 1.2 1991-1995 CWI yes + 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes + 1.6 1.5.2 2000 CNRI no + 2.0 1.6 2000 BeOpen.com no + 1.6.1 1.6 2001 CNRI yes (2) + 2.1 2.0+1.6.1 2001 PSF no + 2.0.1 2.0+1.6.1 2001 PSF yes + 2.1.1 2.1+2.0.1 2001 PSF yes + 2.2 2.1.1 2001 PSF yes + 2.1.2 2.1.1 2002 PSF yes + 2.1.3 2.1.2 2002 PSF yes + 2.2.1 2.2 2002 PSF yes + 2.2.2 2.2.1 2002 PSF yes + 2.2.3 2.2.2 2003 PSF yes + 2.3 2.2.2 2002-2003 PSF yes + 2.3.1 2.3 2002-2003 PSF yes + 2.3.2 2.3.1 2002-2003 PSF yes + 2.3.3 2.3.2 2002-2003 PSF yes + 2.3.4 2.3.3 2004 PSF yes + 2.3.5 2.3.4 2005 PSF yes + 2.4 2.3 2004 PSF yes + 2.4.1 2.4 2005 PSF yes + 2.4.2 2.4.1 2005 PSF yes + 2.4.3 2.4.2 2006 PSF yes + 2.4.4 2.4.3 2006 PSF yes + 2.5 2.4 2006 PSF yes + 2.5.1 2.5 2007 PSF yes + 2.5.2 2.5.1 2008 PSF yes + 2.5.3 2.5.2 2008 PSF yes + 2.6 2.5 2008 PSF yes + 2.6.1 2.6 2008 PSF yes + 2.6.2 2.6.1 2009 PSF yes + 2.6.3 2.6.2 2009 PSF yes + 2.6.4 2.6.3 2009 PSF yes + 2.6.5 2.6.4 2010 PSF yes + 3.0 2.6 2008 PSF yes + 3.0.1 3.0 2009 PSF yes + 3.1 3.0.1 2009 PSF yes + 3.1.1 3.1 2009 PSF yes + 3.1.2 3.1 2010 PSF yes + 3.2 3.1 2010 PSF yes + +Footnotes: + +(1) GPL-compatible doesn't mean that we're distributing Python under + the GPL. All Python licenses, unlike the GPL, let you distribute + a modified version without making your changes open source. The + GPL-compatible licenses make it possible to combine Python with + other software that is released under the GPL; the others don't. + +(2) According to Richard Stallman, 1.6.1 is not GPL-compatible, + because its license has a choice of law clause. According to + CNRI, however, Stallman's lawyer has told CNRI's lawyer that 1.6.1 + is "not incompatible" with the GPL. + +Thanks to the many outside volunteers who have worked under Guido's +direction to make these releases possible. + + +B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON +=============================================================== + +PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 +-------------------------------------------- + +1. This LICENSE AGREEMENT is between the Python Software Foundation +("PSF"), and the Individual or Organization ("Licensee") accessing and +otherwise using this software ("Python") in source or binary form and +its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, PSF hereby +grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, +analyze, test, perform and/or display publicly, prepare derivative works, +distribute, and otherwise use Python alone or in any derivative version, +provided, however, that PSF's License Agreement and PSF's notice of copyright, +i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 +Python Software Foundation; All Rights Reserved" are retained in Python alone or +in any derivative version prepared by Licensee. + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python. + +4. PSF is making Python available to Licensee on an "AS IS" +basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. Nothing in this License Agreement shall be deemed to create any +relationship of agency, partnership, or joint venture between PSF and +Licensee. This License Agreement does not grant permission to use PSF +trademarks or trade name in a trademark sense to endorse or promote +products or services of Licensee, or any third party. + +8. By copying, installing or otherwise using Python, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0 +------------------------------------------- + +BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1 + +1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an +office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the +Individual or Organization ("Licensee") accessing and otherwise using +this software in source or binary form and its associated +documentation ("the Software"). + +2. Subject to the terms and conditions of this BeOpen Python License +Agreement, BeOpen hereby grants Licensee a non-exclusive, +royalty-free, world-wide license to reproduce, analyze, test, perform +and/or display publicly, prepare derivative works, distribute, and +otherwise use the Software alone or in any derivative version, +provided, however, that the BeOpen Python License is retained in the +Software, alone or in any derivative version prepared by Licensee. + +3. BeOpen is making the Software available to Licensee on an "AS IS" +basis. BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE +SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS +AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY +DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +5. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +6. This License Agreement shall be governed by and interpreted in all +respects by the law of the State of California, excluding conflict of +law provisions. Nothing in this License Agreement shall be deemed to +create any relationship of agency, partnership, or joint venture +between BeOpen and Licensee. This License Agreement does not grant +permission to use BeOpen trademarks or trade names in a trademark +sense to endorse or promote products or services of Licensee, or any +third party. As an exception, the "BeOpen Python" logos available at +http://www.pythonlabs.com/logos.html may be used according to the +permissions granted on that web page. + +7. By copying, installing or otherwise using the software, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1 +--------------------------------------- + +1. This LICENSE AGREEMENT is between the Corporation for National +Research Initiatives, having an office at 1895 Preston White Drive, +Reston, VA 20191 ("CNRI"), and the Individual or Organization +("Licensee") accessing and otherwise using Python 1.6.1 software in +source or binary form and its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, CNRI +hereby grants Licensee a nonexclusive, royalty-free, world-wide +license to reproduce, analyze, test, perform and/or display publicly, +prepare derivative works, distribute, and otherwise use Python 1.6.1 +alone or in any derivative version, provided, however, that CNRI's +License Agreement and CNRI's notice of copyright, i.e., "Copyright (c) +1995-2001 Corporation for National Research Initiatives; All Rights +Reserved" are retained in Python 1.6.1 alone or in any derivative +version prepared by Licensee. Alternately, in lieu of CNRI's License +Agreement, Licensee may substitute the following text (omitting the +quotes): "Python 1.6.1 is made available subject to the terms and +conditions in CNRI's License Agreement. This Agreement together with +Python 1.6.1 may be located on the Internet using the following +unique, persistent identifier (known as a handle): 1895.22/1013. This +Agreement may also be obtained from a proxy server on the Internet +using the following URL: http://hdl.handle.net/1895.22/1013". + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python 1.6.1 or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python 1.6.1. + +4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS" +basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1 WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. This License Agreement shall be governed by the federal +intellectual property law of the United States, including without +limitation the federal copyright law, and, to the extent such +U.S. federal law does not apply, by the law of the Commonwealth of +Virginia, excluding Virginia's conflict of law provisions. +Notwithstanding the foregoing, with regard to derivative works based +on Python 1.6.1 that incorporate non-separable material that was +previously distributed under the GNU General Public License (GPL), the +law of the Commonwealth of Virginia shall govern this License +Agreement only as to issues arising under or with respect to +Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this +License Agreement shall be deemed to create any relationship of +agency, partnership, or joint venture between CNRI and Licensee. This +License Agreement does not grant permission to use CNRI trademarks or +trade name in a trademark sense to endorse or promote products or +services of Licensee, or any third party. + +8. By clicking on the "ACCEPT" button where indicated, or by copying, +installing or otherwise using Python 1.6.1, Licensee agrees to be +bound by the terms and conditions of this License Agreement. + + ACCEPT + + +CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2 +-------------------------------------------------- + +Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, +The Netherlands. All rights reserved. + +Permission to use, copy, modify, and distribute this software and its +documentation for any purpose and without fee is hereby granted, +provided that the above copyright notice appear in all copies and that +both that copyright notice and this permission notice appear in +supporting documentation, and that the name of Stichting Mathematisch +Centrum or CWI not be used in advertising or publicity pertaining to +distribution of the software without specific, written prior +permission. + +STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO +THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND +FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE +FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT +OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + + +distro +Apache Software License +https://github.com/python-distro/distro +Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +dm-tree +Apache Software License +https://github.com/deepmind/tree + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +docstring_parser +MIT License +https://github.com/rr-/docstring_parser +The MIT License (MIT) + +Copyright (c) 2018 Marcin Kurczewski + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +donfig +MIT License +https://github.com/pytroll/donfig +Copyright (c) 2018- Donfig Developers +Copyright (c) 2014-2018, Anaconda, Inc. and contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +einops +MIT License +https://github.com/arogozhnikov/einops +MIT License + +Copyright (c) 2018 Alex Rogozhnikov + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +einx +MIT +https://github.com/fferflo/einx +MIT License + +Copyright (c) 2023- Florian Fervers + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +evdev +BSD-3-Clause +https://github.com/gvalkov/python-evdev +Copyright (c) 2012-2025 Georgi Valkov. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + 1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + + 3. Neither the name of author nor the names of its contributors may + be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +execnet +MIT +https://execnet.readthedocs.io/en/latest/ + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + + +executing +MIT License +https://github.com/alexmojaki/executing +MIT License + +Copyright (c) 2019 Alex Hall + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +fastapi +MIT +https://github.com/fastapi/fastapi +The MIT License (MIT) + +Copyright (c) 2018 Sebastián Ramírez + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +fastjsonschema +BSD License +https://github.com/horejsek/python-fastjsonschema +Copyright (c) 2018, Michal Horejsek +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + + Redistributions in binary form must reproduce the above copyright notice, this + list of conditions and the following disclaimer in the documentation and/or + other materials provided with the distribution. + + Neither the name of the {organization} nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +fastparquet +Apache Software License +https://github.com/dask/fastparquet/ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + +ffmpy +MIT +UNKNOWN +UNKNOWN + +filelock +MIT +https://github.com/tox-dev/py-filelock +MIT License + +Copyright (c) 2025 Bernát Gábor and contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +flopth +UNKNOWN +UNKNOWN +MIT License + +Copyright (c) 2019 Yunfeng Wang + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +fonttools +MIT +http://github.com/fonttools/fonttools +MIT License + +Copyright (c) 2017 Just van Rossum + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +fqdn +Mozilla Public License 2.0 (MPL 2.0) +https://github.com/ypcrts/fqdn +Mozilla Public License Version 2.0 +================================== + +1. Definitions +-------------- + +1.1. "Contributor" + means each individual or legal entity that creates, contributes to + the creation of, or owns Covered Software. + +1.2. "Contributor Version" + means the combination of the Contributions of others (if any) used + by a Contributor and that particular Contributor's Contribution. + +1.3. "Contribution" + means Covered Software of a particular Contributor. + +1.4. "Covered Software" + means Source Code Form to which the initial Contributor has attached + the notice in Exhibit A, the Executable Form of such Source Code + Form, and Modifications of such Source Code Form, in each case + including portions thereof. + +1.5. "Incompatible With Secondary Licenses" + means + + (a) that the initial Contributor has attached the notice described + in Exhibit B to the Covered Software; or + + (b) that the Covered Software was made available under the terms of + version 1.1 or earlier of the License, but not also under the + terms of a Secondary License. + +1.6. "Executable Form" + means any form of the work other than Source Code Form. + +1.7. "Larger Work" + means a work that combines Covered Software with other material, in + a separate file or files, that is not Covered Software. + +1.8. "License" + means this document. + +1.9. "Licensable" + means having the right to grant, to the maximum extent possible, + whether at the time of the initial grant or subsequently, any and + all of the rights conveyed by this License. + +1.10. "Modifications" + means any of the following: + + (a) any file in Source Code Form that results from an addition to, + deletion from, or modification of the contents of Covered + Software; or + + (b) any new file in Source Code Form that contains any Covered + Software. + +1.11. "Patent Claims" of a Contributor + means any patent claim(s), including without limitation, method, + process, and apparatus claims, in any patent Licensable by such + Contributor that would be infringed, but for the grant of the + License, by the making, using, selling, offering for sale, having + made, import, or transfer of either its Contributions or its + Contributor Version. + +1.12. "Secondary License" + means either the GNU General Public License, Version 2.0, the GNU + Lesser General Public License, Version 2.1, the GNU Affero General + Public License, Version 3.0, or any later versions of those + licenses. + +1.13. "Source Code Form" + means the form of the work preferred for making modifications. + +1.14. "You" (or "Your") + means an individual or a legal entity exercising rights under this + License. For legal entities, "You" includes any entity that + controls, is controlled by, or is under common control with You. For + purposes of this definition, "control" means (a) the power, direct + or indirect, to cause the direction or management of such entity, + whether by contract or otherwise, or (b) ownership of more than + fifty percent (50%) of the outstanding shares or beneficial + ownership of such entity. + +2. License Grants and Conditions +-------------------------------- + +2.1. Grants + +Each Contributor hereby grants You a world-wide, royalty-free, +non-exclusive license: + +(a) under intellectual property rights (other than patent or trademark) + Licensable by such Contributor to use, reproduce, make available, + modify, display, perform, distribute, and otherwise exploit its + Contributions, either on an unmodified basis, with Modifications, or + as part of a Larger Work; and + +(b) under Patent Claims of such Contributor to make, use, sell, offer + for sale, have made, import, and otherwise transfer either its + Contributions or its Contributor Version. + +2.2. Effective Date + +The licenses granted in Section 2.1 with respect to any Contribution +become effective for each Contribution on the date the Contributor first +distributes such Contribution. + +2.3. Limitations on Grant Scope + +The licenses granted in this Section 2 are the only rights granted under +this License. No additional rights or licenses will be implied from the +distribution or licensing of Covered Software under this License. +Notwithstanding Section 2.1(b) above, no patent license is granted by a +Contributor: + +(a) for any code that a Contributor has removed from Covered Software; + or + +(b) for infringements caused by: (i) Your and any other third party's + modifications of Covered Software, or (ii) the combination of its + Contributions with other software (except as part of its Contributor + Version); or + +(c) under Patent Claims infringed by Covered Software in the absence of + its Contributions. + +This License does not grant any rights in the trademarks, service marks, +or logos of any Contributor (except as may be necessary to comply with +the notice requirements in Section 3.4). + +2.4. Subsequent Licenses + +No Contributor makes additional grants as a result of Your choice to +distribute the Covered Software under a subsequent version of this +License (see Section 10.2) or under the terms of a Secondary License (if +permitted under the terms of Section 3.3). + +2.5. Representation + +Each Contributor represents that the Contributor believes its +Contributions are its original creation(s) or it has sufficient rights +to grant the rights to its Contributions conveyed by this License. + +2.6. Fair Use + +This License is not intended to limit any rights You have under +applicable copyright doctrines of fair use, fair dealing, or other +equivalents. + +2.7. Conditions + +Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted +in Section 2.1. + +3. Responsibilities +------------------- + +3.1. Distribution of Source Form + +All distribution of Covered Software in Source Code Form, including any +Modifications that You create or to which You contribute, must be under +the terms of this License. You must inform recipients that the Source +Code Form of the Covered Software is governed by the terms of this +License, and how they can obtain a copy of this License. You may not +attempt to alter or restrict the recipients' rights in the Source Code +Form. + +3.2. Distribution of Executable Form + +If You distribute Covered Software in Executable Form then: + +(a) such Covered Software must also be made available in Source Code + Form, as described in Section 3.1, and You must inform recipients of + the Executable Form how they can obtain a copy of such Source Code + Form by reasonable means in a timely manner, at a charge no more + than the cost of distribution to the recipient; and + +(b) You may distribute such Executable Form under the terms of this + License, or sublicense it under different terms, provided that the + license for the Executable Form does not attempt to limit or alter + the recipients' rights in the Source Code Form under this License. + +3.3. Distribution of a Larger Work + +You may create and distribute a Larger Work under terms of Your choice, +provided that You also comply with the requirements of this License for +the Covered Software. If the Larger Work is a combination of Covered +Software with a work governed by one or more Secondary Licenses, and the +Covered Software is not Incompatible With Secondary Licenses, this +License permits You to additionally distribute such Covered Software +under the terms of such Secondary License(s), so that the recipient of +the Larger Work may, at their option, further distribute the Covered +Software under the terms of either this License or such Secondary +License(s). + +3.4. Notices + +You may not remove or alter the substance of any license notices +(including copyright notices, patent notices, disclaimers of warranty, +or limitations of liability) contained within the Source Code Form of +the Covered Software, except that You may alter any license notices to +the extent required to remedy known factual inaccuracies. + +3.5. Application of Additional Terms + +You may choose to offer, and to charge a fee for, warranty, support, +indemnity or liability obligations to one or more recipients of Covered +Software. However, You may do so only on Your own behalf, and not on +behalf of any Contributor. You must make it absolutely clear that any +such warranty, support, indemnity, or liability obligation is offered by +You alone, and You hereby agree to indemnify every Contributor for any +liability incurred by such Contributor as a result of warranty, support, +indemnity or liability terms You offer. You may include additional +disclaimers of warranty and limitations of liability specific to any +jurisdiction. + +4. Inability to Comply Due to Statute or Regulation +--------------------------------------------------- + +If it is impossible for You to comply with any of the terms of this +License with respect to some or all of the Covered Software due to +statute, judicial order, or regulation then You must: (a) comply with +the terms of this License to the maximum extent possible; and (b) +describe the limitations and the code they affect. Such description must +be placed in a text file included with all distributions of the Covered +Software under this License. Except to the extent prohibited by statute +or regulation, such description must be sufficiently detailed for a +recipient of ordinary skill to be able to understand it. + +5. Termination +-------------- + +5.1. The rights granted under this License will terminate automatically +if You fail to comply with any of its terms. However, if You become +compliant, then the rights granted under this License from a particular +Contributor are reinstated (a) provisionally, unless and until such +Contributor explicitly and finally terminates Your grants, and (b) on an +ongoing basis, if such Contributor fails to notify You of the +non-compliance by some reasonable means prior to 60 days after You have +come back into compliance. Moreover, Your grants from a particular +Contributor are reinstated on an ongoing basis if such Contributor +notifies You of the non-compliance by some reasonable means, this is the +first time You have received notice of non-compliance with this License +from such Contributor, and You become compliant prior to 30 days after +Your receipt of the notice. + +5.2. If You initiate litigation against any entity by asserting a patent +infringement claim (excluding declaratory judgment actions, +counter-claims, and cross-claims) alleging that a Contributor Version +directly or indirectly infringes any patent, then the rights granted to +You by any and all Contributors for the Covered Software under Section +2.1 of this License shall terminate. + +5.3. In the event of termination under Sections 5.1 or 5.2 above, all +end user license agreements (excluding distributors and resellers) which +have been validly granted by You or Your distributors under this License +prior to termination shall survive termination. + +************************************************************************ +* * +* 6. Disclaimer of Warranty * +* ------------------------- * +* * +* Covered Software is provided under this License on an "as is" * +* basis, without warranty of any kind, either expressed, implied, or * +* statutory, including, without limitation, warranties that the * +* Covered Software is free of defects, merchantable, fit for a * +* particular purpose or non-infringing. The entire risk as to the * +* quality and performance of the Covered Software is with You. * +* Should any Covered Software prove defective in any respect, You * +* (not any Contributor) assume the cost of any necessary servicing, * +* repair, or correction. This disclaimer of warranty constitutes an * +* essential part of this License. No use of any Covered Software is * +* authorized under this License except under this disclaimer. * +* * +************************************************************************ + +************************************************************************ +* * +* 7. Limitation of Liability * +* -------------------------- * +* * +* Under no circumstances and under no legal theory, whether tort * +* (including negligence), contract, or otherwise, shall any * +* Contributor, or anyone who distributes Covered Software as * +* permitted above, be liable to You for any direct, indirect, * +* special, incidental, or consequential damages of any character * +* including, without limitation, damages for lost profits, loss of * +* goodwill, work stoppage, computer failure or malfunction, or any * +* and all other commercial damages or losses, even if such party * +* shall have been informed of the possibility of such damages. This * +* limitation of liability shall not apply to liability for death or * +* personal injury resulting from such party's negligence to the * +* extent applicable law prohibits such limitation. Some * +* jurisdictions do not allow the exclusion or limitation of * +* incidental or consequential damages, so this exclusion and * +* limitation may not apply to You. * +* * +************************************************************************ + +8. Litigation +------------- + +Any litigation relating to this License may be brought only in the +courts of a jurisdiction where the defendant maintains its principal +place of business and such litigation shall be governed by laws of that +jurisdiction, without reference to its conflict-of-law provisions. +Nothing in this Section shall prevent a party's ability to bring +cross-claims or counter-claims. + +9. Miscellaneous +---------------- + +This License represents the complete agreement concerning the subject +matter hereof. If any provision of this License is held to be +unenforceable, such provision shall be reformed only to the extent +necessary to make it enforceable. Any law or regulation which provides +that the language of a contract shall be construed against the drafter +shall not be used to construe this License against a Contributor. + +10. Versions of the License +--------------------------- + +10.1. New Versions + +Mozilla Foundation is the license steward. Except as provided in Section +10.3, no one other than the license steward has the right to modify or +publish new versions of this License. Each version will be given a +distinguishing version number. + +10.2. Effect of New Versions + +You may distribute the Covered Software under the terms of the version +of the License under which You originally received the Covered Software, +or under the terms of any subsequent version published by the license +steward. + +10.3. Modified Versions + +If you create software not governed by this License, and you want to +create a new license for such software, you may create and use a +modified version of this License if you rename the license and remove +any references to the name of the license steward (except to note that +such modified license differs from this License). + +10.4. Distributing Source Code Form that is Incompatible With Secondary +Licenses + +If You choose to distribute Source Code Form that is Incompatible With +Secondary Licenses under the terms of this version of the License, the +notice described in Exhibit B of this License must be attached. + +Exhibit A - Source Code Form License Notice +------------------------------------------- + + This Source Code Form is subject to the terms of the Mozilla Public + License, v. 2.0. If a copy of the MPL was not distributed with this + file, You can obtain one at http://mozilla.org/MPL/2.0/. + +If it is not possible or desirable to put the notice in a particular +file, then You may include the notice in a location (such as a LICENSE +file in a relevant directory) where a recipient would be likely to look +for such a notice. + +You may add additional accurate notices of copyright ownership. + +Exhibit B - "Incompatible With Secondary Licenses" Notice +--------------------------------------------------------- + + This Source Code Form is "Incompatible With Secondary Licenses", as + defined by the Mozilla Public License, v. 2.0. + + +frozendict +GNU Lesser General Public License v3 (LGPLv3) +https://github.com/Marco-Sulla/python-frozendict + GNU LESSER GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + + This version of the GNU Lesser General Public License incorporates +the terms and conditions of version 3 of the GNU General Public +License, supplemented by the additional permissions listed below. + + 0. Additional Definitions. + + As used herein, "this License" refers to version 3 of the GNU Lesser +General Public License, and the "GNU GPL" refers to version 3 of the GNU +General Public License. + + "The Library" refers to a covered work governed by this License, +other than an Application or a Combined Work as defined below. + + An "Application" is any work that makes use of an interface provided +by the Library, but which is not otherwise based on the Library. +Defining a subclass of a class defined by the Library is deemed a mode +of using an interface provided by the Library. + + A "Combined Work" is a work produced by combining or linking an +Application with the Library. The particular version of the Library +with which the Combined Work was made is also called the "Linked +Version". + + The "Minimal Corresponding Source" for a Combined Work means the +Corresponding Source for the Combined Work, excluding any source code +for portions of the Combined Work that, considered in isolation, are +based on the Application, and not on the Linked Version. + + The "Corresponding Application Code" for a Combined Work means the +object code and/or source code for the Application, including any data +and utility programs needed for reproducing the Combined Work from the +Application, but excluding the System Libraries of the Combined Work. + + 1. Exception to Section 3 of the GNU GPL. + + You may convey a covered work under sections 3 and 4 of this License +without being bound by section 3 of the GNU GPL. + + 2. Conveying Modified Versions. + + If you modify a copy of the Library, and, in your modifications, a +facility refers to a function or data to be supplied by an Application +that uses the facility (other than as an argument passed when the +facility is invoked), then you may convey a copy of the modified +version: + + a) under this License, provided that you make a good faith effort to + ensure that, in the event an Application does not supply the + function or data, the facility still operates, and performs + whatever part of its purpose remains meaningful, or + + b) under the GNU GPL, with none of the additional permissions of + this License applicable to that copy. + + 3. Object Code Incorporating Material from Library Header Files. + + The object code form of an Application may incorporate material from +a header file that is part of the Library. You may convey such object +code under terms of your choice, provided that, if the incorporated +material is not limited to numerical parameters, data structure +layouts and accessors, or small macros, inline functions and templates +(ten or fewer lines in length), you do both of the following: + + a) Give prominent notice with each copy of the object code that the + Library is used in it and that the Library and its use are + covered by this License. + + b) Accompany the object code with a copy of the GNU GPL and this license + document. + + 4. Combined Works. + + You may convey a Combined Work under terms of your choice that, +taken together, effectively do not restrict modification of the +portions of the Library contained in the Combined Work and reverse +engineering for debugging such modifications, if you also do each of +the following: + + a) Give prominent notice with each copy of the Combined Work that + the Library is used in it and that the Library and its use are + covered by this License. + + b) Accompany the Combined Work with a copy of the GNU GPL and this license + document. + + c) For a Combined Work that displays copyright notices during + execution, include the copyright notice for the Library among + these notices, as well as a reference directing the user to the + copies of the GNU GPL and this license document. + + d) Do one of the following: + + 0) Convey the Minimal Corresponding Source under the terms of this + License, and the Corresponding Application Code in a form + suitable for, and under terms that permit, the user to + recombine or relink the Application with a modified version of + the Linked Version to produce a modified Combined Work, in the + manner specified by section 6 of the GNU GPL for conveying + Corresponding Source. + + 1) Use a suitable shared library mechanism for linking with the + Library. A suitable mechanism is one that (a) uses at run time + a copy of the Library already present on the user's computer + system, and (b) will operate properly with a modified version + of the Library that is interface-compatible with the Linked + Version. + + e) Provide Installation Information, but only if you would otherwise + be required to provide such information under section 6 of the + GNU GPL, and only to the extent that such information is + necessary to install and execute a modified version of the + Combined Work produced by recombining or relinking the + Application with a modified version of the Linked Version. (If + you use option 4d0, the Installation Information must accompany + the Minimal Corresponding Source and Corresponding Application + Code. If you use option 4d1, you must provide the Installation + Information in the manner specified by section 6 of the GNU GPL + for conveying Corresponding Source.) + + 5. Combined Libraries. + + You may place library facilities that are a work based on the +Library side by side in a single library together with other library +facilities that are not Applications and are not covered by this +License, and convey such a combined library under terms of your +choice, if you do both of the following: + + a) Accompany the combined library with a copy of the same work based + on the Library, uncombined with any other library facilities, + conveyed under the terms of this License. + + b) Give prominent notice with the combined library that part of it + is a work based on the Library, and explaining where to find the + accompanying uncombined form of the same work. + + 6. Revised Versions of the GNU Lesser General Public License. + + The Free Software Foundation may publish revised and/or new versions +of the GNU Lesser General Public License from time to time. Such new +versions will be similar in spirit to the present version, but may +differ in detail to address new problems or concerns. + + Each version is given a distinguishing version number. If the +Library as you received it specifies that a certain numbered version +of the GNU Lesser General Public License "or any later version" +applies to it, you have the option of following the terms and +conditions either of that published version or of any later version +published by the Free Software Foundation. If the Library as you +received it does not specify a version number of the GNU Lesser +General Public License, you may choose any version of the GNU Lesser +General Public License ever published by the Free Software Foundation. + + If the Library as you received it specifies that a proxy can decide +whether future versions of the GNU Lesser General Public License shall +apply, that proxy's public statement of acceptance of any version is +permanent authorization for you to choose that version for the +Library. + + +frozenlist +Apache-2.0 +https://github.com/aio-libs/frozenlist +Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2013-2019 Nikolay Kim and Andrew Svetlov + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +fsspec +BSD-3-Clause +https://github.com/fsspec/filesystem_spec +BSD 3-Clause License + +Copyright (c) 2018, Martin Durant +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +ftfy +Apache-2.0 +https://ftfy.readthedocs.io/en/latest/ +Copyright 2023 Robyn Speer + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + + +future +MIT License +https://python-future.org +Copyright (c) 2013-2024 Python Charmers, Australia + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +futureproof +MIT License +https://github.com/yeraydiazdiaz/futureproof +MIT License + +Copyright © 2019, Yeray Díaz Díaz + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +fvcore +Apache 2.0 +https://github.com/facebookresearch/fvcore +UNKNOWN + +gast +BSD License +https://github.com/serge-sans-paille/gast/ +Copyright (c) 2016, Serge Guelton +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + + Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + Neither the name of HPCProject, Serge Guelton nor the names of its + contributors may be used to endorse or promote products derived from this + software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + + + +gitdb +BSD License +https://github.com/gitpython-developers/gitdb +Copyright (C) 2010, 2011 Sebastian Thiel and contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +* Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +* Neither the name of the GitDB project nor the names of +its contributors may be used to endorse or promote products derived +from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +Additional Licenses +------------------- +The files at +gitdb/test/fixtures/packs/pack-11fdfa9e156ab73caae3b6da867192221f2089c2.idx +and +gitdb/test/fixtures/packs/pack-11fdfa9e156ab73caae3b6da867192221f2089c2.pack +are licensed under GNU GPL as part of the git source repository, +see http://en.wikipedia.org/wiki/Git_%28software%29 for more information. + +They are not required for the actual operation, which is why they are not found +in the distribution package. + + +glfw +MIT License +https://github.com/FlorianRhiem/pyGLFW +The MIT License (MIT) + +Copyright (c) 2013-2019 Florian Rhiem + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +google-api-core +Apache Software License +https://github.com/googleapis/google-cloud-python/tree/main/packages/google-api-core + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +google-auth +Apache Software License +https://github.com/googleapis/google-auth-library-python + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +google-cloud-core +Apache Software License +https://github.com/googleapis/python-cloud-core + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +google-cloud-storage +Apache Software License +https://github.com/googleapis/python-storage + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +google-crc32c +UNKNOWN +https://github.com/googleapis/python-crc32c + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +google-resumable-media +Apache Software License +https://github.com/googleapis/google-resumable-media-python + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +googleapis-common-protos +Apache Software License +https://github.com/googleapis/google-cloud-python/tree/main/packages/googleapis-common-protos + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +gradio +Apache-2.0 +https://github.com/gradio-app/gradio + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +gradio_client +Apache-2.0 +https://github.com/gradio-app/gradio +UNKNOWN + +groovy +MIT License +https://github.com/gradio-app/groovy + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +grpcio +Apache-2.0 +https://grpc.io + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +----------------------------------------------------------- + +BSD 3-Clause License + +Copyright 2016, Google Inc. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, +this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, +this list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its +contributors may be used to endorse or promote products derived from this +software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF +THE POSSIBILITY OF SUCH DAMAGE. + +----------------------------------------------------------- + +Mozilla Public License Version 2.0 +================================== + +1. Definitions +-------------- + +1.1. "Contributor" + means each individual or legal entity that creates, contributes to + the creation of, or owns Covered Software. + +1.2. "Contributor Version" + means the combination of the Contributions of others (if any) used + by a Contributor and that particular Contributor's Contribution. + +1.3. "Contribution" + means Covered Software of a particular Contributor. + +1.4. "Covered Software" + means Source Code Form to which the initial Contributor has attached + the notice in Exhibit A, the Executable Form of such Source Code + Form, and Modifications of such Source Code Form, in each case + including portions thereof. + +1.5. "Incompatible With Secondary Licenses" + means + + (a) that the initial Contributor has attached the notice described + in Exhibit B to the Covered Software; or + + (b) that the Covered Software was made available under the terms of + version 1.1 or earlier of the License, but not also under the + terms of a Secondary License. + +1.6. "Executable Form" + means any form of the work other than Source Code Form. + +1.7. "Larger Work" + means a work that combines Covered Software with other material, in + a separate file or files, that is not Covered Software. + +1.8. "License" + means this document. + +1.9. "Licensable" + means having the right to grant, to the maximum extent possible, + whether at the time of the initial grant or subsequently, any and + all of the rights conveyed by this License. + +1.10. "Modifications" + means any of the following: + + (a) any file in Source Code Form that results from an addition to, + deletion from, or modification of the contents of Covered + Software; or + + (b) any new file in Source Code Form that contains any Covered + Software. + +1.11. "Patent Claims" of a Contributor + means any patent claim(s), including without limitation, method, + process, and apparatus claims, in any patent Licensable by such + Contributor that would be infringed, but for the grant of the + License, by the making, using, selling, offering for sale, having + made, import, or transfer of either its Contributions or its + Contributor Version. + +1.12. "Secondary License" + means either the GNU General Public License, Version 2.0, the GNU + Lesser General Public License, Version 2.1, the GNU Affero General + Public License, Version 3.0, or any later versions of those + licenses. + +1.13. "Source Code Form" + means the form of the work preferred for making modifications. + +1.14. "You" (or "Your") + means an individual or a legal entity exercising rights under this + License. For legal entities, "You" includes any entity that + controls, is controlled by, or is under common control with You. For + purposes of this definition, "control" means (a) the power, direct + or indirect, to cause the direction or management of such entity, + whether by contract or otherwise, or (b) ownership of more than + fifty percent (50%) of the outstanding shares or beneficial + ownership of such entity. + +2. License Grants and Conditions +-------------------------------- + +2.1. Grants + +Each Contributor hereby grants You a world-wide, royalty-free, +non-exclusive license: + +(a) under intellectual property rights (other than patent or trademark) + Licensable by such Contributor to use, reproduce, make available, + modify, display, perform, distribute, and otherwise exploit its + Contributions, either on an unmodified basis, with Modifications, or + as part of a Larger Work; and + +(b) under Patent Claims of such Contributor to make, use, sell, offer + for sale, have made, import, and otherwise transfer either its + Contributions or its Contributor Version. + +2.2. Effective Date + +The licenses granted in Section 2.1 with respect to any Contribution +become effective for each Contribution on the date the Contributor first +distributes such Contribution. + +2.3. Limitations on Grant Scope + +The licenses granted in this Section 2 are the only rights granted under +this License. No additional rights or licenses will be implied from the +distribution or licensing of Covered Software under this License. +Notwithstanding Section 2.1(b) above, no patent license is granted by a +Contributor: + +(a) for any code that a Contributor has removed from Covered Software; + or + +(b) for infringements caused by: (i) Your and any other third party's + modifications of Covered Software, or (ii) the combination of its + Contributions with other software (except as part of its Contributor + Version); or + +(c) under Patent Claims infringed by Covered Software in the absence of + its Contributions. + +This License does not grant any rights in the trademarks, service marks, +or logos of any Contributor (except as may be necessary to comply with +the notice requirements in Section 3.4). + +2.4. Subsequent Licenses + +No Contributor makes additional grants as a result of Your choice to +distribute the Covered Software under a subsequent version of this +License (see Section 10.2) or under the terms of a Secondary License (if +permitted under the terms of Section 3.3). + +2.5. Representation + +Each Contributor represents that the Contributor believes its +Contributions are its original creation(s) or it has sufficient rights +to grant the rights to its Contributions conveyed by this License. + +2.6. Fair Use + +This License is not intended to limit any rights You have under +applicable copyright doctrines of fair use, fair dealing, or other +equivalents. + +2.7. Conditions + +Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted +in Section 2.1. + +3. Responsibilities +------------------- + +3.1. Distribution of Source Form + +All distribution of Covered Software in Source Code Form, including any +Modifications that You create or to which You contribute, must be under +the terms of this License. You must inform recipients that the Source +Code Form of the Covered Software is governed by the terms of this +License, and how they can obtain a copy of this License. You may not +attempt to alter or restrict the recipients' rights in the Source Code +Form. + +3.2. Distribution of Executable Form + +If You distribute Covered Software in Executable Form then: + +(a) such Covered Software must also be made available in Source Code + Form, as described in Section 3.1, and You must inform recipients of + the Executable Form how they can obtain a copy of such Source Code + Form by reasonable means in a timely manner, at a charge no more + than the cost of distribution to the recipient; and + +(b) You may distribute such Executable Form under the terms of this + License, or sublicense it under different terms, provided that the + license for the Executable Form does not attempt to limit or alter + the recipients' rights in the Source Code Form under this License. + +3.3. Distribution of a Larger Work + +You may create and distribute a Larger Work under terms of Your choice, +provided that You also comply with the requirements of this License for +the Covered Software. If the Larger Work is a combination of Covered +Software with a work governed by one or more Secondary Licenses, and the +Covered Software is not Incompatible With Secondary Licenses, this +License permits You to additionally distribute such Covered Software +under the terms of such Secondary License(s), so that the recipient of +the Larger Work may, at their option, further distribute the Covered +Software under the terms of either this License or such Secondary +License(s). + +3.4. Notices + +You may not remove or alter the substance of any license notices +(including copyright notices, patent notices, disclaimers of warranty, +or limitations of liability) contained within the Source Code Form of +the Covered Software, except that You may alter any license notices to +the extent required to remedy known factual inaccuracies. + +3.5. Application of Additional Terms + +You may choose to offer, and to charge a fee for, warranty, support, +indemnity or liability obligations to one or more recipients of Covered +Software. However, You may do so only on Your own behalf, and not on +behalf of any Contributor. You must make it absolutely clear that any +such warranty, support, indemnity, or liability obligation is offered by +You alone, and You hereby agree to indemnify every Contributor for any +liability incurred by such Contributor as a result of warranty, support, +indemnity or liability terms You offer. You may include additional +disclaimers of warranty and limitations of liability specific to any +jurisdiction. + +4. Inability to Comply Due to Statute or Regulation +--------------------------------------------------- + +If it is impossible for You to comply with any of the terms of this +License with respect to some or all of the Covered Software due to +statute, judicial order, or regulation then You must: (a) comply with +the terms of this License to the maximum extent possible; and (b) +describe the limitations and the code they affect. Such description must +be placed in a text file included with all distributions of the Covered +Software under this License. Except to the extent prohibited by statute +or regulation, such description must be sufficiently detailed for a +recipient of ordinary skill to be able to understand it. + +5. Termination +-------------- + +5.1. The rights granted under this License will terminate automatically +if You fail to comply with any of its terms. However, if You become +compliant, then the rights granted under this License from a particular +Contributor are reinstated (a) provisionally, unless and until such +Contributor explicitly and finally terminates Your grants, and (b) on an +ongoing basis, if such Contributor fails to notify You of the +non-compliance by some reasonable means prior to 60 days after You have +come back into compliance. Moreover, Your grants from a particular +Contributor are reinstated on an ongoing basis if such Contributor +notifies You of the non-compliance by some reasonable means, this is the +first time You have received notice of non-compliance with this License +from such Contributor, and You become compliant prior to 30 days after +Your receipt of the notice. + +5.2. If You initiate litigation against any entity by asserting a patent +infringement claim (excluding declaratory judgment actions, +counter-claims, and cross-claims) alleging that a Contributor Version +directly or indirectly infringes any patent, then the rights granted to +You by any and all Contributors for the Covered Software under Section +2.1 of this License shall terminate. + +5.3. In the event of termination under Sections 5.1 or 5.2 above, all +end user license agreements (excluding distributors and resellers) which +have been validly granted by You or Your distributors under this License +prior to termination shall survive termination. + +************************************************************************ +* * +* 6. Disclaimer of Warranty * +* ------------------------- * +* * +* Covered Software is provided under this License on an "as is" * +* basis, without warranty of any kind, either expressed, implied, or * +* statutory, including, without limitation, warranties that the * +* Covered Software is free of defects, merchantable, fit for a * +* particular purpose or non-infringing. The entire risk as to the * +* quality and performance of the Covered Software is with You. * +* Should any Covered Software prove defective in any respect, You * +* (not any Contributor) assume the cost of any necessary servicing, * +* repair, or correction. This disclaimer of warranty constitutes an * +* essential part of this License. No use of any Covered Software is * +* authorized under this License except under this disclaimer. * +* * +************************************************************************ + +************************************************************************ +* * +* 7. Limitation of Liability * +* -------------------------- * +* * +* Under no circumstances and under no legal theory, whether tort * +* (including negligence), contract, or otherwise, shall any * +* Contributor, or anyone who distributes Covered Software as * +* permitted above, be liable to You for any direct, indirect, * +* special, incidental, or consequential damages of any character * +* including, without limitation, damages for lost profits, loss of * +* goodwill, work stoppage, computer failure or malfunction, or any * +* and all other commercial damages or losses, even if such party * +* shall have been informed of the possibility of such damages. This * +* limitation of liability shall not apply to liability for death or * +* personal injury resulting from such party's negligence to the * +* extent applicable law prohibits such limitation. Some * +* jurisdictions do not allow the exclusion or limitation of * +* incidental or consequential damages, so this exclusion and * +* limitation may not apply to You. * +* * +************************************************************************ + +8. Litigation +------------- + +Any litigation relating to this License may be brought only in the +courts of a jurisdiction where the defendant maintains its principal +place of business and such litigation shall be governed by laws of that +jurisdiction, without reference to its conflict-of-law provisions. +Nothing in this Section shall prevent a party's ability to bring +cross-claims or counter-claims. + +9. Miscellaneous +---------------- + +This License represents the complete agreement concerning the subject +matter hereof. If any provision of this License is held to be +unenforceable, such provision shall be reformed only to the extent +necessary to make it enforceable. Any law or regulation which provides +that the language of a contract shall be construed against the drafter +shall not be used to construe this License against a Contributor. + +10. Versions of the License +--------------------------- + +10.1. New Versions + +Mozilla Foundation is the license steward. Except as provided in Section +10.3, no one other than the license steward has the right to modify or +publish new versions of this License. Each version will be given a +distinguishing version number. + +10.2. Effect of New Versions + +You may distribute the Covered Software under the terms of the version +of the License under which You originally received the Covered Software, +or under the terms of any subsequent version published by the license +steward. + +10.3. Modified Versions + +If you create software not governed by this License, and you want to +create a new license for such software, you may create and use a +modified version of this License if you rename the license and remove +any references to the name of the license steward (except to note that +such modified license differs from this License). + +10.4. Distributing Source Code Form that is Incompatible With Secondary +Licenses + +If You choose to distribute Source Code Form that is Incompatible With +Secondary Licenses under the terms of this version of the License, the +notice described in Exhibit B of this License must be attached. + +Exhibit A - Source Code Form License Notice +------------------------------------------- + + This Source Code Form is subject to the terms of the Mozilla Public + License, v. 2.0. If a copy of the MPL was not distributed with this + file, You can obtain one at http://mozilla.org/MPL/2.0/. + +If it is not possible or desirable to put the notice in a particular +file, then You may include the notice in a location (such as a LICENSE +file in a relevant directory) where a recipient would be likely to look +for such a notice. + +You may add additional accurate notices of copyright ownership. + +Exhibit B - "Incompatible With Secondary Licenses" Notice +--------------------------------------------------------- + + This Source Code Form is "Incompatible With Secondary Licenses", as + defined by the Mozilla Public License, v. 2.0. + + +h11 +MIT License +https://github.com/python-hyper/h11 +The MIT License (MIT) + +Copyright (c) 2016 Nathaniel J. Smith and other contributors + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +h5py +BSD-3-Clause +https://www.h5py.org/ +Copyright (c) 2008 Andrew Collette and contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the + distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +hatch +MIT +https://hatch.pypa.io/latest/ +MIT License + +Copyright (c) 2017-present Ofek Lev + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +hatchling +MIT +https://hatch.pypa.io/latest/ +MIT License + +Copyright (c) 2021-present Ofek Lev + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +hf-xet +Apache-2.0 +https://github.com/huggingface/xet-core + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +httpcore +BSD-3-Clause +https://www.encode.io/httpcore/ +Copyright © 2020, [Encode OSS Ltd](https://www.encode.io/). +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +httptools +MIT +https://github.com/MagicStack/httptools +The MIT License + +Copyright (c) 2015 MagicStack Inc. http://magic.io + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +httpx +BSD-3-Clause +https://github.com/encode/httpx +Copyright © 2019, [Encode OSS Ltd](https://www.encode.io/). +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +huggingface_hub +Apache Software License +https://github.com/huggingface/huggingface_hub + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +hvac +Apache Software License +https://github.com/hvac/hvac + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +hydra-core +MIT License +https://github.com/facebookresearch/hydra +MIT License + +Copyright (c) Facebook, Inc. and its affiliates. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +hyperlink +MIT License +https://github.com/python-hyper/hyperlink +Copyright (c) 2017 +Glyph Lefkowitz +Itamar Turner-Trauring +Jean Paul Calderone +Adi Roiban +Amber Hawkie Brown +Mahmoud Hashemi +Wilfredo Sanchez Vega + +and others that have contributed code to the public domain. + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +idna +BSD-3-Clause +https://github.com/kjd/idna +BSD 3-Clause License + +Copyright (c) 2013-2025, Kim Davies and contributors. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +imagecodecs +BSD-3-Clause +https://www.cgohlke.com +BSD-3-Clause license + +Copyright (c) 2008-2026, Christoph Gohlke +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + + +imageio-ffmpeg +BSD License +https://github.com/imageio/imageio-ffmpeg +BSD 2-Clause License + +Copyright (c) 2019-2025, imageio +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +importlib_metadata +Apache-2.0 +https://github.com/python/importlib_metadata +Apache License +Version 2.0, January 2004 +http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + +"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. + +"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. + +"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. + +"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. + +"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. + +"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. + +"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). + +"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. + +"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." + +"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. + +3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. + +4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: + + (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. + + You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. + +5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. + +6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. + +8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. + +END OF TERMS AND CONDITIONS + +APPENDIX: How to apply the Apache License to your work. + +To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. + +Copyright 2025 [name of copyright owner] + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + + +iniconfig +MIT +https://github.com/pytest-dev/iniconfig +The MIT License (MIT) + +Copyright (c) 2010 - 2023 Holger Krekel and others + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +iopath +MIT licensed, as found in the LICENSE file +https://github.com/facebookresearch/iopath +MIT License + +Copyright (c) Facebook, Inc. and its affiliates. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +ipycanvas +BSD License +https://github.com/jupyter-widgets-contrib/ipycanvas +Copyright (c) 2019 Martin Renou +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +ipyevents +BSD License +https://github.com/mwcraig/ipyevents +Copyright (c) 2017, Matt Craig +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +ipykernel +BSD-3-Clause +https://ipython.org +BSD 3-Clause License + +Copyright (c) 2015, IPython Development Team + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +ipython +BSD-3-Clause +https://ipython.org +============================= + The IPython licensing terms +============================= + +IPython is licensed under the terms of the Modified BSD License (also known as +New or Revised or 3-Clause BSD). See the LICENSE file. + + +About the IPython Development Team +---------------------------------- + +Fernando Perez began IPython in 2001 based on code from Janko Hauser + and Nathaniel Gray . Fernando is still +the project lead. + +The IPython Development Team is the set of all contributors to the IPython +project. This includes all of the IPython subprojects. + +The core team that coordinates development on GitHub can be found here: +https://github.com/ipython/. + +Our Copyright Policy +-------------------- + +IPython uses a shared copyright model. Each contributor maintains copyright +over their contributions to IPython. But, it is important to note that these +contributions are typically only changes to the repositories. Thus, the IPython +source code, in its entirety is not the copyright of any single person or +institution. Instead, it is the collective copyright of the entire IPython +Development Team. If individual contributors want to maintain a record of what +changes/contributions they have specific copyright on, they should indicate +their copyright in the commit message of the change, when they commit the +change to one of the IPython repositories. + +With this in mind, the following banner should be used in any source code file +to indicate the copyright and license terms: + +:: + + # Copyright (c) IPython Development Team. + # Distributed under the terms of the Modified BSD License. + + +ipython_pygments_lexers +BSD License +https://github.com/ipython/ipython-pygments-lexers +BSD 3-Clause License + +- Copyright (c) 2012-Present, IPython Development Team + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +ipywidgets +BSD License +http://jupyter.org +Copyright (c) 2015 Project Jupyter Contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +isoduration +ISC License (ISCL) +https://github.com/bolsote/isoduration +Copyright (c) 2020 Víctor Muñoz + +Permission to use, copy, modify, and distribute this software for any +purpose with or without fee is hereby granted, provided that the above +copyright notice and this permission notice appear in all copies. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF +OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + + +itsdangerous +BSD License +https://github.com/pallets/itsdangerous/ +Copyright 2011 Pallets + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +jaraco.classes +MIT License +https://github.com/jaraco/jaraco.classes +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to +deal in the Software without restriction, including without limitation the +rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +sell copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. + + +jaraco.context +MIT +https://github.com/jaraco/jaraco.context +MIT License + +Copyright (c) 2026 + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and +associated documentation files (the "Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the +following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial +portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT +LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO +EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE +USE OR OTHER DEALINGS IN THE SOFTWARE. + + +jaraco.functools +MIT +https://github.com/jaraco/jaraco.functools +MIT License + +Copyright (c) 2025 + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and +associated documentation files (the "Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the +following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial +portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT +LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO +EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE +USE OR OTHER DEALINGS IN THE SOFTWARE. + + +jedi +MIT License +https://github.com/davidhalter/jedi +All contributions towards Jedi are MIT licensed. + +------------------------------------------------------------------------------- +The MIT License (MIT) + +Copyright (c) <2013> + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +jeepney +MIT +https://gitlab.com/takluyver/jeepney +The MIT License (MIT) + +Copyright (c) 2017 Thomas Kluyver + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +jiter +MIT License +https://github.com/pydantic/jiter/ +The MIT License (MIT) + +Copyright (c) 2022 to present Samuel Colvin + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +jmespath +MIT License +https://github.com/jmespath/jmespath.py +MIT License + +Copyright (c) 2013 Amazon.com, Inc. or its affiliates. All Rights Reserved + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +joblib +BSD-3-Clause +https://joblib.readthedocs.io +BSD 3-Clause License + +Copyright (c) 2008-2021, The joblib developers. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +json5 +Apache Software License +https://github.com/dpranke/pyjson5 +Files: Everything except for the benchmarks/*.json files. + +Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +--- + +File: benchmarks/64KB-min.json + +MIT License + +Copyright (c) Microsoft Corporation. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE + +--- + +File: benchmarks/bitly-usa-gov.json + +The MIT License (MIT) + +Copyright (c) 2017 Wes McKinney + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +--- + +File: benchmarks/twitter.json + +The MIT License (MIT) + +Copyright (c) 2014 Milo Yip + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +jsonlines +BSD License +https://github.com/wbolster/jsonlines +*(This is the OSI approved 3-clause "New BSD License".)* + +Copyright © 2016, wouter bolsterlee + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, this + list of conditions and the following disclaimer in the documentation and/or + other materials provided with the distribution. + +* Neither the name of the author nor the names of the contributors may be used + to endorse or promote products derived from this software without specific + prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +jsonpointer +BSD License +https://github.com/stefankoegl/python-json-pointer +Copyright (c) 2011 Stefan Kögl +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. +3. The name of the author may not be used to endorse or promote products + derived from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES +OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. +IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT +NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF +THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + + +jsonschema +MIT +https://github.com/python-jsonschema/jsonschema +Copyright (c) 2013 Julian Berman + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +jsonschema-specifications +MIT +https://github.com/python-jsonschema/jsonschema-specifications +Copyright (c) 2022 Julian Berman + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +jupyter-compare-view +MIT License +UNKNOWN +MIT License + +Copyright (c) 2022 Octoframes + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +jupyter-events +BSD License +http://jupyter.org +BSD 3-Clause License + +Copyright (c) 2022-, Jupyter Development Team + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +jupyter-lsp +BSD License +https://github.com/jupyter-lsp/jupyterlab-lsp/issues +BSD 3-Clause License + +Copyright (c) 2022, jupyter-lsp contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +jupyter_client +BSD License +https://jupyter.org +BSD 3-Clause License + +- Copyright (c) 2001-2015, IPython Development Team +- Copyright (c) 2015-, Jupyter Development Team + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +jupyter_core +BSD-3-Clause +https://jupyter.org +BSD 3-Clause License + +- Copyright (c) 2015-, Jupyter Development Team + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +jupyter_server +BSD License +https://jupyter-server.readthedocs.io +BSD 3-Clause License + +- Copyright (c) 2001-2015, IPython Development Team +- Copyright (c) 2015-, Jupyter Development Team + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +jupyter_server_terminals +BSD License +https://jupyter.org +BSD 3-Clause License + +- Copyright (c) 2021-, Jupyter Development Team + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +All rights reserved. + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +jupyterlab +BSD License +https://jupyter.org +Copyright (c) 2015-2025 Project Jupyter Contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Semver File License +=================== + +The semver.py file is from https://github.com/podhmo/python-semver +which is licensed under the "MIT" license. See the semver.py file for details. + + +jupyterlab_pygments +BSD License +https://github.com/jupyterlab/jupyterlab_pygments +Copyright (c) 2015 Project Jupyter Contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +jupyterlab_server +BSD License +https://jupyterlab-server.readthedocs.io +Copyright (c) 2015-2017, Project Jupyter Contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +jupyterlab_widgets +BSD License +https://github.com/jupyter-widgets/ipywidgets +Copyright (c) 2015 Project Jupyter Contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ + +This package bundles several JavaScript npm packages in the +jupyterlab_widgets/static directory. Their licenses (as packaged in their +distributions in the node_modules package installation directory) are copied +below. + +------------------------------------------------------------------------------ +From css-loader/LICENSE: + +Copyright JS Foundation and other contributors + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +'Software'), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, +TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +------------------------------------------------------------------------------ +From style-loader/LICENSE: + +Copyright JS Foundation and other contributors + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +'Software'), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, +TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +------------------------------------------------------------------------------ +From backbone/backbone.js + +// (c) 2010-2015 Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors +// Backbone may be freely distributed under the MIT license. +// For all details and documentation: +// http://backbonejs.org + +------------------------------------------------------------------------------ +From base-64/LICENSE + +The MIT License (MIT) + +Copyright (c) 2014 Jameson Little + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +------------------------------------------------------------------------------ +From lodash/LICENSE + +Copyright OpenJS Foundation and other contributors + +Based on Underscore.js, copyright Jeremy Ashkenas, +DocumentCloud and Investigative Reporters & Editors + +This software consists of voluntary contributions made by many +individuals. For exact contribution history, see the revision history +available at https://github.com/lodash/lodash + +The following license applies to all parts of this software except as +documented below: + +==== + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +==== + +Copyright and related rights for sample code are waived via CC0. Sample +code is defined as all source code displayed within the prose of the +documentation. + +CC0: http://creativecommons.org/publicdomain/zero/1.0/ + +==== + +Files located in the node_modules and vendor directories are externally +maintained libraries used by this software which have their own +licenses; we recommend you read them, as their terms may differ from the +terms above. + +------------------------------------------------------------------------------ +From d3-format/LICENSE: + +Copyright 2010-2015 Mike Bostock +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the author nor the names of contributors may be used to + endorse or promote products derived from this software without specific prior + written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +From noUISlider/LICENSE.md (https://github.com/leongersen/noUiSlider/blob/eca62f9e56aaf02f0841b36e7993adf8db3721d5/LICENSE.md) + +MIT License + +Copyright (c) 2019 Léon Gersen + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +------------------------------------------------------------------ +From jquery/LICENSE.txt + +Copyright JS Foundation and other contributors, https://js.foundation/ + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +------------------------------------------------------------------ +From semver/LICENSE: + +The ISC License + +Copyright (c) Isaac Z. Schlueter and Contributors + +Permission to use, copy, modify, and/or distribute this software for any +purpose with or without fee is hereby granted, provided that the above +copyright notice and this permission notice appear in all copies. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR +IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + +------------------------------------------------------------------ +From underscore/LICENSE + +Copyright (c) 2009-2018 Jeremy Ashkenas, DocumentCloud and Investigative +Reporters & Editors + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated documentation +files (the "Software"), to deal in the Software without +restriction, including without limitation the rights to use, +copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +OTHER DEALINGS IN THE SOFTWARE. + + +keyring +MIT +https://github.com/jaraco/keyring +MIT License + +Copyright (c) 2025 + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and +associated documentation files (the "Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the +following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial +portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT +LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO +EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE +USE OR OTHER DEALINGS IN THE SOFTWARE. + + +kiwisolver +BSD License +https://github.com/nucleic/kiwi +========================= + The Kiwi licensing terms +========================= +Kiwi is licensed under the terms of the Modified BSD License (also known as +New or Revised BSD), as follows: + +Copyright (c) 2013-2025, Nucleic Development Team + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +Redistributions of source code must retain the above copyright notice, this +list of conditions and the following disclaimer. + +Redistributions in binary form must reproduce the above copyright notice, this +list of conditions and the following disclaimer in the documentation and/or +other materials provided with the distribution. + +Neither the name of the Nucleic Development Team nor the names of its +contributors may be used to endorse or promote products derived from this +software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +About Kiwi +---------- +Chris Colbert began the Kiwi project in December 2013 in an effort to +create a blisteringly fast UI constraint solver. Chris is still the +project lead. + +The Nucleic Development Team is the set of all contributors to the Nucleic +project and its subprojects. + +The core team that coordinates development on GitHub can be found here: +http://github.com/nucleic. The current team consists of: + +* Chris Colbert + +Our Copyright Policy +-------------------- +Nucleic uses a shared copyright model. Each contributor maintains copyright +over their contributions to Nucleic. But, it is important to note that these +contributions are typically only changes to the repositories. Thus, the Nucleic +source code, in its entirety is not the copyright of any single person or +institution. Instead, it is the collective copyright of the entire Nucleic +Development Team. If individual contributors want to maintain a record of what +changes/contributions they have specific copyright on, they should indicate +their copyright in the commit message of the change, when they commit the +change to one of the Nucleic repositories. + +With this in mind, the following banner should be used in any source code file +to indicate the copyright and license terms: + +#------------------------------------------------------------------------------ +# Copyright (c) 2013-2025, Nucleic Development Team. +# +# Distributed under the terms of the Modified BSD License. +# +# The full license is in the file LICENSE, distributed with this software. +#------------------------------------------------------------------------------ + + +kornia +Apache Software License +https://kornia.github.io/ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + +kornia_rs +Apache Software License +http://kornia.org + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +lark +MIT License +https://github.com/lark-parser/lark +Copyright © 2017 Erez Shinan + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +lazy_loader +BSD License +https://github.com/scientific-python/lazy_loader +BSD 3-Clause License + +Copyright (c) 2022--2023, Scientific Python project +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +lerobot +Apache Software License +https://huggingface.co/lerobot +Copyright 2024 The Hugging Face team. All rights reserved. + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +## Some of lerobot's code is derived from Diffusion Policy, which is subject to the following copyright notice: + +MIT License + +Copyright (c) 2023 Columbia Artificial Intelligence and Robotics Lab + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +## Some of lerobot's code is derived from FOWM, which is subject to the following copyright notice: + +MIT License + +Copyright (c) 2023 Yunhai Feng + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +## Some of lerobot's code is derived from simxarm, which is subject to the following copyright notice: + +MIT License + +Copyright (c) 2023 Nicklas Hansen & Yanjie Ze + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +## Some of lerobot's code is derived from ALOHA, which is subject to the following copyright notice: + +MIT License + +Copyright (c) 2023 Tony Z. Zhao + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +## Some of lerobot's code is derived from DETR, which is subject to the following copyright notice: + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2020 - present, Facebook, Inc + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +lightning +Apache Software License +https://github.com/Lightning-AI/lightning + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2018-2021 William Falcon + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +lightning-utilities +Apache-2.0 +https://github.com/Lightning-AI/utilities + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2018-2021 William Falcon + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +loguru +MIT License +https://github.com/Delgan/loguru +UNKNOWN + +lpips +BSD License +https://github.com/richzhang/PerceptualSimilarity +Copyright (c) 2018, Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, Oliver Wang +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + + +lxml +BSD-3-Clause +https://lxml.de/ +BSD 3-Clause License + +Copyright (c) 2004 Infrae. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + 1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + + 3. Neither the name of Infrae nor the names of its contributors may + be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL INFRAE OR +CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +lz4 +BSD License +https://github.com/python-lz4/python-lz4 +Copyright (c) 2012-2013, Steeve Morin +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of Steeve Morin nor the names of its contributors may be + used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + + +makefun +BSD License +https://github.com/smarie/python-makefun +BSD 3-Clause License + +Copyright (c) 2019-2022, Sylvain Marié, Schneider Electric Industries +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +markdown-it-py +MIT License +https://github.com/executablebooks/markdown-it-py +MIT License + +Copyright (c) 2020 ExecutableBookProject + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +marshmallow +MIT License +https://github.com/marshmallow-code/marshmallow +Copyright Steven Loria and contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +matplotlib +Python Software Foundation License +https://matplotlib.org +License agreement for matplotlib versions 1.3.0 and later +========================================================= + +1. This LICENSE AGREEMENT is between the Matplotlib Development Team +("MDT"), and the Individual or Organization ("Licensee") accessing and +otherwise using matplotlib software in source or binary form and its +associated documentation. + +2. Subject to the terms and conditions of this License Agreement, MDT +hereby grants Licensee a nonexclusive, royalty-free, world-wide license +to reproduce, analyze, test, perform and/or display publicly, prepare +derivative works, distribute, and otherwise use matplotlib +alone or in any derivative version, provided, however, that MDT's +License Agreement and MDT's notice of copyright, i.e., "Copyright (c) +2012- Matplotlib Development Team; All Rights Reserved" are retained in +matplotlib alone or in any derivative version prepared by +Licensee. + +3. In the event Licensee prepares a derivative work that is based on or +incorporates matplotlib or any part thereof, and wants to +make the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to matplotlib . + +4. MDT is making matplotlib available to Licensee on an "AS +IS" basis. MDT MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, MDT MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF MATPLOTLIB +WILL NOT INFRINGE ANY THIRD PARTY RIGHTS. + +5. MDT SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF MATPLOTLIB + FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR +LOSS AS A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING +MATPLOTLIB , OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF +THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. Nothing in this License Agreement shall be deemed to create any +relationship of agency, partnership, or joint venture between MDT and +Licensee. This License Agreement does not grant permission to use MDT +trademarks or trade name in a trademark sense to endorse or promote +products or services of Licensee, or any third party. + +8. By copying, installing or otherwise using matplotlib , +Licensee agrees to be bound by the terms and conditions of this License +Agreement. + +License agreement for matplotlib versions prior to 1.3.0 +======================================================== + +1. This LICENSE AGREEMENT is between John D. Hunter ("JDH"), and the +Individual or Organization ("Licensee") accessing and otherwise using +matplotlib software in source or binary form and its associated +documentation. + +2. Subject to the terms and conditions of this License Agreement, JDH +hereby grants Licensee a nonexclusive, royalty-free, world-wide license +to reproduce, analyze, test, perform and/or display publicly, prepare +derivative works, distribute, and otherwise use matplotlib +alone or in any derivative version, provided, however, that JDH's +License Agreement and JDH's notice of copyright, i.e., "Copyright (c) +2002-2011 John D. Hunter; All Rights Reserved" are retained in +matplotlib alone or in any derivative version prepared by +Licensee. + +3. In the event Licensee prepares a derivative work that is based on or +incorporates matplotlib or any part thereof, and wants to +make the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to matplotlib. + +4. JDH is making matplotlib available to Licensee on an "AS +IS" basis. JDH MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, JDH MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF MATPLOTLIB +WILL NOT INFRINGE ANY THIRD PARTY RIGHTS. + +5. JDH SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF MATPLOTLIB + FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR +LOSS AS A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING +MATPLOTLIB , OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF +THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. Nothing in this License Agreement shall be deemed to create any +relationship of agency, partnership, or joint venture between JDH and +Licensee. This License Agreement does not grant permission to use JDH +trademarks or trade name in a trademark sense to endorse or promote +products or services of Licensee, or any third party. + +8. By copying, installing or otherwise using matplotlib, +Licensee agrees to be bound by the terms and conditions of this License +Agreement. +---- + +This binary distrubution of Matplotlib can also bundle the following software +(depending on the build): + +Name: AMS Fonts +Files: matplotlib/tests/cmr10.pfb +Description: Type-1 version of one of Knuth's Computer Modern fonts +License: OFL-1.1 + The cmr10.pfb file is a Type-1 version of one of Knuth's Computer Modern fonts. + It is included here as test data only, but the following license applies. + + Copyright (c) 1997, 2009, American Mathematical Society (http://www.ams.org). + All Rights Reserved. + + "cmb10" is a Reserved Font Name for this Font Software. + "cmbsy10" is a Reserved Font Name for this Font Software. + "cmbsy5" is a Reserved Font Name for this Font Software. + "cmbsy6" is a Reserved Font Name for this Font Software. + "cmbsy7" is a Reserved Font Name for this Font Software. + "cmbsy8" is a Reserved Font Name for this Font Software. + "cmbsy9" is a Reserved Font Name for this Font Software. + "cmbx10" is a Reserved Font Name for this Font Software. + "cmbx12" is a Reserved Font Name for this Font Software. + "cmbx5" is a Reserved Font Name for this Font Software. + "cmbx6" is a Reserved Font Name for this Font Software. + "cmbx7" is a Reserved Font Name for this Font Software. + "cmbx8" is a Reserved Font Name for this Font Software. + "cmbx9" is a Reserved Font Name for this Font Software. + "cmbxsl10" is a Reserved Font Name for this Font Software. + "cmbxti10" is a Reserved Font Name for this Font Software. + "cmcsc10" is a Reserved Font Name for this Font Software. + "cmcsc8" is a Reserved Font Name for this Font Software. + "cmcsc9" is a Reserved Font Name for this Font Software. + "cmdunh10" is a Reserved Font Name for this Font Software. + "cmex10" is a Reserved Font Name for this Font Software. + "cmex7" is a Reserved Font Name for this Font Software. + "cmex8" is a Reserved Font Name for this Font Software. + "cmex9" is a Reserved Font Name for this Font Software. + "cmff10" is a Reserved Font Name for this Font Software. + "cmfi10" is a Reserved Font Name for this Font Software. + "cmfib8" is a Reserved Font Name for this Font Software. + "cminch" is a Reserved Font Name for this Font Software. + "cmitt10" is a Reserved Font Name for this Font Software. + "cmmi10" is a Reserved Font Name for this Font Software. + "cmmi12" is a Reserved Font Name for this Font Software. + "cmmi5" is a Reserved Font Name for this Font Software. + "cmmi6" is a Reserved Font Name for this Font Software. + "cmmi7" is a Reserved Font Name for this Font Software. + "cmmi8" is a Reserved Font Name for this Font Software. + "cmmi9" is a Reserved Font Name for this Font Software. + "cmmib10" is a Reserved Font Name for this Font Software. + "cmmib5" is a Reserved Font Name for this Font Software. + "cmmib6" is a Reserved Font Name for this Font Software. + "cmmib7" is a Reserved Font Name for this Font Software. + "cmmib8" is a Reserved Font Name for this Font Software. + "cmmib9" is a Reserved Font Name for this Font Software. + "cmr10" is a Reserved Font Name for this Font Software. + "cmr12" is a Reserved Font Name for this Font Software. + "cmr17" is a Reserved Font Name for this Font Software. + "cmr5" is a Reserved Font Name for this Font Software. + "cmr6" is a Reserved Font Name for this Font Software. + "cmr7" is a Reserved Font Name for this Font Software. + "cmr8" is a Reserved Font Name for this Font Software. + "cmr9" is a Reserved Font Name for this Font Software. + "cmsl10" is a Reserved Font Name for this Font Software. + "cmsl12" is a Reserved Font Name for this Font Software. + "cmsl8" is a Reserved Font Name for this Font Software. + "cmsl9" is a Reserved Font Name for this Font Software. + "cmsltt10" is a Reserved Font Name for this Font Software. + "cmss10" is a Reserved Font Name for this Font Software. + "cmss12" is a Reserved Font Name for this Font Software. + "cmss17" is a Reserved Font Name for this Font Software. + "cmss8" is a Reserved Font Name for this Font Software. + "cmss9" is a Reserved Font Name for this Font Software. + "cmssbx10" is a Reserved Font Name for this Font Software. + "cmssdc10" is a Reserved Font Name for this Font Software. + "cmssi10" is a Reserved Font Name for this Font Software. + "cmssi12" is a Reserved Font Name for this Font Software. + "cmssi17" is a Reserved Font Name for this Font Software. + "cmssi8" is a Reserved Font Name for this Font Software. + "cmssi9" is a Reserved Font Name for this Font Software. + "cmssq8" is a Reserved Font Name for this Font Software. + "cmssqi8" is a Reserved Font Name for this Font Software. + "cmsy10" is a Reserved Font Name for this Font Software. + "cmsy5" is a Reserved Font Name for this Font Software. + "cmsy6" is a Reserved Font Name for this Font Software. + "cmsy7" is a Reserved Font Name for this Font Software. + "cmsy8" is a Reserved Font Name for this Font Software. + "cmsy9" is a Reserved Font Name for this Font Software. + "cmtcsc10" is a Reserved Font Name for this Font Software. + "cmtex10" is a Reserved Font Name for this Font Software. + "cmtex8" is a Reserved Font Name for this Font Software. + "cmtex9" is a Reserved Font Name for this Font Software. + "cmti10" is a Reserved Font Name for this Font Software. + "cmti12" is a Reserved Font Name for this Font Software. + "cmti7" is a Reserved Font Name for this Font Software. + "cmti8" is a Reserved Font Name for this Font Software. + "cmti9" is a Reserved Font Name for this Font Software. + "cmtt10" is a Reserved Font Name for this Font Software. + "cmtt12" is a Reserved Font Name for this Font Software. + "cmtt8" is a Reserved Font Name for this Font Software. + "cmtt9" is a Reserved Font Name for this Font Software. + "cmu10" is a Reserved Font Name for this Font Software. + "cmvtt10" is a Reserved Font Name for this Font Software. + "euex10" is a Reserved Font Name for this Font Software. + "euex7" is a Reserved Font Name for this Font Software. + "euex8" is a Reserved Font Name for this Font Software. + "euex9" is a Reserved Font Name for this Font Software. + "eufb10" is a Reserved Font Name for this Font Software. + "eufb5" is a Reserved Font Name for this Font Software. + "eufb7" is a Reserved Font Name for this Font Software. + "eufm10" is a Reserved Font Name for this Font Software. + "eufm5" is a Reserved Font Name for this Font Software. + "eufm7" is a Reserved Font Name for this Font Software. + "eurb10" is a Reserved Font Name for this Font Software. + "eurb5" is a Reserved Font Name for this Font Software. + "eurb7" is a Reserved Font Name for this Font Software. + "eurm10" is a Reserved Font Name for this Font Software. + "eurm5" is a Reserved Font Name for this Font Software. + "eurm7" is a Reserved Font Name for this Font Software. + "eusb10" is a Reserved Font Name for this Font Software. + "eusb5" is a Reserved Font Name for this Font Software. + "eusb7" is a Reserved Font Name for this Font Software. + "eusm10" is a Reserved Font Name for this Font Software. + "eusm5" is a Reserved Font Name for this Font Software. + "eusm7" is a Reserved Font Name for this Font Software. + "lasy10" is a Reserved Font Name for this Font Software. + "lasy5" is a Reserved Font Name for this Font Software. + "lasy6" is a Reserved Font Name for this Font Software. + "lasy7" is a Reserved Font Name for this Font Software. + "lasy8" is a Reserved Font Name for this Font Software. + "lasy9" is a Reserved Font Name for this Font Software. + "lasyb10" is a Reserved Font Name for this Font Software. + "lcircle1" is a Reserved Font Name for this Font Software. + "lcirclew" is a Reserved Font Name for this Font Software. + "lcmss8" is a Reserved Font Name for this Font Software. + "lcmssb8" is a Reserved Font Name for this Font Software. + "lcmssi8" is a Reserved Font Name for this Font Software. + "line10" is a Reserved Font Name for this Font Software. + "linew10" is a Reserved Font Name for this Font Software. + "msam10" is a Reserved Font Name for this Font Software. + "msam5" is a Reserved Font Name for this Font Software. + "msam6" is a Reserved Font Name for this Font Software. + "msam7" is a Reserved Font Name for this Font Software. + "msam8" is a Reserved Font Name for this Font Software. + "msam9" is a Reserved Font Name for this Font Software. + "msbm10" is a Reserved Font Name for this Font Software. + "msbm5" is a Reserved Font Name for this Font Software. + "msbm6" is a Reserved Font Name for this Font Software. + "msbm7" is a Reserved Font Name for this Font Software. + "msbm8" is a Reserved Font Name for this Font Software. + "msbm9" is a Reserved Font Name for this Font Software. + "wncyb10" is a Reserved Font Name for this Font Software. + "wncyi10" is a Reserved Font Name for this Font Software. + "wncyr10" is a Reserved Font Name for this Font Software. + "wncysc10" is a Reserved Font Name for this Font Software. + "wncyss10" is a Reserved Font Name for this Font Software. + + This Font Software is licensed under the SIL Open Font License, Version 1.1. + This license is copied below, and is also available with a FAQ at: + http://scripts.sil.org/OFL + + ----------------------------------------------------------- + SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007 + ----------------------------------------------------------- + + PREAMBLE + The goals of the Open Font License (OFL) are to stimulate worldwide + development of collaborative font projects, to support the font creation + efforts of academic and linguistic communities, and to provide a free and + open framework in which fonts may be shared and improved in partnership + with others. + + The OFL allows the licensed fonts to be used, studied, modified and + redistributed freely as long as they are not sold by themselves. The + fonts, including any derivative works, can be bundled, embedded, + redistributed and/or sold with any software provided that any reserved + names are not used by derivative works. The fonts and derivatives, + however, cannot be released under any other type of license. The + requirement for fonts to remain under this license does not apply + to any document created using the fonts or their derivatives. + + DEFINITIONS + "Font Software" refers to the set of files released by the Copyright + Holder(s) under this license and clearly marked as such. This may + include source files, build scripts and documentation. + + "Reserved Font Name" refers to any names specified as such after the + copyright statement(s). + + "Original Version" refers to the collection of Font Software components as + distributed by the Copyright Holder(s). + + "Modified Version" refers to any derivative made by adding to, deleting, + or substituting -- in part or in whole -- any of the components of the + Original Version, by changing formats or by porting the Font Software to a + new environment. + + "Author" refers to any designer, engineer, programmer, technical + writer or other person who contributed to the Font Software. + + PERMISSION & CONDITIONS + Permission is hereby granted, free of charge, to any person obtaining + a copy of the Font Software, to use, study, copy, merge, embed, modify, + redistribute, and sell modified and unmodified copies of the Font + Software, subject to the following conditions: + + 1) Neither the Font Software nor any of its individual components, + in Original or Modified Versions, may be sold by itself. + + 2) Original or Modified Versions of the Font Software may be bundled, + redistributed and/or sold with any software, provided that each copy + contains the above copyright notice and this license. These can be + included either as stand-alone text files, human-readable headers or + in the appropriate machine-readable metadata fields within text or + binary files as long as those fields can be easily viewed by the user. + + 3) No Modified Version of the Font Software may use the Reserved Font + Name(s) unless explicit written permission is granted by the corresponding + Copyright Holder. This restriction only applies to the primary font name as + presented to the users. + + 4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font + Software shall not be used to promote, endorse or advertise any + Modified Version, except to acknowledge the contribution(s) of the + Copyright Holder(s) and the Author(s) or with their explicit written + permission. + + 5) The Font Software, modified or unmodified, in part or in whole, + must be distributed entirely under this license, and must not be + distributed under any other license. The requirement for fonts to + remain under this license does not apply to any document created + using the Font Software. + + TERMINATION + This license becomes null and void if any of the above conditions are + not met. + + DISCLAIMER + THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF + MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT + OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE + COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL + DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM + OTHER DEALINGS IN THE FONT SOFTWARE. + + + +Name: BaKoMa Fonts +Files: matplotlib/mpl-data/fonts/ttf/cm*.ttf matplotlib/mpl-data/fonts/afm/cm*.afm +Description: Computer Modern Fonts in PostScript Type 1 and TrueType font formats. +License: BaKoMa Fonts Licence + BaKoMa Fonts Licence + -------------------- + + This licence covers two font packs (known as BaKoMa Fonts Collection, + which is available at `CTAN:fonts/cm/ps-type1/bakoma/'): + + 1) BaKoMa-CM (1.1/12-Nov-94) + Computer Modern Fonts in PostScript Type 1 and TrueType font formats. + + 2) BaKoMa-AMS (1.2/19-Jan-95) + AMS TeX fonts in PostScript Type 1 and TrueType font formats. + + Copyright (C) 1994, 1995, Basil K. Malyshev. All Rights Reserved. + + Permission to copy and distribute these fonts for any purpose is + hereby granted without fee, provided that the above copyright notice, + author statement and this permission notice appear in all copies of + these fonts and related documentation. + + Permission to modify and distribute modified fonts for any purpose is + hereby granted without fee, provided that the copyright notice, + author statement, this permission notice and location of original + fonts (http://www.ctan.org/tex-archive/fonts/cm/ps-type1/bakoma) + appear in all copies of modified fonts and related documentation. + + Permission to use these fonts (embedding into PostScript, PDF, SVG + and printing by using any software) is hereby granted without fee. + It is not required to provide any notices about using these fonts. + + Basil K. Malyshev + INSTITUTE FOR HIGH ENERGY PHYSICS + IHEP, OMVT + Moscow Region + 142281 PROTVINO + RUSSIA + + E-Mail: bakoma@mail.ru + or malyshev@mail.ihep.ru + + + + +Name: ColorBrewer Color Schemes +Files: lib/matplotlib/_cm.py +Description: Color schemes from ColorBrewer +License: Apache-2.0 + Apache-Style Software License for ColorBrewer software and ColorBrewer Color Schemes + + Copyright (c) 2002 Cynthia Brewer, Mark Harrower, and The Pennsylvania State University. + + Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software distributed + under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR + CONDITIONS OF ANY KIND, either express or implied. See the License for the + specific language governing permissions and limitations under the License. + + +Name: Courier 10 +Files: matplotlib/tests/Courier10PitchBT-Bold.pfb +Description: Courier 10 font, used in tests. +License: Bitstream-Charter + The Courier10PitchBT-Bold.pfb file is a Type-1 version of + Courier 10 Pitch BT Bold by Bitstream, obtained from + . It is included + here as test data only, but the following license applies. + + + (c) Copyright 1989-1992, Bitstream Inc., Cambridge, MA. + + You are hereby granted permission under all Bitstream propriety rights + to use, copy, modify, sublicense, sell, and redistribute the 4 Bitstream + Charter (r) Type 1 outline fonts and the 4 Courier Type 1 outline fonts + for any purpose and without restriction; provided, that this notice is + left intact on all copies of such fonts and that Bitstream's trademark + is acknowledged as shown below on all unmodified copies of the 4 Charter + Type 1 fonts. + + BITSTREAM CHARTER is a registered trademark of Bitstream Inc. + + + +Name: JSXTools resize observer +Files: +Description: Minimal polyfill for the ResizeObserver API +License: CC0-1.0 + # CC0 1.0 Universal + + ## Statement of Purpose + + The laws of most jurisdictions throughout the world automatically confer + exclusive Copyright and Related Rights (defined below) upon the creator and + subsequent owner(s) (each and all, an “owner”) of an original work of + authorship and/or a database (each, a “Work”). + + Certain owners wish to permanently relinquish those rights to a Work for the + purpose of contributing to a commons of creative, cultural and scientific works + (“Commons”) that the public can reliably and without fear of later claims of + infringement build upon, modify, incorporate in other works, reuse and + redistribute as freely as possible in any form whatsoever and for any purposes, + including without limitation commercial purposes. These owners may contribute + to the Commons to promote the ideal of a free culture and the further + production of creative, cultural and scientific works, or to gain reputation or + greater distribution for their Work in part through the use and efforts of + others. + + For these and/or other purposes and motivations, and without any expectation of + additional consideration or compensation, the person associating CC0 with a + Work (the “Affirmer”), to the extent that he or she is an owner of Copyright + and Related Rights in the Work, voluntarily elects to apply CC0 to the Work and + publicly distribute the Work under its terms, with knowledge of his or her + Copyright and Related Rights in the Work and the meaning and intended legal + effect of CC0 on those rights. + + 1. Copyright and Related Rights. A Work made available under CC0 may be + protected by copyright and related or neighboring rights (“Copyright and + Related Rights”). Copyright and Related Rights include, but are not limited + to, the following: + 1. the right to reproduce, adapt, distribute, perform, display, communicate, + and translate a Work; + 2. moral rights retained by the original author(s) and/or performer(s); + 3. publicity and privacy rights pertaining to a person’s image or likeness + depicted in a Work; + 4. rights protecting against unfair competition in regards to a Work, + subject to the limitations in paragraph 4(i), below; + 5. rights protecting the extraction, dissemination, use and reuse of data in + a Work; + 6. database rights (such as those arising under Directive 96/9/EC of the + European Parliament and of the Council of 11 March 1996 on the legal + protection of databases, and under any national implementation thereof, + including any amended or successor version of such directive); and + 7. other similar, equivalent or corresponding rights throughout the world + based on applicable law or treaty, and any national implementations + thereof. + + 2. Waiver. To the greatest extent permitted by, but not in contravention of, + applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and + unconditionally waives, abandons, and surrenders all of Affirmer’s Copyright + and Related Rights and associated claims and causes of action, whether now + known or unknown (including existing as well as future claims and causes of + action), in the Work (i) in all territories worldwide, (ii) for the maximum + duration provided by applicable law or treaty (including future time + extensions), (iii) in any current or future medium and for any number of + copies, and (iv) for any purpose whatsoever, including without limitation + commercial, advertising or promotional purposes (the “Waiver”). Affirmer + makes the Waiver for the benefit of each member of the public at large and + to the detriment of Affirmer’s heirs and successors, fully intending that + such Waiver shall not be subject to revocation, rescission, cancellation, + termination, or any other legal or equitable action to disrupt the quiet + enjoyment of the Work by the public as contemplated by Affirmer’s express + Statement of Purpose. + + 3. Public License Fallback. Should any part of the Waiver for any reason be + judged legally invalid or ineffective under applicable law, then the Waiver + shall be preserved to the maximum extent permitted taking into account + Affirmer’s express Statement of Purpose. In addition, to the extent the + Waiver is so judged Affirmer hereby grants to each affected person a + royalty-free, non transferable, non sublicensable, non exclusive, + irrevocable and unconditional license to exercise Affirmer’s Copyright and + Related Rights in the Work (i) in all territories worldwide, (ii) for the + maximum duration provided by applicable law or treaty (including future time + extensions), (iii) in any current or future medium and for any number of + copies, and (iv) for any purpose whatsoever, including without limitation + commercial, advertising or promotional purposes (the “License”). The License + shall be deemed effective as of the date CC0 was applied by Affirmer to the + Work. Should any part of the License for any reason be judged legally + invalid or ineffective under applicable law, such partial invalidity or + ineffectiveness shall not invalidate the remainder of the License, and in + such case Affirmer hereby affirms that he or she will not (i) exercise any + of his or her remaining Copyright and Related Rights in the Work or (ii) + assert any associated claims and causes of action with respect to the Work, + in either case contrary to Affirmer’s express Statement of Purpose. + + 4. Limitations and Disclaimers. + 1. No trademark or patent rights held by Affirmer are waived, abandoned, + surrendered, licensed or otherwise affected by this document. + 2. Affirmer offers the Work as-is and makes no representations or warranties + of any kind concerning the Work, express, implied, statutory or + otherwise, including without limitation warranties of title, + merchantability, fitness for a particular purpose, non infringement, or + the absence of latent or other defects, accuracy, or the present or + absence of errors, whether or not discoverable, all to the greatest + extent permissible under applicable law. + 3. Affirmer disclaims responsibility for clearing rights of other persons + that may apply to the Work or any use thereof, including without + limitation any person’s Copyright and Related Rights in the Work. + Further, Affirmer disclaims responsibility for obtaining any necessary + consents, permissions or other rights required for any use of the Work. + 4. Affirmer understands and acknowledges that Creative Commons is not a + party to this document and has no duty or obligation with respect to this + CC0 or use of the Work. + + For more information, please see + http://creativecommons.org/publicdomain/zero/1.0/. + + +Name: QHull +Files: matplotlib/_qhull.*.so +Description: Convex hull, Delaunay triangulation, Voronoi diagrams, Halfspace intersection +License: Qhull + Qhull, Copyright (c) 1993-2020 + + C.B. Barber + Arlington, MA + + and + + The National Science and Technology Research Center for + Computation and Visualization of Geometric Structures + (The Geometry Center) + University of Minnesota + + email: qhull@qhull.org + + This software includes Qhull from C.B. Barber and The Geometry Center. + Files derived from Qhull 1.0 are copyrighted by the Geometry Center. The + remaining files are copyrighted by C.B. Barber. Qhull is free software + and may be obtained via http from www.qhull.org. It may be freely copied, + modified, and redistributed under the following conditions: + + 1. All copyright notices must remain intact in all files. + + 2. A copy of this text file must be distributed along with any copies + of Qhull that you redistribute; this includes copies that you have + modified, or copies of programs or other software products that + include Qhull. + + 3. If you modify Qhull, you must include a notice giving the + name of the person performing the modification, the date of + modification, and the reason for such modification. + + 4. When distributing modified versions of Qhull, or other software + products that include Qhull, you must provide notice that the original + source code may be obtained as noted above. + + 5. There is no warranty or other guarantee of fitness for Qhull, it is + provided solely "as is". Bug reports or fixes may be sent to + qhull_bug@qhull.org; the authors may or may not act on them as + they desire. + + +Name: Qt4 Editor +Files: matplotlib/backends/qt_editor +Description: Module creating PyQt4 form dialogs/layouts to edit various type of parameters +License: MIT + Module creating PyQt4 form dialogs/layouts to edit various type of parameters + + + formlayout License Agreement (MIT License) + ------------------------------------------ + + Copyright (c) 2009 Pierre Raybaut + + Permission is hereby granted, free of charge, to any person + obtaining a copy of this software and associated documentation + files (the "Software"), to deal in the Software without + restriction, including without limitation the rights to use, + copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the + Software is furnished to do so, subject to the following + conditions: + + The above copyright notice and this permission notice shall be + included in all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + OTHER DEALINGS IN THE SOFTWARE. + """ + + +Name: Solarized +Files: matplotlib/mpl-data/stylelib/Solarize_Light2.mplstyle +Description: Solarized color scheme/style +License: MIT + https://github.com/altercation/solarized/blob/master/LICENSE + Copyright (c) 2011 Ethan Schoonover + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. + + +Name: Stix fonts +Files: matplotlib/mpl-data/fonts/ttf/STIX*.ttf +Description: STIX fonts +License: + TERMS AND CONDITIONS + + 1. Permission is hereby granted, free of charge, to any person + obtaining a copy of the STIX Fonts-TM set accompanying this license + (collectively, the "Fonts") and the associated documentation files + (collectively with the Fonts, the "Font Software"), to reproduce and + distribute the Font Software, including the rights to use, copy, merge + and publish copies of the Font Software, and to permit persons to whom + the Font Software is furnished to do so same, subject to the following + terms and conditions (the "License"). + + 2. The following copyright and trademark notice and these Terms and + Conditions shall be included in all copies of one or more of the Font + typefaces and any derivative work created as permitted under this + License: + + Copyright (c) 2001-2005 by the STI Pub Companies, consisting of + the American Institute of Physics, the American Chemical Society, the + American Mathematical Society, the American Physical Society, Elsevier, + Inc., and The Institute of Electrical and Electronic Engineers, Inc. + Portions copyright (c) 1998-2003 by MicroPress, Inc. Portions copyright + (c) 1990 by Elsevier, Inc. All rights reserved. STIX Fonts-TM is a + trademark of The Institute of Electrical and Electronics Engineers, Inc. + + 3. You may (a) convert the Fonts from one format to another (e.g., + from TrueType to PostScript), in which case the normal and reasonable + distortion that occurs during such conversion shall be permitted and (b) + embed or include a subset of the Fonts in a document for the purposes of + allowing users to read text in the document that utilizes the Fonts. In + each case, you may use the STIX Fonts-TM mark to designate the resulting + Fonts or subset of the Fonts. + + 4. You may also (a) add glyphs or characters to the Fonts, or modify + the shape of existing glyphs, so long as the base set of glyphs is not + removed and (b) delete glyphs or characters from the Fonts, provided + that the resulting font set is distributed with the following + disclaimer: "This [name] font does not include all the Unicode points + covered in the STIX Fonts-TM set but may include others." In each case, + the name used to denote the resulting font set shall not include the + term "STIX" or any similar term. + + 5. You may charge a fee in connection with the distribution of the + Font Software, provided that no copy of one or more of the individual + Font typefaces that form the STIX Fonts-TM set may be sold by itself. + + 6. THE FONT SOFTWARE IS PROVIDED "AS IS," WITHOUT WARRANTY OF ANY + KIND, EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTIES + OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT + OF COPYRIGHT, PATENT, TRADEMARK OR OTHER RIGHT. IN NO EVENT SHALL + MICROPRESS OR ANY OF THE STI PUB COMPANIES BE LIABLE FOR ANY CLAIM, + DAMAGES OR OTHER LIABILITY, INCLUDING, BUT NOT LIMITED TO, ANY GENERAL, + SPECIAL, INDIRECT, INCIDENTAL OR CONSEQUENTIAL DAMAGES, WHETHER IN AN + ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM OR OUT OF THE USE OR + INABILITY TO USE THE FONT SOFTWARE OR FROM OTHER DEALINGS IN THE FONT + SOFTWARE. + + 7. Except as contained in the notice set forth in Section 2, the + names MicroPress Inc. and STI Pub Companies, as well as the names of the + companies/organizations that compose the STI Pub Companies, shall not be + used in advertising or otherwise to promote the sale, use or other + dealings in the Font Software without the prior written consent of the + respective company or organization. + + 8. This License shall become null and void in the event of any + material breach of the Terms and Conditions herein by licensee. + + 9. A substantial portion of the STIX Fonts set was developed by + MicroPress Inc. for the STI Pub Companies. To obtain additional + mathematical fonts, please contact MicroPress, Inc., 68-30 Harrow + Street, Forest Hills, NY 11375, USA - Phone: (718) 575-1816. + + +Name: Yorick Colormaps +Files: lib/matplotlib/_cm.py +Description: Gist/Yorick colormaps +License: + BSD-style license for gist/yorick colormaps. + + Copyright: + + Copyright (c) 1996. The Regents of the University of California. + All rights reserved. + + Permission to use, copy, modify, and distribute this software for any + purpose without fee is hereby granted, provided that this entire + notice is included in all copies of any software which is or includes + a copy or modification of this software and in all copies of the + supporting documentation for such software. + + This work was produced at the University of California, Lawrence + Livermore National Laboratory under contract no. W-7405-ENG-48 between + the U.S. Department of Energy and The Regents of the University of + California for the operation of UC LLNL. + + + DISCLAIMER + + This software was prepared as an account of work sponsored by an + agency of the United States Government. Neither the United States + Government nor the University of California nor any of their + employees, makes any warranty, express or implied, or assumes any + liability or responsibility for the accuracy, completeness, or + usefulness of any information, apparatus, product, or process + disclosed, or represents that its use would not infringe + privately-owned rights. Reference herein to any specific commercial + products, process, or service by trade name, trademark, manufacturer, + or otherwise, does not necessarily constitute or imply its + endorsement, recommendation, or favoring by the United States + Government or the University of California. The views and opinions of + authors expressed herein do not necessarily state or reflect those of + the United States Government or the University of California, and + shall not be used for advertising or product endorsement purposes. + + + AUTHOR + + David H. Munro wrote Yorick and Gist. Berkeley Yacc (byacc) generated + the Yorick parser. The routines in Math are from LAPACK and FFTPACK; + MathC contains C translations by David H. Munro. The algorithms for + Yorick's random number generator and several special functions in + Yorick/include were taken from Numerical Recipes by Press, et. al., + although the Yorick implementations are unrelated to those in + Numerical Recipes. A small amount of code in Gist was adapted from + the X11R4 release, copyright M.I.T. -- the complete copyright notice + may be found in the (unused) file Gist/host.c. + + +matplotlib-inline +UNKNOWN +https://github.com/ipython/matplotlib-inline +BSD 3-Clause License + +Copyright (c) 2019-2022, IPython Development Team. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +mdurl +MIT License +https://github.com/executablebooks/mdurl +Copyright (c) 2015 Vitaly Puzrin, Alex Kocharin. +Copyright (c) 2021 Taneli Hukkinen + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated documentation +files (the "Software"), to deal in the Software without +restriction, including without limitation the rights to use, +copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +OTHER DEALINGS IN THE SOFTWARE. + +-------------------------------------------------------------------------------- + +.parse() is based on Joyent's node.js `url` code: + +Copyright Joyent, Inc. and other Node contributors. All rights reserved. +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to +deal in the Software without restriction, including without limitation the +rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +sell copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. + + +mediapy +Apache Software License +https://github.com/google/mediapy + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +megatron-core +BSD License +https://github.com/NVIDIA/Megatron-LM/megatron/core +The following applies to all files unless otherwise noted: + +# Copyright (c) 2019-2025, NVIDIA CORPORATION. All rights reserved. +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of NVIDIA CORPORATION nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY +# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY +# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +-- + +This repository also contains code from Hugging Face Inc., Google Research, +Facebook (from their Fairseq, Dino, and ParlAI projects), Microsoft (from their +Swin-Transformer project), Philip Popien, the Mamba project (Tri Dao and +Albert Gu), and the Triton language and compiler project (Philippe Tillet and +OpenAI). Files from these organizations have notices at the top of each file. +Below are licenses used in those files, as indicated. + + +-------------------------------------------------------------------------------------- +-- LICENSE FOR Facebook, huggingface, Google Research, LLaVA, Mamba, TinyZero and vLLM code -- + + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +-------------------------------------------------------------------------------- +LICENSE FOR +Facebook, Inc. and its affiliates, +Meta Platforms, Inc. and its affiliates, +Microsoft Corporation, +OpenGVLab/InternVL, +Triton language and compiler, +and DeepSeek. + +MIT License + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +-------------------------------------------------------------------------------- +LICENSE FOR Thinking Machines Lab + +MIT License + +Copyright 2025 Thinking Machines Lab + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +-------------------------------------------------------------------------------- +LICENSE FOR +Meta Platforms, Inc. and affiliates. + +BSD License + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + * Neither the name Meta nor the names of its contributors may be used to + endorse or promote products derived from this software without specific + prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +mistune +BSD License +https://github.com/lepture/mistune +Copyright (c) 2014, Hsiaoming Yang + +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. + +* Neither the name of the creator nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. + + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +ml_dtypes +Apache-2.0 +https://github.com/jax-ml/ml_dtypes + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +more-itertools +MIT +https://github.com/more-itertools/more-itertools +Copyright (c) 2012 Erik Rose + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +moviepy +MIT License +UNKNOWN +The MIT License (MIT) + +Copyright (c) 2015 Zulko + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +mpmath +BSD License +http://mpmath.org/ +Copyright (c) 2005-2021 Fredrik Johansson and mpmath contributors + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + a. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + b. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + c. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH +DAMAGE. + + +msgpack +Apache-2.0 +https://msgpack.org/ +Copyright (C) 2008-2011 INADA Naoki + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +multi-storage-client +Apache-2.0 +https://github.com/NVIDIA/multi-storage-client +UNKNOWN + +multidict +Apache License 2.0 +https://github.com/aio-libs/multidict + Copyright 2016 Andrew Svetlov and aio-libs contributors + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +multiprocess +BSD License +https://github.com/uqfoundation/multiprocess +Copyright (c) 2006-2008, R Oudkerk + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. +3. Neither the name of author nor the names of any contributors may be + used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +SUCH DAMAGE. + + +mypy_extensions +MIT +https://github.com/python/mypy_extensions +Mypy extensions are licensed under the terms of the MIT license, reproduced below. + += = = = = + +The MIT License + +Copyright (c) 2016-2017 Jukka Lehtosalo and contributors + +Permission is hereby granted, free of charge, to any person obtaining a +copy of this software and associated documentation files (the "Software"), +to deal in the Software without restriction, including without limitation +the rights to use, copy, modify, merge, publish, distribute, sublicense, +and/or sell copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. + += = = = = + + +nbclient +BSD License +https://jupyter.org +BSD 3-Clause License + +Copyright (c) 2020-, Jupyter Development Team + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +nbconvert +BSD License +https://jupyter.org +BSD 3-Clause License + +- Copyright (c) 2001-2015, IPython Development Team +- Copyright (c) 2015-, Jupyter Development Team + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +nbformat +BSD License +https://jupyter.org +BSD 3-Clause License + +- Copyright (c) 2001-2015, IPython Development Team +- Copyright (c) 2015-, Jupyter Development Team + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +nest-asyncio +BSD License +https://github.com/erdewit/nest_asyncio +BSD 2-Clause License + +Copyright (c) 2018-2020, Ewald de Wit +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +networkx +BSD-3-Clause +https://networkx.org/ +NetworkX is distributed with the 3-clause BSD license. + +:: + + Copyright (c) 2004-2025, NetworkX Developers + Aric Hagberg + Dan Schult + Pieter Swart + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are + met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided + with the distribution. + + * Neither the name of the NetworkX Developers nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +ninja +Apache Software License; BSD License +http://ninja-build.org/ +Apache License +Version 2.0, January 2004 +http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + +"License" shall mean the terms and conditions for use, reproduction, and +distribution as defined by Sections 1 through 9 of this document. + +"Licensor" shall mean the copyright owner or entity authorized by the copyright +owner that is granting the License. + +"Legal Entity" shall mean the union of the acting entity and all other entities +that control, are controlled by, or are under common control with that entity. +For the purposes of this definition, "control" means (i) the power, direct or +indirect, to cause the direction or management of such entity, whether by +contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the +outstanding shares, or (iii) beneficial ownership of such entity. + +"You" (or "Your") shall mean an individual or Legal Entity exercising +permissions granted by this License. + +"Source" form shall mean the preferred form for making modifications, including +but not limited to software source code, documentation source, and configuration +files. + +"Object" form shall mean any form resulting from mechanical transformation or +translation of a Source form, including but not limited to compiled object code, +generated documentation, and conversions to other media types. + +"Work" shall mean the work of authorship, whether in Source or Object form, made +available under the License, as indicated by a copyright notice that is included +in or attached to the work (an example is provided in the Appendix below). + +"Derivative Works" shall mean any work, whether in Source or Object form, that +is based on (or derived from) the Work and for which the editorial revisions, +annotations, elaborations, or other modifications represent, as a whole, an +original work of authorship. For the purposes of this License, Derivative Works +shall not include works that remain separable from, or merely link (or bind by +name) to the interfaces of, the Work and Derivative Works thereof. + +"Contribution" shall mean any work of authorship, including the original version +of the Work and any modifications or additions to that Work or Derivative Works +thereof, that is intentionally submitted to Licensor for inclusion in the Work +by the copyright owner or by an individual or Legal Entity authorized to submit +on behalf of the copyright owner. For the purposes of this definition, +"submitted" means any form of electronic, verbal, or written communication sent +to the Licensor or its representatives, including but not limited to +communication on electronic mailing lists, source code control systems, and +issue tracking systems that are managed by, or on behalf of, the Licensor for +the purpose of discussing and improving the Work, but excluding communication +that is conspicuously marked or otherwise designated in writing by the copyright +owner as "Not a Contribution." + +"Contributor" shall mean Licensor and any individual or Legal Entity on behalf +of whom a Contribution has been received by Licensor and subsequently +incorporated within the Work. + +2. Grant of Copyright License. + +Subject to the terms and conditions of this License, each Contributor hereby +grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, +irrevocable copyright license to reproduce, prepare Derivative Works of, +publicly display, publicly perform, sublicense, and distribute the Work and such +Derivative Works in Source or Object form. + +3. Grant of Patent License. + +Subject to the terms and conditions of this License, each Contributor hereby +grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, +irrevocable (except as stated in this section) patent license to make, have +made, use, offer to sell, sell, import, and otherwise transfer the Work, where +such license applies only to those patent claims licensable by such Contributor +that are necessarily infringed by their Contribution(s) alone or by combination +of their Contribution(s) with the Work to which such Contribution(s) was +submitted. If You institute patent litigation against any entity (including a +cross-claim or counterclaim in a lawsuit) alleging that the Work or a +Contribution incorporated within the Work constitutes direct or contributory +patent infringement, then any patent licenses granted to You under this License +for that Work shall terminate as of the date such litigation is filed. + +4. Redistribution. + +You may reproduce and distribute copies of the Work or Derivative Works thereof +in any medium, with or without modifications, and in Source or Object form, +provided that You meet the following conditions: + +You must give any other recipients of the Work or Derivative Works a copy of +this License; and +You must cause any modified files to carry prominent notices stating that You +changed the files; and +You must retain, in the Source form of any Derivative Works that You distribute, +all copyright, patent, trademark, and attribution notices from the Source form +of the Work, excluding those notices that do not pertain to any part of the +Derivative Works; and +If the Work includes a "NOTICE" text file as part of its distribution, then any +Derivative Works that You distribute must include a readable copy of the +attribution notices contained within such NOTICE file, excluding those notices +that do not pertain to any part of the Derivative Works, in at least one of the +following places: within a NOTICE text file distributed as part of the +Derivative Works; within the Source form or documentation, if provided along +with the Derivative Works; or, within a display generated by the Derivative +Works, if and wherever such third-party notices normally appear. The contents of +the NOTICE file are for informational purposes only and do not modify the +License. You may add Your own attribution notices within Derivative Works that +You distribute, alongside or as an addendum to the NOTICE text from the Work, +provided that such additional attribution notices cannot be construed as +modifying the License. +You may add Your own copyright statement to Your modifications and may provide +additional or different license terms and conditions for use, reproduction, or +distribution of Your modifications, or for any such Derivative Works as a whole, +provided Your use, reproduction, and distribution of the Work otherwise complies +with the conditions stated in this License. + +5. Submission of Contributions. + +Unless You explicitly state otherwise, any Contribution intentionally submitted +for inclusion in the Work by You to the Licensor shall be under the terms and +conditions of this License, without any additional terms or conditions. +Notwithstanding the above, nothing herein shall supersede or modify the terms of +any separate license agreement you may have executed with Licensor regarding +such Contributions. + +6. Trademarks. + +This License does not grant permission to use the trade names, trademarks, +service marks, or product names of the Licensor, except as required for +reasonable and customary use in describing the origin of the Work and +reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. + +Unless required by applicable law or agreed to in writing, Licensor provides the +Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, +including, without limitation, any warranties or conditions of TITLE, +NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are +solely responsible for determining the appropriateness of using or +redistributing the Work and assume any risks associated with Your exercise of +permissions under this License. + +8. Limitation of Liability. + +In no event and under no legal theory, whether in tort (including negligence), +contract, or otherwise, unless required by applicable law (such as deliberate +and grossly negligent acts) or agreed to in writing, shall any Contributor be +liable to You for damages, including any direct, indirect, special, incidental, +or consequential damages of any character arising as a result of this License or +out of the use or inability to use the Work (including but not limited to +damages for loss of goodwill, work stoppage, computer failure or malfunction, or +any and all other commercial damages or losses), even if such Contributor has +been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. + +While redistributing the Work or Derivative Works thereof, You may choose to +offer, and charge a fee for, acceptance of support, warranty, indemnity, or +other liability obligations and/or rights consistent with this License. However, +in accepting such obligations, You may act only on Your own behalf and on Your +sole responsibility, not on behalf of any other Contributor, and only if You +agree to indemnify, defend, and hold each Contributor harmless for any liability +incurred by, or claims asserted against, such Contributor by reason of your +accepting any such warranty or additional liability. + +END OF TERMS AND CONDITIONS + +APPENDIX: How to apply the Apache License to your work + +To apply the Apache License to your work, attach the following boilerplate +notice, with the fields enclosed by brackets "[]" replaced with your own +identifying information. (Don't include the brackets!) The text should be +enclosed in the appropriate comment syntax for the file format. We also +recommend that a file or class name and description of purpose be included on +the same "printed page" as the copyright notice for easier identification within +third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +nltk +Apache Software License +https://www.nltk.org/ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +notebook_shim +BSD License +UNKNOWN +BSD 3-Clause License + +Copyright (c) 2022 Project Jupyter Contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +numcodecs +MIT +https://github.com/zarr-developers/numcodecs +The MIT License (MIT) + +Copyright (c) 2015-2018 Zarr Developers + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +numpy +BSD-3-Clause AND 0BSD AND MIT AND Zlib AND CC0-1.0 +https://numpy.org +Copyright (c) 2005-2025, NumPy Developers. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided + with the distribution. + + * Neither the name of the NumPy Developers nor the names of any + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +---- + + +---- + +This binary distribution of NumPy also bundles the following software: + + +Name: OpenBLAS +Files: numpy.libs/libscipy_openblas*.so +Description: bundled as a dynamically linked library +Availability: https://github.com/OpenMathLib/OpenBLAS/ +License: BSD-3-Clause + Copyright (c) 2011-2014, The OpenBLAS Project + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are + met: + + 1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + 3. Neither the name of the OpenBLAS project nor the names of + its contributors may be used to endorse or promote products + derived from this software without specific prior written + permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE + USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +Name: LAPACK +Files: numpy.libs/libscipy_openblas*.so +Description: bundled in OpenBLAS +Availability: https://github.com/OpenMathLib/OpenBLAS/ +License: BSD-3-Clause-Open-MPI + Copyright (c) 1992-2013 The University of Tennessee and The University + of Tennessee Research Foundation. All rights + reserved. + Copyright (c) 2000-2013 The University of California Berkeley. All + rights reserved. + Copyright (c) 2006-2013 The University of Colorado Denver. All rights + reserved. + + $COPYRIGHT$ + + Additional copyrights may follow + + $HEADER$ + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are + met: + + - Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + - Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer listed + in this license in the documentation and/or other materials + provided with the distribution. + + - Neither the name of the copyright holders nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + + The copyright holders provide no reassurances that the source code + provided does not infringe any patent, copyright, or any other + intellectual property rights of third parties. The copyright holders + disclaim any liability to any recipient for claims brought against + recipient by any third party for infringement of that parties + intellectual property rights. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +Name: GCC runtime library +Files: numpy.libs/libgfortran*.so +Description: dynamically linked to files compiled with gcc +Availability: https://gcc.gnu.org/git/?p=gcc.git;a=tree;f=libgfortran +License: GPL-3.0-or-later WITH GCC-exception-3.1 + Copyright (C) 2002-2017 Free Software Foundation, Inc. + + Libgfortran is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 3, or (at your option) + any later version. + + Libgfortran is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + Under Section 7 of GPL version 3, you are granted additional + permissions described in the GCC Runtime Library Exception, version + 3.1, as published by the Free Software Foundation. + + You should have received a copy of the GNU General Public License and + a copy of the GCC Runtime Library Exception along with this program; + see the files COPYING3 and COPYING.RUNTIME respectively. If not, see + . + +---- + +Full text of license texts referred to above follows (that they are +listed below does not necessarily imply the conditions apply to the +present binary release): + +---- + +GCC RUNTIME LIBRARY EXCEPTION + +Version 3.1, 31 March 2009 + +Copyright (C) 2009 Free Software Foundation, Inc. + +Everyone is permitted to copy and distribute verbatim copies of this +license document, but changing it is not allowed. + +This GCC Runtime Library Exception ("Exception") is an additional +permission under section 7 of the GNU General Public License, version +3 ("GPLv3"). It applies to a given file (the "Runtime Library") that +bears a notice placed by the copyright holder of the file stating that +the file is governed by GPLv3 along with this Exception. + +When you use GCC to compile a program, GCC may combine portions of +certain GCC header files and runtime libraries with the compiled +program. The purpose of this Exception is to allow compilation of +non-GPL (including proprietary) programs to use, in this way, the +header files and runtime libraries covered by this Exception. + +0. Definitions. + +A file is an "Independent Module" if it either requires the Runtime +Library for execution after a Compilation Process, or makes use of an +interface provided by the Runtime Library, but is not otherwise based +on the Runtime Library. + +"GCC" means a version of the GNU Compiler Collection, with or without +modifications, governed by version 3 (or a specified later version) of +the GNU General Public License (GPL) with the option of using any +subsequent versions published by the FSF. + +"GPL-compatible Software" is software whose conditions of propagation, +modification and use would permit combination with GCC in accord with +the license of GCC. + +"Target Code" refers to output from any compiler for a real or virtual +target processor architecture, in executable form or suitable for +input to an assembler, loader, linker and/or execution +phase. Notwithstanding that, Target Code does not include data in any +format that is used as a compiler intermediate representation, or used +for producing a compiler intermediate representation. + +The "Compilation Process" transforms code entirely represented in +non-intermediate languages designed for human-written code, and/or in +Java Virtual Machine byte code, into Target Code. Thus, for example, +use of source code generators and preprocessors need not be considered +part of the Compilation Process, since the Compilation Process can be +understood as starting with the output of the generators or +preprocessors. + +A Compilation Process is "Eligible" if it is done using GCC, alone or +with other GPL-compatible software, or if it is done without using any +work based on GCC. For example, using non-GPL-compatible Software to +optimize any GCC intermediate representations would not qualify as an +Eligible Compilation Process. + +1. Grant of Additional Permission. + +You have permission to propagate a work of Target Code formed by +combining the Runtime Library with Independent Modules, even if such +propagation would otherwise violate the terms of GPLv3, provided that +all Target Code was generated by Eligible Compilation Processes. You +may then convey such a combination under terms of your choice, +consistent with the licensing of the Independent Modules. + +2. No Weakening of GCC Copyleft. + +The availability of this Exception does not imply any general +presumption that third-party software is unaffected by the copyleft +requirements of the license of GCC. + +---- + + GNU GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + Copyright (C) + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, your program's commands +might be different; for a GUI interface, you would use an "about box". + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU GPL, see +. + + The GNU General Public License does not permit incorporating your program +into proprietary programs. If your program is a subroutine library, you +may consider it more useful to permit linking proprietary applications with +the library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. But first, please read +. + +Name: libquadmath +Files: numpy.libs/libquadmath*.so +Description: dynamically linked to files compiled with gcc +Availability: https://gcc.gnu.org/git/?p=gcc.git;a=tree;f=libquadmath +License: LGPL-2.1-or-later + + GCC Quad-Precision Math Library + Copyright (C) 2010-2019 Free Software Foundation, Inc. + Written by Francois-Xavier Coudert + + This file is part of the libquadmath library. + Libquadmath is free software; you can redistribute it and/or + modify it under the terms of the GNU Library General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + Libquadmath is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html + + +nvdlfw_inspect +Apache2 +https://github.com/NVIDIA/nvidia-dlfw-inspect + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +nvidia-cublas +LicenseRef-NVIDIA-Proprietary +https://developer.nvidia.com/cuda-zone +UNKNOWN + +nvidia-cuda-cupti +Other/Proprietary License +https://developer.nvidia.com/cuda-zone +UNKNOWN + +nvidia-cuda-nvrtc +Other/Proprietary License +https://developer.nvidia.com/cuda-zone +UNKNOWN + +nvidia-cuda-runtime +LicenseRef-NVIDIA-Proprietary +https://developer.nvidia.com/cuda-zone +UNKNOWN + +nvidia-cudnn-cu13 +LicenseRef-NVIDIA-Proprietary +https://developer.nvidia.com/cuda-zone +UNKNOWN + +nvidia-cudnn-frontend +NVIDIA Proprietary Software +https://github.com/nvidia/cudnn-frontend +MIT License + +Copyright (c) 2013-2022 Niels Lohmann + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +nvidia-cufft +Other/Proprietary License +https://developer.nvidia.com/cuda-zone +UNKNOWN + +nvidia-cufile +Other/Proprietary License +https://developer.nvidia.com/cuda-zone +UNKNOWN + +nvidia-curand +Other/Proprietary License +https://developer.nvidia.com/cuda-zone +UNKNOWN + +nvidia-cusolver +Other/Proprietary License +https://developer.nvidia.com/cuda-zone +UNKNOWN + +nvidia-cusparse +Other/Proprietary License +https://developer.nvidia.com/cuda-zone +UNKNOWN + +nvidia-cusparselt-cu13 +NVIDIA Proprietary Software +https://developer.nvidia.com/cusparselt +LICENSE AGREEMENT FOR NVIDIA SOFTWARE DEVELOPMENT KITS + +This license agreement, including exhibits attached ("Agreement”) is a legal agreement between you and NVIDIA Corporation ("NVIDIA") and governs your use of a NVIDIA software development kit (“SDK”). + +Each SDK has its own set of software and materials, but here is a description of the types of items that may be included in a SDK: source code, header files, APIs, data sets and assets (examples include images, textures, models, scenes, videos, native API input/output files), binary software, sample code, libraries, utility programs, programming code and documentation. + +This Agreement can be accepted only by an adult of legal age of majority in the country in which the SDK is used. + +If you are entering into this Agreement on behalf of a company or other legal entity, you represent that you have the legal authority to bind the entity to this Agreement, in which case “you” will mean the entity you represent. + +If you don’t have the required age or authority to accept this Agreement, or if you don’t accept all the terms and conditions of this Agreement, do not download, install or use the SDK. + +You agree to use the SDK only for purposes that are permitted by (a) this Agreement, and (b) any applicable law, regulation or generally accepted practices or guidelines in the relevant jurisdictions. + +1. License. + +1.1 Grant + +Subject to the terms of this Agreement, NVIDIA hereby grants you a non-exclusive, non-transferable license, without the right to sublicense (except as expressly provided in this Agreement) to: + +(i) Install and use the SDK, + +(ii) Modify and create derivative works of sample source code delivered in the SDK, and + +(iii) Distribute those portions of the SDK that are identified in this Agreement as distributable, as incorporated in object code format into a software application that meets the distribution requirements indicated in this Agreement. + +1.2 Distribution Requirements + +These are the distribution requirements for you to exercise the distribution grant: + +(i) Your application must have material additional functionality, beyond the included portions of the SDK. + +(ii) The distributable portions of the SDK shall only be accessed by your application. + +(iii) The following notice shall be included in modifications and derivative works of sample source code distributed: “This software contains source code provided by NVIDIA Corporation.” + +(iv) Unless a developer tool is identified in this Agreement as distributable, it is delivered for your internal use only. + +(v) The terms under which you distribute your application must be consistent with the terms of this Agreement, including (without limitation) terms relating to the license grant and license restrictions and protection of NVIDIA’s intellectual property rights. Additionally, you agree that you will protect the privacy, security and legal rights of your application users. + +(vi) You agree to notify NVIDIA in writing of any known or suspected distribution or use of the SDK not in compliance with the requirements of this Agreement, and to enforce the terms of your agreements with respect to distributed SDK. + +1.3 Authorized Users + +You may allow employees and contractors of your entity or of your subsidiary(ies) to access and use the SDK from your secure network to perform work on your behalf. + +If you are an academic institution you may allow users enrolled or employed by the academic institution to access and use the SDK from your secure network. + +You are responsible for the compliance with the terms of this Agreement by your authorized users. If you become aware that your authorized users didn’t follow the terms of this Agreement, you agree to take reasonable steps to resolve the non-compliance and prevent new occurrences. + +1.4 Pre-Release SDK +The SDK versions identified as alpha, beta, preview or otherwise as pre-release, may not be fully functional, may contain errors or design flaws, and may have reduced or different security, privacy, accessibility, availability, and reliability standards relative to commercial versions of NVIDIA software and materials. Use of a pre-release SDK may result in unexpected results, loss of data, project delays or other unpredictable damage or loss. +You may use a pre-release SDK at your own risk, understanding that pre-release SDKs are not intended for use in production or business-critical systems. +NVIDIA may choose not to make available a commercial version of any pre-release SDK. NVIDIA may also choose to abandon development and terminate the availability of a pre-release SDK at any time without liability. +1.5 Updates + +NVIDIA may, at its option, make available patches, workarounds or other updates to this SDK. Unless the updates are provided with their separate governing terms, they are deemed part of the SDK licensed to you as provided in this Agreement. + +You agree that the form and content of the SDK that NVIDIA provides may change without prior notice to you. While NVIDIA generally maintains compatibility between versions, NVIDIA may in some cases make changes that introduce incompatibilities in future versions of the SDK. + +1.6 Third Party Licenses + +The SDK may come bundled with, or otherwise include or be distributed with, third-party software licensed by a NVIDIA supplier and/or open source software provided under an open source license. Use of third-party software is subject to the third-party license terms, or in the absence of third-party terms, the terms of this Agreement. Copyright to third party software is held by the copyright holders indicated in the third-party software or license. + +1.7 Reservation of Rights + +NVIDIA reserves all rights, title and interest in and to the SDK not expressly granted to you under this Agreement. + +2. Limitations. + +The following license limitations apply to your use of the SDK: + +2.1 You may not reverse engineer, decompile or disassemble, or remove copyright or other proprietary notices from any portion of the SDK or copies of the SDK. + +2.2 Except as expressly provided in this Agreement, you may not copy, sell, rent, sublicense, transfer, distribute, modify, or create derivative works of any portion of the SDK. For clarity, you may not distribute or sublicense the SDK as a stand-alone product. + +2.3 Unless you have an agreement with NVIDIA for this purpose, you may not indicate that an application created with the SDK is sponsored or endorsed by NVIDIA. + +2.4 You may not bypass, disable, or circumvent any encryption, security, digital rights management or authentication mechanism in the SDK. + +2.5 You may not use the SDK in any manner that would cause it to become subject to an open source software license. As examples, licenses that require as a condition of use, modification, and/or distribution that the SDK be (i) disclosed or distributed in source code form; (ii) licensed for the purpose of making derivative works; or (iii) redistributable at no charge. + +2.6 Unless you have an agreement with NVIDIA for this purpose, you may not use the SDK with any system or application where the use or failure of the system or application can reasonably be expected to threaten or result in personal injury, death, or catastrophic loss. Examples include use in avionics, navigation, military, medical, life support or other life critical applications. NVIDIA does not design, test or manufacture the SDK for these critical uses and NVIDIA shall not be liable to you or any third party, in whole or in part, for any claims or damages arising from such uses. + +2.7 You agree to defend, indemnify and hold harmless NVIDIA and its affiliates, and their respective employees, contractors, agents, officers and directors, from and against any and all claims, damages, obligations, losses, liabilities, costs or debt, fines, restitutions and expenses (including but not limited to attorney’s fees and costs incident to establishing the right of indemnification) arising out of or related to your use of the SDK outside of the scope of this Agreement, or not in compliance with its terms. + +3. Ownership. + +3.1 NVIDIA or its licensors hold all rights, title and interest in and to the SDK and its modifications and derivative works, including their respective intellectual property rights, subject to your rights under Section 3.2. This SDK may include software and materials from NVIDIA’s licensors, and these licensors are intended third party beneficiaries that may enforce this Agreement with respect to their intellectual property rights. + +3.2 You hold all rights, title and interest in and to your applications and your derivative works of the sample source code delivered in the SDK, including their respective intellectual property rights, subject to NVIDIA’s rights under section 3.1. + +3.3 You may, but don’t have to, provide to NVIDIA suggestions, feature requests or other feedback regarding the SDK, including possible enhancements or modifications to the SDK. For any feedback that you voluntarily provide, you hereby grant NVIDIA and its affiliates a perpetual, non-exclusive, worldwide, irrevocable license to use, reproduce, modify, license, sublicense (through multiple tiers of sublicensees), and distribute (through multiple tiers of distributors) it without the payment of any royalties or fees to you. NVIDIA will use feedback at its choice. NVIDIA is constantly looking for ways to improve its products, so you may send feedback to NVIDIA through the developer portal at https://developer.nvidia.com. + +4. No Warranties. + +THE SDK IS PROVIDED BY NVIDIA “AS IS” AND “WITH ALL FAULTS.” TO THE MAXIMUM EXTENT PERMITTED BY LAW, NVIDIA AND ITS AFFILIATES EXPRESSLY DISCLAIM ALL WARRANTIES OF ANY KIND OR NATURE, WHETHER EXPRESS, IMPLIED OR STATUTORY, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, NON-INFRINGEMENT, OR THE ABSENCE OF ANY DEFECTS THEREIN, WHETHER LATENT OR PATENT. NO WARRANTY IS MADE ON THE BASIS OF TRADE USAGE, COURSE OF DEALING OR COURSE OF TRADE. + +5. Limitations of Liability. + +TO THE MAXIMUM EXTENT PERMITTED BY LAW, NVIDIA AND ITS AFFILIATES SHALL NOT BE LIABLE FOR ANY SPECIAL, INCIDENTAL, PUNITIVE OR CONSEQUENTIAL DAMAGES, OR ANY LOST PROFITS, LOSS OF USE, LOSS OF DATA OR LOSS OF GOODWILL, OR THE COSTS OF PROCURING SUBSTITUTE PRODUCTS, ARISING OUT OF OR IN CONNECTION WITH THIS AGREEMENT OR THE USE OR PERFORMANCE OF THE SDK, WHETHER SUCH LIABILITY ARISES FROM ANY CLAIM BASED UPON BREACH OF CONTRACT, BREACH OF WARRANTY, TORT (INCLUDING NEGLIGENCE), PRODUCT LIABILITY OR ANY OTHER CAUSE OF ACTION OR THEORY OF LIABILITY. IN NO EVENT WILL NVIDIA’S AND ITS AFFILIATES TOTAL CUMULATIVE LIABILITY UNDER OR ARISING OUT OF THIS AGREEMENT EXCEED US$10.00. THE NATURE OF THE LIABILITY OR THE NUMBER OF CLAIMS OR SUITS SHALL NOT ENLARGE OR EXTEND THIS LIMIT. + +These exclusions and limitations of liability shall apply regardless if NVIDIA or its affiliates have been advised of the possibility of such damages, and regardless of whether a remedy fails its essential purpose. These exclusions and limitations of liability form an essential basis of the bargain between the parties, and, absent any of these exclusions or limitations of liability, the provisions of this Agreement, including, without limitation, the economic terms, would be substantially different. + +6. Termination. + +6.1 This Agreement will continue to apply until terminated by either you or NVIDIA as described below. + +6.2 If you want to terminate this Agreement, you may do so by stopping to use the SDK. + +6.3 NVIDIA may, at any time, terminate this Agreement if: (i) you fail to comply with any term of this Agreement and the non-compliance is not fixed within thirty (30) days following notice from NVIDIA (or immediately if you violate NVIDIA’s intellectual property rights); (ii) you commence or participate in any legal proceeding against NVIDIA with respect to the SDK; or (iii) NVIDIA decides to no longer provide the SDK in a country or, in NVIDIA’s sole discretion, the continued use of it is no longer commercially viable. + +6.4 Upon any termination of this Agreement, you agree to promptly discontinue use of the SDK and destroy all copies in your possession or control. Your prior distributions in accordance with this Agreement are not affected by the termination of this Agreement. Upon written request, you will certify in writing that you have complied with your commitments under this section. Upon any termination of this Agreement all provisions survive except for the licenses granted to you. + +7. General. + +If you wish to assign this Agreement or your rights and obligations, including by merger, consolidation, dissolution or operation of law, contact NVIDIA to ask for permission. Any attempted assignment not approved by NVIDIA in writing shall be void and of no effect. NVIDIA may assign, delegate or transfer this Agreement and its rights and obligations, and if to a non-affiliate you will be notified. + +You agree to cooperate with NVIDIA and provide reasonably requested information to verify your compliance with this Agreement. + +This Agreement will be governed in all respects by the laws of the United States and of the State of Delaware as those laws are applied to contracts entered into and performed entirely within Delaware by Delaware residents, without regard to the conflicts of laws principles. The United Nations Convention on Contracts for the International Sale of Goods is specifically disclaimed. You agree to all terms of this Agreement in the English language. + +The state or federal courts residing in Santa Clara County, California shall have exclusive jurisdiction over any dispute or claim arising out of this Agreement. Notwithstanding this, you agree that NVIDIA shall still be allowed to apply for injunctive remedies or an equivalent type of urgent legal relief in any jurisdiction. + +If any court of competent jurisdiction determines that any provision of this Agreement is illegal, invalid or unenforceable, such provision will be construed as limited to the extent necessary to be consistent with and fully enforceable under the law and the remaining provisions will remain in full force and effect. Unless otherwise specified, remedies are cumulative. + +Each party acknowledges and agrees that the other is an independent contractor in the performance of this Agreement. + +The SDK has been developed entirely at private expense and is “commercial items” consisting of “commercial computer software” and “commercial computer software documentation” provided with RESTRICTED RIGHTS. Use, duplication or disclosure by the U.S. Government or a U.S. Government subcontractor is subject to the restrictions in this Agreement pursuant to DFARS 227.7202-3(a) or as set forth in subparagraphs (b)(1) and (2) of the Commercial Computer Software - Restricted Rights clause at FAR 52.227-19, as applicable. Contractor/manufacturer is NVIDIA, 2788 San Tomas Expressway, Santa Clara, CA 95051. + +The SDK is subject to United States export laws and regulations. You agree that you will not ship, transfer or export the SDK into any country, or use the SDK in any manner, prohibited by the United States Bureau of Industry and Security or economic sanctions regulations administered by the U.S. Department of Treasury’s Office of Foreign Assets Control (OFAC), or any applicable export laws, restrictions or regulations. These laws include restrictions on destinations, end users and end use. By accepting this Agreement, you confirm that you are not a resident or citizen of any country currently embargoed by the U.S. and that you are not otherwise prohibited from receiving the SDK. + +Any notice delivered by NVIDIA to you under this Agreement will be delivered via mail, email or fax. You agree that any notices that NVIDIA sends you electronically will satisfy any legal communication requirements. Please direct your legal notices or other correspondence to NVIDIA Corporation, 2788 San Tomas Expressway, Santa Clara, California 95051, United States of America, Attention: Legal Department. + +This Agreement and any exhibits incorporated into this Agreement constitute the entire agreement of the parties with respect to the subject matter of this Agreement and supersede all prior negotiations or documentation exchanged between the parties relating to this subject matter. Any additional and/or conflicting terms on documents issued by you are null, void, and invalid. Any amendment or waiver under this Agreement shall be in writing and signed by representatives of both parties. + +(v. October 12, 2020) + + + + + + + + + + + + + + + +cuSPARSELt SUPPLEMENT TO SOFTWARE LICENSE AGREEMENT FOR NVIDIA SOFTWARE DEVELOPMENT KITS + +The terms in this supplement govern your use of the NVIDIA cuSPARSELt SDK under the terms of your license agreement (“Agreement”) as modified by this supplement. Capitalized terms used but not defined below have the meaning assigned to them in the Agreement. + +This supplement is an exhibit to the Agreement and is incorporated as an integral part of the Agreement. In the event of conflict between the terms in this supplement and the terms in the Agreement, the terms in this supplement govern. + +1. License Scope. The SDK is licensed for you to develop applications only for use in systems with NVIDIA GPUs. + +2. Distribution. The following portions of the SDK are distributable under the Agreement: the runtimes files ending with .so and .h as part of your application. + +3. Licensing. If the distribution terms in this Agreement are not suitable for your organization, or for any questions regarding this Agreement, please contact NVIDIA at nvidia-compute-license-questions@nvidia.com + +(v. October 12, 2020) + + +nvidia-dali-cuda120 +Apache License 2.0 +https://github.com/NVIDIA/dali + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + + +nvidia-libnvcomp-cu12 +LICENSE AGREEMENT FOR NVIDIA SOFTWARE DEVELOPMENT KITS +https://developer.nvidia.com/nvcomp +LICENSE AGREEMENT FOR NVIDIA SOFTWARE DEVELOPMENT KITS + +This license agreement("Agreement") is a legal agreement between you and NVIDIA Corporation ("NVIDIA") and governs your use of the NVIDIA nvCOMP software development kit as available at NVIDIA's discretion (each, a "SDK"). + +Each SDK has its own set of software and materials, but here is a description of the types of items that may be included in a SDK: source code, header files, APIs, data sets and assets (examples include images, textures, models, scenes, videos, native API input/output files), binary software, sample code, libraries, utility programs, programming code and documentation. + +This Agreement can be accepted only by an adult of legal age of majority in the country in which the SDK is used. + +If you are entering into this Agreement on behalf of a company or other legal entity, you represent that you have the legal authority to bind the entity to this Agreement, in which case "you" will mean the entity you represent. + +If you don't have the required age or authority to accept this Agreement, or if you don't accept all the terms and conditions of this Agreement, do not download, install or use the SDK. + +You agree to use the SDK only for purposes that are permitted by (a) this Agreement, and (b) any applicable law, regulation or generally accepted practices or guidelines in the relevant jurisdictions. + +1. License. + +1.1 Grant + +Subject to the terms of this Agreement, NVIDIA hereby grants you a non-exclusive, non-transferable license, without the right to sublicense (except as expressly provided in this Agreement) to: + +(i) Install and use the SDK, + +(ii) Modify and create derivative works of sample source code delivered in the SDK, and + +(ii) Distribute the binary files, files identified as samples, and headers as incorporated into a software application that meets the distribution requirements indicated in this Agreement. + +1.2 Distribution Requirements + +These are the distribution requirements for you to exercise the distribution grant: + +(i) Your application must have material additional functionality, beyond the included portions of the SDK. + +(ii) The distributable portions of the SDK shall only be accessed by your application. + +(iii) The following notice shall be included in modifications and derivative works of sample source code distributed: "This software contains source code provided by NVIDIA Corporation." + +(iv) Unless a developer tool is identified in this Agreement as distributable, it is delivered for your internal use only. + +(v) The terms under which you distribute your application must be consistent with the terms of this Agreement, including (without limitation) terms relating to the license grant and license restrictions and protection of NVIDIA's intellectual property rights. Additionally, you agree that you will protect the privacy, security and legal rights of your application users. + +(vi) You agree to notify NVIDIA in writing of any known or suspected distribution or use of the SDK not in compliance with the requirements of this Agreement, and to enforce the terms of your agreements with respect to distributed SDK. + +1.3 Authorized Users + +You may allow employees and contractors of your entity or of your subsidiary(ies) to access and use the SDK from your secure network to perform work on your behalf. + +If you are an academic institution you may allow users enrolled or employed by the academic institution to access and use the SDK from your secure network. + +You are responsible for the compliance with the terms of this Agreement by your authorized users. If you become aware that your authorized users didn't follow the terms of this Agreement, you agree to take reasonable steps to resolve the non-compliance and prevent new occurrences. + +1.4 Pre-Release SDK + +The SDK versions identified as alpha, beta, preview or otherwise as pre-release, may not be fully functional, may contain errors or design flaws, and may have reduced or different security, privacy, accessibility, availability, and reliability standards relative to commercial versions of NVIDIA software and materials. Use of a pre-release SDK may result in unexpected results, loss of data, project delays or other unpredictable damage or loss. + +You may use a pre-release SDK at your own risk, understanding that pre-release SDKs are not intended for use in production or business-critical systems. + +NVIDIA may choose not to make available a commercial version of any pre-release SDK. NVIDIA may also choose to abandon development and terminate the availability of a pre-release SDK at any time without liability. + +1.5 Updates + +NVIDIA may, at its option, make available patches, workarounds or other updates to this SDK. Unless the updates are provided with their separate governing terms, they are deemed part of the SDK licensed to you as provided in this Agreement. + +You agree that the form and content of the SDK that NVIDIA provides may change without prior notice to you. While NVIDIA generally maintains compatibility between versions, NVIDIA may in some cases make changes that introduce incompatibilities in future versions of the SDK. + +1.6 Components Under Other Licenses. + +The SDK may include NVIDIA or third-party components with separate legal notices or terms as may be described in proprietary notices accompanying the SDK, such as components governed by open source software licenses. If and to the extent there is a conflict between the terms in this license and the license terms associated with a component, the license terms associated with the components control only to the extent necessary to resolve the conflict. + +1.7 Reservation of Rights + +NVIDIA reserves all rights, title and interest in and to the SDK not expressly granted to you under this Agreement. + +2. Limitations. + +The following license limitations apply to your use of the SDK: + +2.1 The SDK is licensed for you to develop applications only for use in systems with NVIDIA GPUs. + +2.2 You may not reverse engineer, decompile or disassemble, or remove copyright or other proprietary notices from any portion of the SDK or copies of the SDK. + +2.3 Except as expressly provided in this Agreement, you may not copy, sell, rent, sublicense, transfer, distribute, modify, or create derivative works of any portion of the SDK. + +2.4 Unless you have an agreement with NVIDIA for this purpose, you may not indicate that an application created with the SDK is sponsored or endorsed by NVIDIA. + +2.5 You may not bypass, disable, or circumvent any encryption, security, digital rights management or authentication mechanism in the SDK. + +2.6 You may not use the SDK in any manner that would cause it to become subject to an open source software license. As examples, licenses that require as a condition of use, modification, and/or distribution that the SDK be (i) disclosed or distributed in source code form; (ii) licensed for the purpose of making derivative works; or (iii) redistributable at no charge. + +2.7 You acknowledge that the SDK as delivered is not tested or certified by NVIDIA for use in connection with the design, construction, maintenance, and/or operation of any system where the use or failure of such system could result in a situation that threatens the safety of human life or results in catastrophic damages (each, a "Critical Application"). Examples of Critical Applications include use in avionics, navigation, autonomous vehicle applications, ai solutions for automotive products, military, medical, life support or other life critical applications. NVIDIA shall not be liable to you or any third party, in whole or in part, for any claims or damages arising from such uses. You are solely responsible for ensuring that any product or service developed with the SDK as a whole includes sufficient features to comply with all applicable legal and regulatory standards and requirements. + +2.8 You agree to defend, indemnify and hold harmless NVIDIA and its affiliates, and their respective employees, contractors, agents, officers and directors, from and against any and all claims, damages, obligations, losses, liabilities, costs or debt, fines, restitutions and expenses (including but not limited to attorney's fees and costs incident to establishing the right of indemnification) arising out of or related to products or services that use the SDK in or for Critical Applications, and for use of the SDK outside of the scope of this Agreement, or not in compliance with its terms. + +3. Ownership. + +3.1 NVIDIA or its licensors hold all rights, title and interest in and to the SDK and its modifications and derivative works, including their respective intellectual property rights, subject to your rights under Section 3.2. This SDK may include software and materials from NVIDIA's licensors, and these licensors are intended third party beneficiaries that may enforce this Agreement with respect to their intellectual property rights. + +3.2 You hold all rights, title and interest in and to your applications and your derivative works of the sample source code delivered in the SDK, including their respective intellectual property rights, subject to NVIDIA's rights under section 3.1. + +3.3 You may, but don't have to, provide to NVIDIA suggestions, feature requests or other feedback regarding the SDK, including possible enhancements or modifications to the SDK. For any feedback that you voluntarily provide, you hereby grant NVIDIA and its affiliates a perpetual, non-exclusive, worldwide, irrevocable license to use, reproduce, modify, license, sublicense (through multiple tiers of sublicensees), and distribute (through multiple tiers of distributors) it without the payment of any royalties or fees to you. NVIDIA will use feedback at its choice. + +4. No Warranties. + +THE SDK IS PROVIDED BY NVIDIA "AS IS" AND "WITH ALL FAULTS." TO THE MAXIMUM EXTENT PERMITTED BY LAW, NVIDIA AND ITS AFFILIATES EXPRESSLY DISCLAIM ALL WARRANTIES OF ANY KIND OR NATURE, WHETHER EXPRESS, IMPLIED OR STATUTORY, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, NON-INFRINGEMENT, OR THE ABSENCE OF ANY DEFECTS THEREIN, WHETHER LATENT OR PATENT. NO WARRANTY IS MADE ON THE BASIS OF TRADE USAGE, COURSE OF DEALING OR COURSE OF TRADE. + +5. Limitations of Liability. + +TO THE MAXIMUM EXTENT PERMITTED BY LAW, NVIDIA AND ITS AFFILIATES SHALL NOT BE LIABLE FOR ANY SPECIAL, INCIDENTAL, PUNITIVE OR CONSEQUENTIAL DAMAGES, OR ANY LOST PROFITS, LOSS OF USE, LOSS OF DATA OR LOSS OF GOODWILL, OR THE COSTS OF PROCURING SUBSTITUTE PRODUCTS, ARISING OUT OF OR IN CONNECTION WITH THIS AGREEMENT OR THE USE OR PERFORMANCE OF THE SDK, WHETHER SUCH LIABILITY ARISES FROM ANY CLAIM BASED UPON BREACH OF CONTRACT, BREACH OF WARRANTY, TORT (INCLUDING NEGLIGENCE), PRODUCT LIABILITY OR ANY OTHER CAUSE OF ACTION OR THEORY OF LIABILITY. IN NO EVENT WILL NVIDIA'S AND ITS AFFILIATES TOTAL CUMULATIVE LIABILITY UNDER OR ARISING OUT OF THIS AGREEMENT EXCEED US$10.00. THE NATURE OF THE LIABILITY OR THE NUMBER OF CLAIMS OR SUITS SHALL NOT ENLARGE OR EXTEND THIS LIMIT. + +These exclusions and limitations of liability shall apply regardless if NVIDIA or its affiliates have been advised of the possibility of such damages, and regardless of whether a remedy fails its essential purpose. These exclusions and limitations of liability form an essential basis of the bargain between the parties, and, absent any of these exclusions or limitations of liability, the provisions of this Agreement, including, without limitation, the economic terms, would be substantially different. + +6. Termination. + +6.1 This Agreement will continue to apply until terminated by either you or NVIDIA as described below. + +6.2 If you want to terminate this Agreement, you may do so by stopping to use the SDK. + +6.3 NVIDIA may, at any time, terminate this Agreement if: (i) you fail to comply with any term of this Agreement and the non-compliance is not fixed within thirty (30) days following notice from NVIDIA (or immediately if you violate NVIDIA's intellectual property rights); (ii) you commence or participate in any legal proceeding against NVIDIA with respect to the SDK; or (iii) NVIDIA decides to no longer provide the SDK in a country or, in NVIDIA's sole discretion, the continued use of it is no longer commercially viable. + +6.4 Upon any termination of this Agreement, you agree to promptly discontinue use of the SDK and destroy all copies in your possession or control. Your prior distributions in accordance with this Agreement are not affected by the termination of this Agreement. Upon written request, you will certify in writing that you have complied with your commitments under this section. Upon any termination of this Agreement all provisions survive except for the licenses granted to you. + +7. General. + +If you wish to assign this Agreement or your rights and obligations, including by merger, consolidation, dissolution or operation of law, contact NVIDIA to ask for permission. Any attempted assignment not approved by NVIDIA in writing shall be void and of no effect. NVIDIA may assign, delegate or transfer this Agreement and its rights and obligations. + +You agree to cooperate with NVIDIA and provide reasonably requested information to verify your compliance with this Agreement. + +This Agreement will be governed in all respects by the laws of the United States and of the State of Delaware as those laws are applied to contracts entered into and performed entirely within Delaware, without regard to the conflicts of laws principles. The United Nations Convention on Contracts for the International Sale of Goods is specifically disclaimed. You agree to all terms of this Agreement in the English language. + +The state or federal courts residing in Santa Clara County, California shall have exclusive jurisdiction over any dispute or claim arising out of this Agreement. Notwithstanding this, you agree that NVIDIA shall still be allowed to apply for injunctive remedies or an equivalent type of urgent legal relief in any jurisdiction. + +If any court of competent jurisdiction determines that any provision of this Agreement is illegal, invalid or unenforceable, such provision will be construed as limited to the extent necessary to be consistent with and fully enforceable under the law and the remaining provisions will remain in full force and effect. Unless otherwise specified, remedies are cumulative. + +Each party acknowledges and agrees that the other is an independent contractor in the performance of this Agreement. + +The SDK has been developed entirely at private expense and is "commercial items" consisting of "commercial computer software" and "commercial computer software documentation" provided with RESTRICTED RIGHTS. Use, duplication or disclosure by the U.S. Government or a U.S. Government subcontractor is subject to the restrictions in this Agreement pursuant to DFARS 227.7202-3(a) or as set forth in subparagraphs (b)(1) and (2) of the Commercial Computer Software - Restricted Rights clause at FAR 52.227-19, as applicable. Contractor/manufacturer is NVIDIA, 2788 San Tomas Expressway, Santa Clara, CA 95051. + +The SDK is subject to United States export laws and regulations. You agree that you will not ship, transfer or export the SDK into any country, or use the SDK in any manner, prohibited by the United States Bureau of Industry and Security or economic sanctions regulations administered by the U.S. Department of Treasury's Office of Foreign Assets Control (OFAC), or any applicable export laws, restrictions or regulations. These laws include restrictions on destinations, end users and end use. By accepting this Agreement, you confirm that you are not a resident or citizen of any country currently embargoed by the U.S. and that you are not otherwise prohibited from receiving the SDK. + +Any notice delivered by NVIDIA to you under this Agreement will be delivered via mail, email or fax. You agree that any notices that NVIDIA sends you electronically will satisfy any legal communication requirements. Please direct your legal notices or other correspondence to NVIDIA Corporation, 2788 San Tomas Expressway, Santa Clara, California 95051, United States of America, Attention: Legal Department. + +This Agreement constitutes the entire agreement of the parties with respect to the subject matter of this Agreement and supersedes all prior negotiations or documentation exchanged between the parties relating to this subject matter. Any additional and/or conflicting terms on documents issued by you are null, void, and invalid. Any amendment or waiver under this Agreement shall be in writing and signed by representatives of both parties. + +If the distribution terms in this Agreement are not suitable for your organization, or for any questions regarding this Agreement, please contact NVIDIA at nvidia-compute-license-questions@nvidia.com. + +(v. April 26, 2022) + + +nvidia-ml-py +BSD License +https://forums.developer.nvidia.com +UNKNOWN + +nvidia-nccl-cu13 +LicenseRef-NVIDIA-Proprietary +https://developer.nvidia.com/cuda-zone +UNKNOWN + +nvidia-nvimgcodec-cu12 +Apache License 2.0 +https://github.com/NVIDIA/nvImageCodec +NVIDIA Software License Agreement + +IMPORTANT NOTICE – PLEASE READ AND AGREE BEFORE USING THE SOFTWARE. + +This license agreement (“Agreement”) is a legal agreement between you, whether an individual or entity (“you”) and NVIDIA Corporation (“NVIDIA”) and governs the use of the NVIDIA Image Codec and any additional software and materials provided under this Agreement (“Software”). + +This Agreement can be accepted only by an adult of legal age of majority in the country in which the Software is used. + +If you don’t have the required age or authority to accept this Agreement, or if you don’t accept all the terms and conditions of this Agreement, do not use the Software. + +You agree to use the Software only for purposes that are permitted by this Agreement and any applicable law or regulation in the relevant jurisdictions. + +1. License Grant. Subject to the terms of this Agreement, NVIDIA grants you a non-exclusive, revocable, non-transferable, non-sublicensable (except as expressly granted in this Agreement), license to: + +1.1 install and use copies of the Software, + +1.2 modify and create derivative works of sample or example Software provided by NVIDIA in source code format, and + +1.3 distribute the Software in binary format as incorporated into a software application subject to the following distribution requirements: + +(a) Your application must have material additional functionality, beyond the included portions of the Software. + +(b) The distributable portions of the Software shall only be accessed by your application. + +(c) The following notice shall be included in modifications and derivative works of sample source code distributed: “This software contains source code provided by NVIDIA Corporation.” + +(d) Unless a developer tool is identified in this Agreement as distributable, it is delivered for your internal use only. + +(e) The terms under which you distribute your application must be consistent with the terms of this Agreement, including (without limitation) terms relating to the license grant and license restrictions and protection of NVIDIA’s intellectual property rights. Additionally, you agree that you will protect the privacy, security and legal rights of your application users. + +(f) You agree to notify NVIDIA in writing of any known or suspected distribution or use of the Software not in compliance with the requirements of this Agreement, and to enforce the terms of your agreements with respect to distributed Software. + +2. Limitations. Your license to use the Software is restricted as follows: + +2.1 The Software is licensed for you to develop applications only for use in systems with NVIDIA GPUs and NVIDIA CPUs (if and when available). + +2.2 You may not reverse engineer, decompile or disassemble the Software components provided in binary form, nor attempt in any other manner to obtain source code of the Software. + +2.3 You may not change or remove copyright or other proprietary notices in the Software. + +2.4 Except as expressly granted in this Agreement, you may not copy, sell, rent, sublicense, transfer, distribute, modify or create derivative works of the Software, or make its functionality available to others. + +2.5 You may not bypass, disable or circumvent any technical limitation, encryption, security, digital rights management or authentication mechanism in the Software. + +2.6 You may not use the Software in any manner that would cause it to become subject to an open source software license; subject to the terms in the “Components Under Other Licenses” section below . + +2.7 You may not use the Software for the purpose of developing competing products or technologies or assist a third party in such activities. + +2.8 You may not indicate that a product or service developed with the Software is sponsored or endorsed by NVIDIA. + +2.9 You may not replace any NVIDIA software components in the Software that are governed by this Agreement with other software that implements NVIDIA APIs. + +2.10 You may not reverse engineer, decompile or disassemble any portion of the output generated using Software elements for the purpose of translating such output artifacts to target a non-NVIDIA platform. + +2.11 You may not distribute or disclose to third parties the output of the Software where the output reveals functionality or performance data pertinent to NVIDIA hardware or software products, results of benchmarking, competitive analysis, or regression or performance data relating to the Software or NVIDIA GPUs without the prior written permission from NVIDIA. + +2.12 You acknowledge that the Software provided under this Agreement is not designed or tested by NVIDIA for use in any system or application where the use or failure of such system or application developed with NVIDIA’s Software could result in injury, death or catastrophic damage (each, a “Mission Critical Application”). Examples of Mission Critical Applications include use in avionics, navigation, autonomous vehicle applications, AI solutions for automotive products, military, medical, life support or other mission-critical or life-critical applications. NVIDIA will not be liable to you or any third party, in whole or in part, for any claims or damages arising from these uses. You are solely responsible for ensuring that systems and applications developed with the Software include sufficient safety and redundancy features and comply with all applicable legal and regulatory standards and requirements. + +2.13 You agree to defend, indemnify and hold harmless NVIDIA and its affiliates, and their respective employees, contractors, agents, officers and directors, from and against any and all claims, damages, obligations, losses, liabilities, costs or debt, fines, restitutions and expenses (including but not limited to attorney’s fees and costs incident to establishing the right of indemnification) arising out of or related to (i) products or services that have been developed or deployed with or use the Software, or claims that they violate laws, or infringe, violate, or misappropriate any third party right; or (ii) a violation of the terms and conditions of this Agreement. + +3. Authorized Users. You may allow employees and contractors of your entity or of your subsidiary(ies) to access and use the Software from your secure network to perform the work authorized by this Agreement on your behalf. If you are an academic institution, you may allow users enrolled or employed by the academic institution to access and use the Software as authorized by this Agreement from your secure network. You are responsible for the compliance with the terms of this Agreement by your authorized users. Any act or omission that if committed by you would constitute a breach of this Agreement will be deemed to constitute a breach of this Agreement if committed by your authorized users. + +4. Pre-Release Versions. Software versions or specific features identified as alpha, beta, preview, early access or otherwise as pre-release may not be fully functional, may contain errors or design flaws, and may have reduced or different security, privacy, availability and reliability standards relative to commercial versions of NVIDIA offerings. You may use pre-release Software at your own risk, understanding that such versions are not intended for use in production or business-critical systems. NVIDIA may choose not to make available a commercial version of any pre-release Software. NVIDIA may also choose to abandon development and terminate the availability of pre-release Software at any time without liability. + +5. Updates. NVIDIA may, at its option, make available patches, workarounds or other updates to the Software. Unless the updates are provided with their separate governing terms, they are deemed part of the Software licensed to you as provided in this Agreement. + +6. Components Under Other Licenses. The Software may include or be distributed with components provided with separate legal notices or terms that accompany the components, such as open source software licenses and other license. The components are subject to the applicable other licenses, including any proprietary notices, disclaimers, requirements and extended use rights; except that this Agreement will prevail regarding the use of third-party open source software, unless a third-party open source software license requires its license terms to prevail. Open source software license means any software, data or documentation subject to any license identified as an open source license by the Open Source Initiative (http://opensource.org), Free Software Foundation (http://www.fsf.org) or other similar open source organization or listed by the Software Package Data Exchange (SPDX) Workgroup under the Linux Foundation (http://www.spdx.org). + +7. Termination . This Agreement will automatically terminate without notice from NVIDIA if you fail to comply with any of the terms in this Agreement or if you commence or participate in any legal proceeding against NVIDIA with respect to the Software. Additionally, NVIDIA may terminate this Agreement with prior written notice to you if, in NVIDIA’s sole discretion, the continued use of the Software is no longer commercially viable or creates liabilities for NVIDIA. You agree to cooperate with NVIDIA and provide reasonably requested information to verify your compliance with this Agreement. Upon any termination, you must stop using and destroy all copies of the Software. Upon written request, you will certify in writing that you have complied with your commitments under this section. All provisions will survive termination, except for the licenses granted to you. + +8. Ownership. + +8.1 NVIDIA Ownership. The Software, including all intellectual property rights, is and will remain the sole and exclusive property of NVIDIA or its licensors. Except as expressly granted in this Agreement, (i) NVIDIA reserves all rights, interests and remedies in connection with the Software and (ii) no other license or right is granted to you by implication, estoppel or otherwise. + +8.2 Your Ownership. Subject to the rights of NVIDIA and its suppliers in the Software, you hold all rights, title and interest in and to your services, applications and derivative works of samples or examples you develop as permitted in this Agreement including their respective intellectual property rights. + +8.3 Non-Assert. You agree that you will not, and will not assist or enable any other party to, assert or threaten to assert any intellectual property rights against NVIDIA or its affiliates with respect to new software samples or examples that NVIDIA or its affiliates may develop and make available in the future. + +9. Feedback. You may, but are not obligated to, provide suggestions, requests, fixes, modifications, enhancements or other feedback regarding or in connection with your use of the Software (collectively, “Feedback”). Feedback, even if designated as confidential by you, will not create any confidentiality obligation for NVIDIA or its affiliates. If you provide Feedback, you hereby grant NVIDIA, its affiliates and its designees a non-exclusive, perpetual, irrevocable, sublicensable, worldwide, royalty-free, fully paid-up and transferable license, under your intellectual property rights, to publicly perform, publicly display, reproduce, use, make, have made, sell, offer for sale, distribute (through multiple tiers of distribution), import, create derivative works of and otherwise commercialize and exploit the Feedback at NVIDIA’s discretion. You will not give Feedback (i) that you have reason to believe is subject to any restriction that impairs the exercise of the grant stated in this section, such as third-party intellectual property rights or (ii) subject to license terms which seek to require any product incorporating or developed using such Feedback, or other intellectual property of NVIDIA or its affiliates, to be licensed to or otherwise shared with any third party. + +10. Disclaimer of Warranties. THE SOFTWARE IS PROVIDED BY NVIDIA AS-IS AND WITH ALL FAULTS. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, NVIDIA DISCLAIMS ALL WARRANTIES AND REPRESENTATIONS OF ANY KIND, WHETHER EXPRESS, IMPLIED OR STATUTORY, RELATING TO OR ARISING UNDER THIS AGREEMENT, INCLUDING, WITHOUT LIMITATION, THE WARRANTIES OF TITLE, NONINFRINGEMENT, MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, USAGE OF TRADE AND COURSE OF DEALING. WITHOUT LIMITING THE FOREGOING, NVIDIA DOES NOT WARRANT THAT THE SOFTWARE WILL MEET YOUR REQUIREMENTS; THAT ANY DEFECTS OR ERRORS WILL BE CORRECTED; THAT ANY CERTAIN CONTENT WILL BE AVAILABLE; OR THAT THE SOFTWARE IS FREE OF VIRUSES OR OTHER HARMFUL COMPONENTS. NO INFORMATION OR ADVICE GIVEN BY NVIDIA WILL IN ANY WAY INCREASE THE SCOPE OF ANY WARRANTY EXPRESSLY PROVIDED IN THIS AGREEMENT. NVIDIA does not warrant or assume responsibility for the accuracy or completeness of any third-party information, text, graphics or links contained in the Software. + +11. Limitations of Liability. + +11.1 DISCLAIMERS. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY (I) INDIRECT, PUNITIVE, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES, OR (II) DAMAGES FOR THE (A) COST OF PROCURING SUBSTITUTE GOODS OR (B) LOSS OF PROFITS, REVENUES, USE, DATA OR GOODWILL ARISING OUT OF OR RELATED TO THIS AGREEMENT, WHETHER BASED ON BREACH OF CONTRACT, TORT (INCLUDING NEGLIGENCE), STRICT LIABILITY, OR OTHERWISE, AND EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES AND EVEN IF A PARTY’S REMEDIES FAIL THEIR ESSENTIAL PURPOSE. + +11.2 DAMAGES CAP. ADDITIONALLY, TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, NVIDIA’S TOTAL CUMULATIVE AGGREGATE LIABILITY FOR ANY AND ALL LIABILITIES, OBLIGATIONS OR CLAIMS ARISING OUT OF OR RELATED TO THIS AGREEMENT WILL NOT EXCEED FIVE U.S. DOLLARS (US$5). + +12. Governing Law and Jurisdiction. This Agreement will be governed in all respects by the laws of the United States and the laws of the State of Delaware, without regard to conflict of laws principles or the United Nations Convention on Contracts for the International Sale of Goods. The state and federal courts residing in Santa Clara County, California will have exclusive jurisdiction over any dispute or claim arising out of or related to this Agreement, and the parties irrevocably consent to personal jurisdiction and venue in those courts; except that either party may apply for injunctive remedies or an equivalent type of urgent legal relief in any jurisdiction. + +13. General. + +13.1 No Assignment. NVIDIA may assign, delegate or transfer its rights or obligations under this Agreement by any means or operation of law. You may not, without NVIDIA’s prior written consent, assign, delegate or transfer any of your rights or obligations under this Agreement by any means or operation of law, and any attempt to do so is null and void. + +13.2 No Waiver. No waiver of any term of the Agreement will be deemed a further or continuing waiver of such term or any other term, and NVIDIA’s failure to assert any right or provision under the Agreement will not constitute a waiver of such right or provision. + +13.3 Trade and Compliance. You agree to comply with all applicable export, import, trade and economic sanctions laws and regulations, including U.S. Export Administration Regulations and Office of Foreign Assets Control regulations. You confirm that you will not export or reexport any products or technology, directly or indirectly, without first obtaining any required license or other approval from appropriate authorities, (i) to any countries that are subject to any U.S. or local export restrictions (currently including, but not necessarily limited to, Cuba, Iran, North Korea, Syria, the Region of Crimea, Donetsk People’s Republic Region and Luhansk People’s Republic Region); (ii) to any end user who you know or have reason to know will utilize them in the design, development or production of nuclear, chemical or biological weapons, missiles, rocket systems, unmanned air vehicles, or any weapons of mass destruction; (iii) to any end-user who has been prohibited from participating in the U.S. or local export transactions by any governing authority; or (iv) to any known military or military-intelligence end-user or for any known military or military-intelligence end-use in accordance with U.S. trade compliance laws and regulations. Use of the Software under this Agreement must be consistent with NVIDIA’s HumanRightsPolicy.pdf (nvidia.com). + +13.4 Government Rights. The Software, documentation and technology (“Protected Items”) are “Commercial products” as this term is defined at 48 C.F.R. 2.101, consisting of “commercial computer software” and “commercial computer software documentation” as such terms are used in, respectively, 48 C.F.R. 12.212 and 48 C.F.R. 227.7202 & 252.227-7014(a)(1). Before any Protected Items are supplied to the U.S. Government, you will (i) inform the U.S. Government in writing that the Protected Items are and must be treated as commercial computer software and commercial computer software documentation developed at private expense; (ii) inform the U.S. Government that the Protected Items are provided subject to the terms of the Agreement; and (iii) mark the Protected Items as commercial computer software and commercial computer software documentation developed at private expense. In no event will you permit the U.S. Government to acquire rights in Protected Items beyond those specified in 48 C.F.R. 52.227-19(b)(1)-(2) or 252.227-7013(c) except as expressly approved by NVIDIA in writing. + +13.5 Notices. Please direct your legal notices or other correspondence to NVIDIA Corporation, 2788 San Tomas Expressway, Santa Clara, California 95051, United States of America, Attention: Legal Department, with a copy emailed to legalnotices@nvidia.com. If NVIDIA needs to contact you about the Software, you consent to receive the notices by email and agree that such notices will satisfy any legal communication requirements. + +13.6 Force Majeure. Neither party will be liable during any period where an event or circumstance prevents or delays that party from performing its obligations under this Agreement and that event or circumstance: (i) is not within the reasonable control of that party and is not the result of that party’s negligence, and (ii) cannot be overcome or avoided by that party using reasonably diligent efforts. + +13.7 Severability and Amendment. If a court of competent jurisdiction rules that a provision of this Agreement is unenforceable, that provision will be deemed modified to the extent necessary to make it enforceable and the remainder of this Agreement will continue in full force and effect. Any amendment to this Agreement must be in writing and signed by authorized representatives of both parties. + +13.8 Construction. The headings in the Agreement are included solely for convenience and are not intended to affect the meaning or interpretation of the Agreement. As required by the context of the Agreement, the singular of a term includes the plural and vice versa. + +13.9 Entire Agreement. Regarding the subject matter of this Agreement, the parties agree that (i) this Agreement constitutes the entire and exclusive agreement between the parties and supersedes all prior and contemporaneous communications and (ii) any additional or different terms or conditions, whether contained in purchase orders, order acknowledgments, invoices or otherwise, will not be binding and are null and void. + + + +(v. November 28, 2023) + + + + + +nvidia-nvjitlink +Other/Proprietary License +https://developer.nvidia.com/cuda-zone +UNKNOWN + +nvidia-nvjpeg-cu12 +Other/Proprietary License +https://developer.nvidia.com/cuda-zone +UNKNOWN + +nvidia-nvjpeg2k-cu12 +Other/Proprietary License +https://developer.nvidia.com/nvjpeg +LICENSE AGREEMENT FOR NVIDIA SOFTWARE DEVELOPMENT KITS + +This license agreement, including exhibits attached ("Agreement”) is a legal agreement between you and NVIDIA Corporation ("NVIDIA") and governs your use of a NVIDIA software development kit (“SDK”). + +Each SDK has its own set of software and materials, but here is a description of the types of items that may be included in a SDK: source code, header files, APIs, data sets and assets (examples include images, textures, models, scenes, videos, native API input/output files), binary software, sample code, libraries, utility programs, programming code and documentation. + +This Agreement can be accepted only by an adult of legal age of majority in the country in which the SDK is used. + +If you are entering into this Agreement on behalf of a company or other legal entity, you represent that you have the legal authority to bind the entity to this Agreement, in which case “you” will mean the entity you represent. + +If you don’t have the required age or authority to accept this Agreement, or if you don’t accept all the terms and conditions of this Agreement, do not download, install or use the SDK. + +You agree to use the SDK only for purposes that are permitted by (a) this Agreement, and (b) any applicable law, regulation or generally accepted practices or guidelines in the relevant jurisdictions. + +1. License. + +1.1 Grant + +Subject to the terms of this Agreement, NVIDIA hereby grants you a non-exclusive, non-transferable license, without the right to sublicense (except as expressly provided in this Agreement) to: + +(i) Install and use the SDK, + +(ii) Modify and create derivative works of sample source code delivered in the SDK, and + +(iii) Distribute those portions of the SDK that are identified in this Agreement as distributable, as incorporated in object code format into a software application that meets the distribution requirements indicated in this Agreement. + +1.2 Distribution Requirements + +These are the distribution requirements for you to exercise the distribution grant: + +(i) Your application must have material additional functionality, beyond the included portions of the SDK. + +(ii) The distributable portions of the SDK shall only be accessed by your application. + +(iii) The following notice shall be included in modifications and derivative works of sample source code distributed: “This software contains source code provided by NVIDIA Corporation.” + +(iv) Unless a developer tool is identified in this Agreement as distributable, it is delivered for your internal use only. + +(v) The terms under which you distribute your application must be consistent with the terms of this Agreement, including (without limitation) terms relating to the license grant and license restrictions and protection of NVIDIA’s intellectual property rights. Additionally, you agree that you will protect the privacy, security and legal rights of your application users. + +(vi) You agree to notify NVIDIA in writing of any known or suspected distribution or use of the SDK not in compliance with the requirements of this Agreement, and to enforce the terms of your agreements with respect to distributed SDK. + +1.3 Authorized Users + +You may allow employees and contractors of your entity or of your subsidiary(ies) to access and use the SDK from your secure network to perform work on your behalf. + +If you are an academic institution you may allow users enrolled or employed by the academic institution to access and use the SDK from your secure network. + +You are responsible for the compliance with the terms of this Agreement by your authorized users. If you become aware that your authorized users didn’t follow the terms of this Agreement, you agree to take reasonable steps to resolve the non-compliance and prevent new occurrences. + +1.4 Pre-Release SDK +The SDK versions identified as alpha, beta, preview or otherwise as pre-release, may not be fully functional, may contain errors or design flaws, and may have reduced or different security, privacy, accessibility, availability, and reliability standards relative to commercial versions of NVIDIA software and materials. Use of a pre-release SDK may result in unexpected results, loss of data, project delays or other unpredictable damage or loss. +You may use a pre-release SDK at your own risk, understanding that pre-release SDKs are not intended for use in production or business-critical systems. +NVIDIA may choose not to make available a commercial version of any pre-release SDK. NVIDIA may also choose to abandon development and terminate the availability of a pre-release SDK at any time without liability. +1.5 Updates + +NVIDIA may, at its option, make available patches, workarounds or other updates to this SDK. Unless the updates are provided with their separate governing terms, they are deemed part of the SDK licensed to you as provided in this Agreement. + +You agree that the form and content of the SDK that NVIDIA provides may change without prior notice to you. While NVIDIA generally maintains compatibility between versions, NVIDIA may in some cases make changes that introduce incompatibilities in future versions of the SDK. + +1.6 Third Party Licenses + +The SDK may come bundled with, or otherwise include or be distributed with, third party software licensed by a NVIDIA supplier and/or open source software provided under an open source license. Use of third party software is subject to the third-party license terms, or in the absence of third party terms, the terms of this Agreement. Copyright to third party software is held by the copyright holders indicated in the third-party software or license. + +1.7 Reservation of Rights + +NVIDIA reserves all rights, title and interest in and to the SDK not expressly granted to you under this Agreement. + +2. Limitations. + +The following license limitations apply to your use of the SDK: + +2.1 You may not reverse engineer, decompile or disassemble, or remove copyright or other proprietary notices from any portion of the SDK or copies of the SDK. + +2.2 Except as expressly provided in this Agreement, you may not copy, sell, rent, sublicense, transfer, distribute, modify, or create derivative works of any portion of the SDK. + +2.3 Unless you have an agreement with NVIDIA for this purpose, you may not indicate that an application created with the SDK is sponsored or endorsed by NVIDIA. + +2.4 You may not bypass, disable, or circumvent any encryption, security, digital rights management or authentication mechanism in the SDK. + +2.5 You may not use the SDK in any manner that would cause it to become subject to an open source software license. As examples, licenses that require as a condition of use, modification, and/or distribution that the SDK be (i) disclosed or distributed in source code form; (ii) licensed for the purpose of making derivative works; or (iii) redistributable at no charge. + +2.6 Unless you have an agreement with NVIDIA for this purpose, you may not use the SDK with any system or application where the use or failure of the system or application can reasonably be expected to threaten or result in personal injury, death, or catastrophic loss. Examples include use in avionics, navigation, military, medical, life support or other life critical applications. NVIDIA does not design, test or manufacture the SDK for these critical uses and NVIDIA shall not be liable to you or any third party, in whole or in part, for any claims or damages arising from such uses. + +2.7 You agree to defend, indemnify and hold harmless NVIDIA and its affiliates, and their respective employees, contractors, agents, officers and directors, from and against any and all claims, damages, obligations, losses, liabilities, costs or debt, fines, restitutions and expenses (including but not limited to attorney’s fees and costs incident to establishing the right of indemnification) arising out of or related to your use of the SDK outside of the scope of this Agreement, or not in compliance with its terms. + +3. Ownership. + +3.1 NVIDIA or its licensors hold all rights, title and interest in and to the SDK and its modifications and derivative works, including their respective intellectual property rights, subject to your rights under Section 3.2. This SDK may include software and materials from NVIDIA’s licensors, and these licensors are intended third party beneficiaries that may enforce this Agreement with respect to their intellectual property rights. + +3.2 You hold all rights, title and interest in and to your applications and your derivative works of the sample source code delivered in the SDK, including their respective intellectual property rights, subject to NVIDIA’s rights under section 3.1. + +3.3 You may, but don’t have to, provide to NVIDIA suggestions, feature requests or other feedback regarding the SDK, including possible enhancements or modifications to the SDK. For any feedback that you voluntarily provide, you hereby grant NVIDIA and its affiliates a perpetual, non-exclusive, worldwide, irrevocable license to use, reproduce, modify, license, sublicense (through multiple tiers of sublicensees), and distribute (through multiple tiers of distributors) it without the payment of any royalties or fees to you. NVIDIA will use feedback at its choice. NVIDIA is constantly looking for ways to improve its products, so you may send feedback to NVIDIA through the developer portal at https://developer.nvidia.com. + +4. No Warranties. + +THE SDK IS PROVIDED BY NVIDIA “AS IS” AND “WITH ALL FAULTS.” TO THE MAXIMUM EXTENT PERMITTED BY LAW, NVIDIA AND ITS AFFILIATES EXPRESSLY DISCLAIM ALL WARRANTIES OF ANY KIND OR NATURE, WHETHER EXPRESS, IMPLIED OR STATUTORY, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, NON-INFRINGEMENT, OR THE ABSENCE OF ANY DEFECTS THEREIN, WHETHER LATENT OR PATENT. NO WARRANTY IS MADE ON THE BASIS OF TRADE USAGE, COURSE OF DEALING OR COURSE OF TRADE. + +5. Limitations of Liability. + +TO THE MAXIMUM EXTENT PERMITTED BY LAW, NVIDIA AND ITS AFFILIATES SHALL NOT BE LIABLE FOR ANY SPECIAL, INCIDENTAL, PUNITIVE OR CONSEQUENTIAL DAMAGES, OR ANY LOST PROFITS, LOSS OF USE, LOSS OF DATA OR LOSS OF GOODWILL, OR THE COSTS OF PROCURING SUBSTITUTE PRODUCTS, ARISING OUT OF OR IN CONNECTION WITH THIS AGREEMENT OR THE USE OR PERFORMANCE OF THE SDK, WHETHER SUCH LIABILITY ARISES FROM ANY CLAIM BASED UPON BREACH OF CONTRACT, BREACH OF WARRANTY, TORT (INCLUDING NEGLIGENCE), PRODUCT LIABILITY OR ANY OTHER CAUSE OF ACTION OR THEORY OF LIABILITY. IN NO EVENT WILL NVIDIA’S AND ITS AFFILIATES TOTAL CUMULATIVE LIABILITY UNDER OR ARISING OUT OF THIS AGREEMENT EXCEED US$10.00. THE NATURE OF THE LIABILITY OR THE NUMBER OF CLAIMS OR SUITS SHALL NOT ENLARGE OR EXTEND THIS LIMIT. + +These exclusions and limitations of liability shall apply regardless if NVIDIA or its affiliates have been advised of the possibility of such damages, and regardless of whether a remedy fails its essential purpose. These exclusions and limitations of liability form an essential basis of the bargain between the parties, and, absent any of these exclusions or limitations of liability, the provisions of this Agreement, including, without limitation, the economic terms, would be substantially different. + +6. Termination. + +6.1 This Agreement will continue to apply until terminated by either you or NVIDIA as described below. + +6.2 If you want to terminate this Agreement, you may do so by stopping to use the SDK. + +6.3 NVIDIA may, at any time, terminate this Agreement if: (i) you fail to comply with any term of this Agreement and the non-compliance is not fixed within thirty (30) days following notice from NVIDIA (or immediately if you violate NVIDIA’s intellectual property rights); (ii) you commence or participate in any legal proceeding against NVIDIA with respect to the SDK; or (iii) NVIDIA decides to no longer provide the SDK in a country or, in NVIDIA’s sole discretion, the continued use of it is no longer commercially viable. + +6.4 Upon any termination of this Agreement, you agree to promptly discontinue use of the SDK and destroy all copies in your possession or control. Your prior distributions in accordance with this Agreement are not affected by the termination of this Agreement. Upon written request, you will certify in writing that you have complied with your commitments under this section. Upon any termination of this Agreement all provisions survive except for the licenses granted to you. + +7. General. + +If you wish to assign this Agreement or your rights and obligations, including by merger, consolidation, dissolution or operation of law, contact NVIDIA to ask for permission. Any attempted assignment not approved by NVIDIA in writing shall be void and of no effect. NVIDIA may assign, delegate or transfer this Agreement and its rights and obligations, and if to a non-affiliate you will be notified. + +You agree to cooperate with NVIDIA and provide reasonably requested information to verify your compliance with this Agreement. + +This Agreement will be governed in all respects by the laws of the United States and of the State of Delaware as those laws are applied to contracts entered into and performed entirely within Delaware by Delaware residents, without regard to the conflicts of laws principles. The United Nations Convention on Contracts for the International Sale of Goods is specifically disclaimed. You agree to all terms of this Agreement in the English language. + +The state or federal courts residing in Santa Clara County, California shall have exclusive jurisdiction over any dispute or claim arising out of this Agreement. Notwithstanding this, you agree that NVIDIA shall still be allowed to apply for injunctive remedies or an equivalent type of urgent legal relief in any jurisdiction. + +If any court of competent jurisdiction determines that any provision of this Agreement is illegal, invalid or unenforceable, such provision will be construed as limited to the extent necessary to be consistent with and fully enforceable under the law and the remaining provisions will remain in full force and effect. Unless otherwise specified, remedies are cumulative. + +Each party acknowledges and agrees that the other is an independent contractor in the performance of this Agreement. + +The SDK has been developed entirely at private expense and is “commercial items” consisting of “commercial computer software” and “commercial computer software documentation” provided with RESTRICTED RIGHTS. Use, duplication or disclosure by the U.S. Government or a U.S. Government subcontractor is subject to the restrictions in this Agreement pursuant to DFARS 227.7202-3(a) or as set forth in subparagraphs (b)(1) and (2) of the Commercial Computer Software - Restricted Rights clause at FAR 52.227-19, as applicable. Contractor/manufacturer is NVIDIA, 2788 San Tomas Expressway, Santa Clara, CA 95051. + +The SDK is subject to United States export laws and regulations. You agree that you will not ship, transfer or export the SDK into any country, or use the SDK in any manner, prohibited by the United States Bureau of Industry and Security or economic sanctions regulations administered by the U.S. Department of Treasury’s Office of Foreign Assets Control (OFAC), or any applicable export laws, restrictions or regulations. These laws include restrictions on destinations, end users and end use. By accepting this Agreement, you confirm that you are not a resident or citizen of any country currently embargoed by the U.S. and that you are not otherwise prohibited from receiving the SDK. + +Any notice delivered by NVIDIA to you under this Agreement will be delivered via mail, email or fax. You agree that any notices that NVIDIA sends you electronically will satisfy any legal communication requirements. Please direct your legal notices or other correspondence to NVIDIA Corporation, 2788 San Tomas Expressway, Santa Clara, California 95051, United States of America, Attention: Legal Department. + +This Agreement and any exhibits incorporated into this Agreement constitute the entire agreement of the parties with respect to the subject matter of this Agreement and supersede all prior negotiations or documentation exchanged between the parties relating to this SDK license. Any additional and/or conflicting terms on documents issued by you are null, void, and invalid. Any amendment or waiver under this Agreement shall be in writing and signed by representatives of both parties. + +(v. January 28, 2020) + + + + + + + + + + + + + + + +nvJPEG2K SUPPLEMENT TO SOFTWARE LICENSE AGREEMENT FOR NVIDIA SOFTWARE DEVELOPMENT KITS + +The terms in this supplement govern your use of the NVIDIA nvJPEG2K SDK under the terms of your license agreement (“Agreement”) as modified by this supplement. Capitalized terms used but not defined below have the meaning assigned to them in the Agreement. + +This supplement is an exhibit to the Agreement and is incorporated as an integral part of the Agreement. In the event of conflict between the terms in this supplement and the terms in the Agreement, the terms in this supplement govern. + +4.1 License Scope. The SDK is licensed for you to develop applications only for use in systems with NVIDIA GPUs. + +2. Distribution. The following portions of the SDK are distributable under the Agreement: the runtime files .so and .h, nvjpeg2k_0.dll, nvjpeg2k.lib and libnvjpeg2k_static.a. + +3. Licensing. If the distribution terms in this Agreement are not suitable for your organization, or for any questions regarding this Agreement, please contact NVIDIA at nvidia-compute-license-questions@nvidia.com. + (v. November 14, 2020) + + +OpenJPEG LICENSE + +/* + * The copyright in this software is being made available under the 2-clauses + * BSD License, included below. This software may be subject to other third + * party and contributor rights, including patent rights, and no such rights + * are granted under this license. + * + * Copyright (c) 2002-2014, Universite catholique de Louvain (UCL), Belgium + * Copyright (c) 2002-2014, Professor Benoit Macq + * Copyright (c) 2003-2014, Antonin Descampe + * Copyright (c) 2003-2009, Francois-Olivier Devaux + * Copyright (c) 2005, Herve Drolon, FreeImage Team + * Copyright (c) 2002-2003, Yannick Verschueren + * Copyright (c) 2001-2003, David Janssens + * Copyright (c) 2011-2012, Centre National d'Etudes Spatiales (CNES), France + * Copyright (c) 2012, CS Systemes d'Information, France + * + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS `AS IS' + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ + + + + + + +nvidia-nvshmem-cu13 +LicenseRef-NVIDIA-Proprietary +https://developer.nvidia.com/cuda-zone +UNKNOWN + +nvidia-nvtiff-cu12 +Other/Proprietary License +https://developer.nvidia.com/nvtiff +NVIDIA nvTIFF +Software License Agreement | NVIDIA Docs + +Table of Contents +LICENSE AGREEMENT FOR NVIDIA SOFTWARE DEVELOPMENT KITS iii +Chapter 1. License. 1 +1.1. Grant 1 +1.2. Distribution Requirements 1 +1.3. Authorized Users 2 +1.4. Pre-Release SDK 2 +1.5. Updates 2 +1.6. Third Party Licenses 2 +1.7. Reservation of Rights 3 +Chapter 2. Limitations. 4 +Chapter 3. Ownership. 5 +Chapter 4. No Warranties. 6 +Chapter 5. Limitations of Liability. 7 +Chapter 6. Termination. 8 +Chapter 7. General. 9 + + + +LICENSE AGREEMENT FOR NVIDIA SOFTWARE DEVELOPMENT KITS + + + +This license agreement, including exhibits attached ("Agreement") is a legal agreement between you and NVIDIA Corporation ("NVIDIA") and governs your use of a NVIDIA software development kit ("SDK"). +Each SDK has its own set of software and materials, but here is a description of the types of items that may be included in a SDK: source code, header files, APIs, data sets and assets (examples include images, textures, models, scenes, videos, native API input/output files), binary software, sample code, libraries, utility programs, programming code and documentation. +This Agreement can be accepted only by an adult of legal age of majority in the country in which the SDK is used. +If you are entering into this Agreement on behalf of a company or other legal entity, you represent that you have the legal authority to bind the entity to this Agreement, in which case "you" will mean the entity you represent. +If you don't have the required age or authority to accept this Agreement, or if you don't accept all the terms and conditions of this Agreement, do not download, install or use the SDK. +You agree to use the SDK only for purposes that are permitted by (a) this Agreement, and (b) any applicable law, regulation or generally accepted practices or guidelines in the relevant jurisdictions. + +Chapter 1. License. + +1.1. Grant +Subject to the terms of this Agreement, NVIDIA hereby grants you a non-exclusive, non- transferable license, without the right to sublicense (except as expressly provided in this Agreement) to: +1. Install and use the SDK, +2. Modify and create derivative works of sample source code delivered in the SDK, and +3. Distribute those portions of the SDK that are identified in this Agreement as distributable, as incorporated in object code format into a software application that meets the distribution requirements indicated in this Agreement. + +1.2. Distribution Requirements +These are the distribution requirements for you to exercise the distribution grant: +1. Your application must have material additional functionality, beyond the included portions of the SDK. +2. The distributable portions of the SDK shall only be accessed by your application. +3. The following notice shall be included in modifications and derivative works of sample source code distributed: "This software contains source code provided by NVIDIA Corporation." +4. Unless a developer tool is identified in this Agreement as distributable, it is delivered for your internal use only. +5. The terms under which you distribute your application must be consistent with the terms of this Agreement, including (without limitation) terms relating to the license grant and license restrictions and protection of NVIDIA's intellectual property rights. Additionally, you agree that you will protect the privacy, security and legal rights of your application users. +6. You agree to notify NVIDIA in writing of any known or suspected distribution or use of the SDK not in compliance with the requirements of this Agreement, and to enforce the terms of your agreements with respect to distributed SDK. + + +1.3. Authorized Users +You may allow employees and contractors of your entity or of your subsidiary(ies) to access and use the SDK from your secure network to perform work on your behalf. +If you are an academic institution you may allow users enrolled or employed by the academic institution to access and use the SDK from your secure network. +You are responsible for the compliance with the terms of this Agreement by your authorized users. If you become aware that your authorized users didn't follow the terms of this Agreement, you agree to take reasonable steps to resolve the non-compliance and prevent new occurrences. + +1.4. Pre-Release SDK +The SDK versions identified as alpha, beta, preview or otherwise as pre-release, may not be fully functional, may contain errors or design flaws, and may have reduced or different security, privacy, accessibility, availability, and reliability standards relative to commercial +versions of NVIDIA software and materials. Use of a pre-release SDK may result in unexpected results, loss of data, project delays or other unpredictable damage or loss. +You may use a pre-release SDK at your own risk, understanding that pre-release SDKs are not intended for use in production or business-critical systems. +NVIDIA may choose not to make available a commercial version of any pre-release SDK. NVIDIA may also choose to abandon development and terminate the availability of a pre- release SDK at any time without liability. + +1.5. Updates +NVIDIA may, at its option, make available patches, workarounds or other updates to this SDK. Unless the updates are provided with their separate governing terms, they are deemed part of the SDK licensed to you as provided in this Agreement. +You agree that the form and content of the SDK that NVIDIA provides may change without prior notice to you. While NVIDIA generally maintains compatibility between versions, NVIDIA may in some cases make changes that introduce incompatibilities in future versions of the SDK. + +1.6. Components Under Other Licenses +The SDK may come bundled with, or otherwise include or be distributed with, NVIDIA or third party software licensed with separate legal notices or terms as may be described in proprietary notices accompanying the SDK. If and to the extent there is a conflict between the terms in this Agreement and the license terms associated with the component, the license terms associated with the components control only to the extent necessary to resolve the conflict. + +1.7. Reservation of Rights +NVIDIA reserves all rights, title and interest in and to the SDK not expressly granted to you under this Agreement. + + +Chapter 2. Limitations. + +The following license limitations apply to your use of the SDK: +2.1 You may not reverse engineer, decompile or disassemble, or remove copyright or other proprietary notices from any portion of the SDK or copies of the SDK. +2.2 Except as expressly provided in this Agreement, you may not copy, sell, rent, sublicense, transfer, distribute, modify, or create derivative works of any portion of the SDK. +2.3 Unless you have an agreement with NVIDIA for this purpose, you may not indicate that an application created with the SDK is sponsored or endorsed by NVIDIA. +2.4 You may not bypass, disable, or circumvent any encryption, security, digital rights management or authentication mechanism in the SDK. +2.5 You may not use the SDK in any manner that would cause it to become subject to an open source software license. As examples, licenses that require as a condition of use, modification, and/or distribution that the SDK be (i) disclosed or distributed in source code form; (ii) licensed for the purpose of making derivative works; or (iii) redistributable at no charge. +2.6 You acknowledge that the SDK as delivered is not tested or certified by NVIDIA for use in connection with the design, construction, maintenance, and/or operation of any system where the use or failure of such system could result in a situation that threatens the safety of human life or results in catastrophic damages (each, a "Critical Application"). Examples of Critical Applications include use in avionics, navigation, autonomous vehicle applications, ai solutions for automotive products, military, medical, life support or other life critical applications. NVIDIA shall not be liable to you or any third party, in whole or in part, for any claims or damages arising from such uses. You are solely responsible for ensuring that any product +or service developed with the SDK as a whole includes sufficient features to comply with all applicable legal and regulatory standards and requirements. +2.7 You agree to defend, indemnify and hold harmless NVIDIA and its affiliates, and their respective employees, contractors, agents, officers and directors, from and against any and all claims, damages, obligations, losses, liabilities, costs or debt, fines, restitutions and expenses (including but not limited to attorney's fees and costs incident to establishing the right of indemnification) arising out of or related to products or services that use the SDK in or for Critical Applications, and for use of the SDK outside of the scope of this Agreement or not in compliance with its terms. + + +Chapter 3. Ownership. + +3.1 NVIDIA or its licensors hold all rights, title and interest in and to the SDK and its modifications and derivative works, including their respective intellectual property rights, subject to your rights under Section 3.2. This SDK may include software and materials from NVIDIA's licensors, and these licensors are intended third party beneficiaries that may enforce this Agreement with respect to their intellectual property rights. +3.2 You hold all rights, title and interest in and to your applications and your derivative works of the sample source code delivered in the SDK, including their respective intellectual property rights, subject to NVIDIA's rights under section 3.1. +3.3 You may, but don't have to, provide to NVIDIA suggestions, feature requests or other feedback regarding the SDK, including possible enhancements or modifications to the SDK. For any feedback that you voluntarily provide, you hereby grant NVIDIA and its affiliates a perpetual, non-exclusive, worldwide, irrevocable license to use, reproduce, modify, license, sublicense (through multiple tiers of sublicensees), and distribute (through multiple tiers of distributors) it without the payment of any royalties or fees to you. NVIDIA will use feedback at its choice. NVIDIA is constantly looking for ways to improve its products, so you may send feedback to NVIDIA through the developer portal at https://developer.nvidia.com. + + +Chapter 4. No Warranties. + + +THE SDK IS PROVIDED BY NVIDIA "AS IS" AND "WITH ALL FAULTS." TO THE MAXIMUM EXTENT PERMITTED BY LAW, NVIDIA AND ITS AFFILIATES EXPRESSLY DISCLAIM ALL WARRANTIES OF ANY KIND OR NATURE, WHETHER EXPRESS, IMPLIED OR STATUTORY, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, NON-INFRINGEMENT, OR THE ABSENCE OF ANY DEFECTS THEREIN, WHETHER LATENT OR PATENT. NO WARRANTY IS MADE ON THE BASIS OF TRADE USAGE, COURSE OF DEALING OR COURSE OF TRADE. + + +Chapter 5. Limitations of Liability. + + +TO THE MAXIMUM EXTENT PERMITTED BY LAW, NVIDIA AND ITS AFFILIATES SHALL NOT BE LIABLE FOR ANY SPECIAL, INCIDENTAL, PUNITIVE OR CONSEQUENTIAL DAMAGES, OR ANY LOST PROFITS, LOSS OF USE, LOSS OF DATA OR LOSS OF GOODWILL, OR THE COSTS OF PROCURING SUBSTITUTE PRODUCTS, ARISING OUT OF OR IN CONNECTION WITH THIS AGREEMENT OR THE USE OR PERFORMANCE OF THE SDK, WHETHER SUCH LIABILITY ARISES FROM ANY CLAIM BASED UPON BREACH OF CONTRACT, BREACH OF WARRANTY, +TORT (INCLUDING NEGLIGENCE), PRODUCT LIABILITY OR ANY OTHER CAUSE OF ACTION OR THEORY OF LIABILITY. IN NO EVENT WILL NVIDIA'S AND ITS AFFILIATES TOTAL CUMULATIVE LIABILITY UNDER OR ARISING OUT OF THIS AGREEMENT EXCEED US$10.00. THE NATURE OF THE LIABILITY OR THE NUMBER OF CLAIMS OR SUITS SHALL NOT ENLARGE OR EXTEND THIS LIMIT. +These exclusions and limitations of liability shall apply regardless if NVIDIA or its affiliates have been advised of the possibility of such damages, and regardless of whether a remedy fails its essential purpose. These exclusions and limitations of liability form an essential basis of the bargain between the parties, and, absent any of these exclusions or limitations of liability, the provisions of this Agreement, including, without limitation, the economic terms, would be substantially different. + + +Chapter 6. Termination. + + + +6.1 This Agreement will continue to apply until terminated by either you or NVIDIA as described below. +6.2 If you want to terminate this Agreement, you may do so by stopping to use the SDK. +6.3 NVIDIA may, at any time, terminate this Agreement if: (i) you fail to comply with any term of this Agreement and the non-compliance is not fixed within thirty (30) days following notice from NVIDIA (or immediately if you violate NVIDIA's intellectual property rights); (ii) you commence or participate in any legal proceeding against NVIDIA with respect to the SDK; or +(iii) NVIDIA decides to no longer provide the SDK in a country or, in NVIDIA's sole discretion, the continued use of it is no longer commercially viable. +6.4 Upon any termination of this Agreement, you agree to promptly discontinue use of the SDK and destroy all copies in your possession or control. Your prior distributions in accordance with this Agreement are not affected by the termination of this Agreement. Upon written request, you will certify in writing that you have complied with your commitments under this section. Upon any termination of this Agreement all provisions survive except for the licenses granted to you. + + +Chapter 7. General. + + +If you wish to assign this Agreement or your rights and obligations, including by merger, consolidation, dissolution or operation of law, contact NVIDIA to ask for permission. Any attempted assignment not approved by NVIDIA in writing shall be void and of no effect. NVIDIA may assign, delegate or transfer this Agreement and its rights and obligations, and if to a non- affiliate you will be notified. +You agree to cooperate with NVIDIA and provide reasonably requested information to verify your compliance with this Agreement. +This Agreement will be governed in all respects by the laws of the United States and of the State of Delaware as those laws are applied to contracts entered into and performed entirely within Delaware by Delaware residents, without regard to the conflicts of laws principles. +The United Nations Convention on Contracts for the International Sale of Goods is specifically disclaimed. You agree to all terms of this Agreement in the English language. +The state or federal courts residing in Santa Clara County, California shall have exclusive jurisdiction over any dispute or claim arising out of this Agreement. Notwithstanding this, you agree that NVIDIA shall still be allowed to apply for injunctive remedies or an equivalent type of urgent legal relief in any jurisdiction. +If any court of competent jurisdiction determines that any provision of this Agreement is illegal, invalid or unenforceable, such provision will be construed as limited to the extent necessary to be consistent with and fully enforceable under the law and the remaining provisions will remain in full force and effect. Unless otherwise specified, remedies are cumulative. +Each party acknowledges and agrees that the other is an independent contractor in the performance of this Agreement. +The SDK has been developed entirely at private expense and is "commercial items" consisting of "commercial computer software" and "commercial computer software documentation" provided with RESTRICTED RIGHTS. Use, duplication or disclosure by the U.S. Government +or a U.S. Government subcontractor is subject to the restrictions in this Agreement pursuant to DFARS 227.7202-3(a) or as set forth in subparagraphs (b)(1) and (2) of the Commercial Computer Software - Restricted Rights clause at FAR 52.227-19, as applicable. Contractor/ manufacturer is NVIDIA, 2788 San Tomas Expressway, Santa Clara, CA 95051. +The SDK is subject to United States export laws and regulations. You agree that you will not ship, transfer or export the SDK into any country, or use the SDK in any manner, prohibited by the United States Bureau of Industry and Security or economic sanctions regulations administered by the U.S. Department of Treasury's Office of Foreign Assets Control (OFAC), + + + + +or any applicable export laws, restrictions or regulations. These laws include restrictions on destinations, end users and end use. By accepting this Agreement, you confirm that you are not a resident or citizen of any country currently embargoed by the U.S. and that you are not otherwise prohibited from receiving the SDK. +Any notice delivered by NVIDIA to you under this Agreement will be delivered via mail, email or fax. You agree that any notices that NVIDIA sends you electronically will satisfy any legal communication requirements. Please direct your legal notices or other correspondence to +NVIDIA Corporation, 2788 San Tomas Expressway, Santa Clara, California 95051, United States of America, Attention: Legal Department. +This Agreement and any exhibits incorporated into this Agreement constitute the entire agreement of the parties with respect to the subject matter of this Agreement and supersede all prior negotiations or documentation exchanged between the parties relating to this SDK license. Any additional and/or conflicting terms on documents issued by you are null, void, and invalid. Any amendment or waiver under this Agreement shall be in writing and signed by representatives of both parties. +(v. March 31, 2022) + + +Chapter 8. nvTIFF SUPPLEMENT +TO SOFTWARE LICENSE AGREEMENT FOR NVIDIA SOFTWARE DEVELOPMENT KITS + + +The terms in this supplement govern your use of the NVIDIA nvTIFF SDK under the terms of your license agreement ("Agreement") as modified by this supplement. Capitalized terms used but not defined below have the meaning assigned to them in the Agreement. +This supplement is an exhibit to the Agreement and is incorporated as an integral part of the Agreement. In the event of conflict between the terms in this supplement and the terms in the Agreement, the terms in this supplement govern. +1. License Scope. The SDK is licensed for you to develop applications only for use in systems with NVIDIA GPUs. +2. Distribution. The following portions of the SDK are distributable under the Agreement: the runtime files .so and .h, nvTIFF_0.dll, nvTIFF.lib and libnvTIFF_static.a. +3. Licensing. If the distribution terms in this Agreement are not suitable for your organization, or for any questions regarding this Agreement, please contact NVIDIA at nvidia-compute- license-questions@nvidia.com +(v. March 31, 2022) + + + + + + + + + + + +nvidia-nvtx +Other/Proprietary License +https://developer.nvidia.com/cuda-zone +UNKNOWN + +nvtx +Apache Software License +https://github.com/NVIDIA/NVTX +============================================================================== +NVTX is under the Apache License v2.0 with LLVM Exceptions: +============================================================================== + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +---- LLVM Exceptions to the Apache 2.0 License ---- + +As an exception, if, as a result of your compiling your source code, portions +of this Software are embedded into an Object form of such source code, you +may redistribute such embedded portions in such Object form without complying +with the conditions of Sections 4(a), 4(b) and 4(d) of the License. + +In addition, if you combine or link compiled forms of this Software with +software that is licensed under the GPLv2 ("Combined Software") and if a +court of competent jurisdiction determines that the patent provision (Section +3), the indemnity provision (Section 9) or other Section of the License +conflicts with the conditions of the GPLv2, you may retroactively and +prospectively choose to deem waived or otherwise exclude such Section(s) of +the License, but only in their entirety and only with respect to the Combined +Software. + + + +obstore +MIT License +https://developmentseed.org/obstore +UNKNOWN + +omegaconf +BSD License +https://github.com/omry/omegaconf +BSD 3-Clause License + +Copyright (c) 2018, Omry Yadan +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +onnx +Apache-2.0 +https://onnx.ai/ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +onnx-ir +Apache-2.0 +https://onnx.ai/ir-py + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +onnxscript +MIT License +https://microsoft.github.io/onnxscript/ +MIT License + +Copyright (c) Microsoft Corporation + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +open_clip_torch +MIT License +https://github.com/mlfoundations/open_clip +Copyright (c) 2012-2021 Gabriel Ilharco, Mitchell Wortsman, +Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, +John Miller, Hongseok Namkoong, Hannaneh Hajishirzi, Ali Farhadi, +Ludwig Schmidt + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +openai +Apache Software License +https://github.com/openai/openai-python + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2026 OpenAI + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +opencensus +Apache Software License +https://github.com/census-instrumentation/opencensus-python + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +opencensus-context +Apache Software License +https://github.com/census-instrumentation/opencensus-python/tree/master/context/opencensus-context +UNKNOWN + +opencv-contrib-python +Apache Software License +https://github.com/opencv/opencv-python +OpenCV library is redistributed within opencv-python package. +This license applies to OpenCV binary in the directory cv2/. + + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +------------------------------------------------------------------------------ +libvpx is redistributed within all opencv-python Linux packages. +This license applies to libvpx binary in the directory cv2/. + +Copyright (c) 2010, The WebM Project authors. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + + * Neither the name of Google, nor the WebM Project, nor the names + of its contributors may be used to endorse or promote products + derived from this software without specific prior written + permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +FFmpeg is redistributed within all opencv-python packages. + +Libbluray, libgnutls, libnettle, libhogweed, libintl, libmp3lame, libp11, +librtmp, libsoxr and libtasn1 are redistributed within all opencv-python macOS packages. + +This license applies to the above library binaries in the directory cv2/. + + GNU LESSER GENERAL PUBLIC LICENSE + Version 2.1, February 1999 + + Copyright (C) 1991, 1999 Free Software Foundation, Inc. + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + +[This is the first released version of the Lesser GPL. It also counts + as the successor of the GNU Library Public License, version 2, hence + the version number 2.1.] + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +Licenses are intended to guarantee your freedom to share and change +free software--to make sure the software is free for all its users. + + This license, the Lesser General Public License, applies to some +specially designated software packages--typically libraries--of the +Free Software Foundation and other authors who decide to use it. You +can use it too, but we suggest you first think carefully about whether +this license or the ordinary General Public License is the better +strategy to use in any particular case, based on the explanations below. + + When we speak of free software, we are referring to freedom of use, +not price. Our General Public Licenses are designed to make sure that +you have the freedom to distribute copies of free software (and charge +for this service if you wish); that you receive source code or can get +it if you want it; that you can change the software and use pieces of +it in new free programs; and that you are informed that you can do +these things. + + To protect your rights, we need to make restrictions that forbid +distributors to deny you these rights or to ask you to surrender these +rights. These restrictions translate to certain responsibilities for +you if you distribute copies of the library or if you modify it. + + For example, if you distribute copies of the library, whether gratis +or for a fee, you must give the recipients all the rights that we gave +you. You must make sure that they, too, receive or can get the source +code. If you link other code with the library, you must provide +complete object files to the recipients, so that they can relink them +with the library after making changes to the library and recompiling +it. And you must show them these terms so they know their rights. + + We protect your rights with a two-step method: (1) we copyright the +library, and (2) we offer you this license, which gives you legal +permission to copy, distribute and/or modify the library. + + To protect each distributor, we want to make it very clear that +there is no warranty for the free library. Also, if the library is +modified by someone else and passed on, the recipients should know +that what they have is not the original version, so that the original +author's reputation will not be affected by problems that might be +introduced by others. + + Finally, software patents pose a constant threat to the existence of +any free program. We wish to make sure that a company cannot +effectively restrict the users of a free program by obtaining a +restrictive license from a patent holder. Therefore, we insist that +any patent license obtained for a version of the library must be +consistent with the full freedom of use specified in this license. + + Most GNU software, including some libraries, is covered by the +ordinary GNU General Public License. This license, the GNU Lesser +General Public License, applies to certain designated libraries, and +is quite different from the ordinary General Public License. We use +this license for certain libraries in order to permit linking those +libraries into non-free programs. + + When a program is linked with a library, whether statically or using +a shared library, the combination of the two is legally speaking a +combined work, a derivative of the original library. The ordinary +General Public License therefore permits such linking only if the +entire combination fits its criteria of freedom. The Lesser General +Public License permits more lax criteria for linking other code with +the library. + + We call this license the "Lesser" General Public License because it +does Less to protect the user's freedom than the ordinary General +Public License. It also provides other free software developers Less +of an advantage over competing non-free programs. These disadvantages +are the reason we use the ordinary General Public License for many +libraries. However, the Lesser license provides advantages in certain +special circumstances. + + For example, on rare occasions, there may be a special need to +encourage the widest possible use of a certain library, so that it becomes +a de-facto standard. To achieve this, non-free programs must be +allowed to use the library. A more frequent case is that a free +library does the same job as widely used non-free libraries. In this +case, there is little to gain by limiting the free library to free +software only, so we use the Lesser General Public License. + + In other cases, permission to use a particular library in non-free +programs enables a greater number of people to use a large body of +free software. For example, permission to use the GNU C Library in +non-free programs enables many more people to use the whole GNU +operating system, as well as its variant, the GNU/Linux operating +system. + + Although the Lesser General Public License is Less protective of the +users' freedom, it does ensure that the user of a program that is +linked with the Library has the freedom and the wherewithal to run +that program using a modified version of the Library. + + The precise terms and conditions for copying, distribution and +modification follow. Pay close attention to the difference between a +"work based on the library" and a "work that uses the library". The +former contains code derived from the library, whereas the latter must +be combined with the library in order to run. + + GNU LESSER GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License Agreement applies to any software library or other +program which contains a notice placed by the copyright holder or +other authorized party saying it may be distributed under the terms of +this Lesser General Public License (also called "this License"). +Each licensee is addressed as "you". + + A "library" means a collection of software functions and/or data +prepared so as to be conveniently linked with application programs +(which use some of those functions and data) to form executables. + + The "Library", below, refers to any such software library or work +which has been distributed under these terms. A "work based on the +Library" means either the Library or any derivative work under +copyright law: that is to say, a work containing the Library or a +portion of it, either verbatim or with modifications and/or translated +straightforwardly into another language. (Hereinafter, translation is +included without limitation in the term "modification".) + + "Source code" for a work means the preferred form of the work for +making modifications to it. For a library, complete source code means +all the source code for all modules it contains, plus any associated +interface definition files, plus the scripts used to control compilation +and installation of the library. + + Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running a program using the Library is not restricted, and output from +such a program is covered only if its contents constitute a work based +on the Library (independent of the use of the Library in a tool for +writing it). Whether that is true depends on what the Library does +and what the program that uses the Library does. + + 1. You may copy and distribute verbatim copies of the Library's +complete source code as you receive it, in any medium, provided that +you conspicuously and appropriately publish on each copy an +appropriate copyright notice and disclaimer of warranty; keep intact +all the notices that refer to this License and to the absence of any +warranty; and distribute a copy of this License along with the +Library. + + You may charge a fee for the physical act of transferring a copy, +and you may at your option offer warranty protection in exchange for a +fee. + + 2. You may modify your copy or copies of the Library or any portion +of it, thus forming a work based on the Library, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) The modified work must itself be a software library. + + b) You must cause the files modified to carry prominent notices + stating that you changed the files and the date of any change. + + c) You must cause the whole of the work to be licensed at no + charge to all third parties under the terms of this License. + + d) If a facility in the modified Library refers to a function or a + table of data to be supplied by an application program that uses + the facility, other than as an argument passed when the facility + is invoked, then you must make a good faith effort to ensure that, + in the event an application does not supply such function or + table, the facility still operates, and performs whatever part of + its purpose remains meaningful. + + (For example, a function in a library to compute square roots has + a purpose that is entirely well-defined independent of the + application. Therefore, Subsection 2d requires that any + application-supplied function or table used by this function must + be optional: if the application does not supply it, the square + root function must still compute square roots.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Library, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Library, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote +it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Library. + +In addition, mere aggregation of another work not based on the Library +with the Library (or with a work based on the Library) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may opt to apply the terms of the ordinary GNU General Public +License instead of this License to a given copy of the Library. To do +this, you must alter all the notices that refer to this License, so +that they refer to the ordinary GNU General Public License, version 2, +instead of to this License. (If a newer version than version 2 of the +ordinary GNU General Public License has appeared, then you can specify +that version instead if you wish.) Do not make any other change in +these notices. + + Once this change is made in a given copy, it is irreversible for +that copy, so the ordinary GNU General Public License applies to all +subsequent copies and derivative works made from that copy. + + This option is useful when you wish to copy part of the code of +the Library into a program that is not a library. + + 4. You may copy and distribute the Library (or a portion or +derivative of it, under Section 2) in object code or executable form +under the terms of Sections 1 and 2 above provided that you accompany +it with the complete corresponding machine-readable source code, which +must be distributed under the terms of Sections 1 and 2 above on a +medium customarily used for software interchange. + + If distribution of object code is made by offering access to copy +from a designated place, then offering equivalent access to copy the +source code from the same place satisfies the requirement to +distribute the source code, even though third parties are not +compelled to copy the source along with the object code. + + 5. A program that contains no derivative of any portion of the +Library, but is designed to work with the Library by being compiled or +linked with it, is called a "work that uses the Library". Such a +work, in isolation, is not a derivative work of the Library, and +therefore falls outside the scope of this License. + + However, linking a "work that uses the Library" with the Library +creates an executable that is a derivative of the Library (because it +contains portions of the Library), rather than a "work that uses the +library". The executable is therefore covered by this License. +Section 6 states terms for distribution of such executables. + + When a "work that uses the Library" uses material from a header file +that is part of the Library, the object code for the work may be a +derivative work of the Library even though the source code is not. +Whether this is true is especially significant if the work can be +linked without the Library, or if the work is itself a library. The +threshold for this to be true is not precisely defined by law. + + If such an object file uses only numerical parameters, data +structure layouts and accessors, and small macros and small inline +functions (ten lines or less in length), then the use of the object +file is unrestricted, regardless of whether it is legally a derivative +work. (Executables containing this object code plus portions of the +Library will still fall under Section 6.) + + Otherwise, if the work is a derivative of the Library, you may +distribute the object code for the work under the terms of Section 6. +Any executables containing that work also fall under Section 6, +whether or not they are linked directly with the Library itself. + + 6. As an exception to the Sections above, you may also combine or +link a "work that uses the Library" with the Library to produce a +work containing portions of the Library, and distribute that work +under terms of your choice, provided that the terms permit +modification of the work for the customer's own use and reverse +engineering for debugging such modifications. + + You must give prominent notice with each copy of the work that the +Library is used in it and that the Library and its use are covered by +this License. You must supply a copy of this License. If the work +during execution displays copyright notices, you must include the +copyright notice for the Library among them, as well as a reference +directing the user to the copy of this License. Also, you must do one +of these things: + + a) Accompany the work with the complete corresponding + machine-readable source code for the Library including whatever + changes were used in the work (which must be distributed under + Sections 1 and 2 above); and, if the work is an executable linked + with the Library, with the complete machine-readable "work that + uses the Library", as object code and/or source code, so that the + user can modify the Library and then relink to produce a modified + executable containing the modified Library. (It is understood + that the user who changes the contents of definitions files in the + Library will not necessarily be able to recompile the application + to use the modified definitions.) + + b) Use a suitable shared library mechanism for linking with the + Library. A suitable mechanism is one that (1) uses at run time a + copy of the library already present on the user's computer system, + rather than copying library functions into the executable, and (2) + will operate properly with a modified version of the library, if + the user installs one, as long as the modified version is + interface-compatible with the version that the work was made with. + + c) Accompany the work with a written offer, valid for at + least three years, to give the same user the materials + specified in Subsection 6a, above, for a charge no more + than the cost of performing this distribution. + + d) If distribution of the work is made by offering access to copy + from a designated place, offer equivalent access to copy the above + specified materials from the same place. + + e) Verify that the user has already received a copy of these + materials or that you have already sent this user a copy. + + For an executable, the required form of the "work that uses the +Library" must include any data and utility programs needed for +reproducing the executable from it. However, as a special exception, +the materials to be distributed need not include anything that is +normally distributed (in either source or binary form) with the major +components (compiler, kernel, and so on) of the operating system on +which the executable runs, unless that component itself accompanies +the executable. + + It may happen that this requirement contradicts the license +restrictions of other proprietary libraries that do not normally +accompany the operating system. Such a contradiction means you cannot +use both them and the Library together in an executable that you +distribute. + + 7. You may place library facilities that are a work based on the +Library side-by-side in a single library together with other library +facilities not covered by this License, and distribute such a combined +library, provided that the separate distribution of the work based on +the Library and of the other library facilities is otherwise +permitted, and provided that you do these two things: + + a) Accompany the combined library with a copy of the same work + based on the Library, uncombined with any other library + facilities. This must be distributed under the terms of the + Sections above. + + b) Give prominent notice with the combined library of the fact + that part of it is a work based on the Library, and explaining + where to find the accompanying uncombined form of the same work. + + 8. You may not copy, modify, sublicense, link with, or distribute +the Library except as expressly provided under this License. Any +attempt otherwise to copy, modify, sublicense, link with, or +distribute the Library is void, and will automatically terminate your +rights under this License. However, parties who have received copies, +or rights, from you under this License will not have their licenses +terminated so long as such parties remain in full compliance. + + 9. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Library or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Library (or any work based on the +Library), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Library or works based on it. + + 10. Each time you redistribute the Library (or any work based on the +Library), the recipient automatically receives a license from the +original licensor to copy, distribute, link with or modify the Library +subject to these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties with +this License. + + 11. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Library at all. For example, if a patent +license would not permit royalty-free redistribution of the Library by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Library. + +If any portion of this section is held invalid or unenforceable under any +particular circumstance, the balance of the section is intended to apply, +and the section as a whole is intended to apply in other circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 12. If the distribution and/or use of the Library is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Library under this License may add +an explicit geographical distribution limitation excluding those countries, +so that distribution is permitted only in or among countries not thus +excluded. In such case, this License incorporates the limitation as if +written in the body of this License. + + 13. The Free Software Foundation may publish revised and/or new +versions of the Lesser General Public License from time to time. +Such new versions will be similar in spirit to the present version, +but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Library +specifies a version number of this License which applies to it and +"any later version", you have the option of following the terms and +conditions either of that version or of any later version published by +the Free Software Foundation. If the Library does not specify a +license version number, you may choose any version ever published by +the Free Software Foundation. + + 14. If you wish to incorporate parts of the Library into other free +programs whose distribution conditions are incompatible with these, +write to the author to ask for permission. For software which is +copyrighted by the Free Software Foundation, write to the Free +Software Foundation; we sometimes make exceptions for this. Our +decision will be guided by the two goals of preserving the free status +of all derivatives of our free software and of promoting the sharing +and reuse of software generally. + + NO WARRANTY + + 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO +WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. +EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR +OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY +KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE +LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME +THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN +WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY +AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU +FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR +CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE +LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING +RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A +FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF +SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH +DAMAGES. + + END OF TERMS AND CONDITIONS + +------------------------------------------------------------------------------ +Qt 5 is redistributed within non-headless opencv-python Linux and macOS packages. +libgmp is redistributed within opencv-python macOS packages. +libidn2 is redistributed within opencv-python macOS packages. +libunistring is redistributed within opencv-python macOS packages. +This license applies to the above binaries in the directory cv2/. + + GNU LESSER GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + This version of the GNU Lesser General Public License incorporates +the terms and conditions of version 3 of the GNU General Public +License, supplemented by the additional permissions listed below. + + 0. Additional Definitions. + + As used herein, "this License" refers to version 3 of the GNU Lesser +General Public License, and the "GNU GPL" refers to version 3 of the GNU +General Public License. + + "The Library" refers to a covered work governed by this License, +other than an Application or a Combined Work as defined below. + + An "Application" is any work that makes use of an interface provided +by the Library, but which is not otherwise based on the Library. +Defining a subclass of a class defined by the Library is deemed a mode +of using an interface provided by the Library. + + A "Combined Work" is a work produced by combining or linking an +Application with the Library. The particular version of the Library +with which the Combined Work was made is also called the "Linked +Version". + + The "Minimal Corresponding Source" for a Combined Work means the +Corresponding Source for the Combined Work, excluding any source code +for portions of the Combined Work that, considered in isolation, are +based on the Application, and not on the Linked Version. + + The "Corresponding Application Code" for a Combined Work means the +object code and/or source code for the Application, including any data +and utility programs needed for reproducing the Combined Work from the +Application, but excluding the System Libraries of the Combined Work. + + 1. Exception to Section 3 of the GNU GPL. + + You may convey a covered work under sections 3 and 4 of this License +without being bound by section 3 of the GNU GPL. + + 2. Conveying Modified Versions. + + If you modify a copy of the Library, and, in your modifications, a +facility refers to a function or data to be supplied by an Application +that uses the facility (other than as an argument passed when the +facility is invoked), then you may convey a copy of the modified +version: + + a) under this License, provided that you make a good faith effort to + ensure that, in the event an Application does not supply the + function or data, the facility still operates, and performs + whatever part of its purpose remains meaningful, or + + b) under the GNU GPL, with none of the additional permissions of + this License applicable to that copy. + + 3. Object Code Incorporating Material from Library Header Files. + + The object code form of an Application may incorporate material from +a header file that is part of the Library. You may convey such object +code under terms of your choice, provided that, if the incorporated +material is not limited to numerical parameters, data structure +layouts and accessors, or small macros, inline functions and templates +(ten or fewer lines in length), you do both of the following: + + a) Give prominent notice with each copy of the object code that the + Library is used in it and that the Library and its use are + covered by this License. + + b) Accompany the object code with a copy of the GNU GPL and this license + document. + + 4. Combined Works. + + You may convey a Combined Work under terms of your choice that, +taken together, effectively do not restrict modification of the +portions of the Library contained in the Combined Work and reverse +engineering for debugging such modifications, if you also do each of +the following: + + a) Give prominent notice with each copy of the Combined Work that + the Library is used in it and that the Library and its use are + covered by this License. + + b) Accompany the Combined Work with a copy of the GNU GPL and this license + document. + + c) For a Combined Work that displays copyright notices during + execution, include the copyright notice for the Library among + these notices, as well as a reference directing the user to the + copies of the GNU GPL and this license document. + + d) Do one of the following: + + 0) Convey the Minimal Corresponding Source under the terms of this + License, and the Corresponding Application Code in a form + suitable for, and under terms that permit, the user to + recombine or relink the Application with a modified version of + the Linked Version to produce a modified Combined Work, in the + manner specified by section 6 of the GNU GPL for conveying + Corresponding Source. + + 1) Use a suitable shared library mechanism for linking with the + Library. A suitable mechanism is one that (a) uses at run time + a copy of the Library already present on the user's computer + system, and (b) will operate properly with a modified version + of the Library that is interface-compatible with the Linked + Version. + + e) Provide Installation Information, but only if you would otherwise + be required to provide such information under section 6 of the + GNU GPL, and only to the extent that such information is + necessary to install and execute a modified version of the + Combined Work produced by recombining or relinking the + Application with a modified version of the Linked Version. (If + you use option 4d0, the Installation Information must accompany + the Minimal Corresponding Source and Corresponding Application + Code. If you use option 4d1, you must provide the Installation + Information in the manner specified by section 6 of the GNU GPL + for conveying Corresponding Source.) + + 5. Combined Libraries. + + You may place library facilities that are a work based on the +Library side by side in a single library together with other library +facilities that are not Applications and are not covered by this +License, and convey such a combined library under terms of your +choice, if you do both of the following: + + a) Accompany the combined library with a copy of the same work based + on the Library, uncombined with any other library facilities, + conveyed under the terms of this License. + + b) Give prominent notice with the combined library that part of it + is a work based on the Library, and explaining where to find the + accompanying uncombined form of the same work. + + 6. Revised Versions of the GNU Lesser General Public License. + + The Free Software Foundation may publish revised and/or new versions +of the GNU Lesser General Public License from time to time. Such new +versions will be similar in spirit to the present version, but may +differ in detail to address new problems or concerns. + + Each version is given a distinguishing version number. If the +Library as you received it specifies that a certain numbered version +of the GNU Lesser General Public License "or any later version" +applies to it, you have the option of following the terms and +conditions either of that published version or of any later version +published by the Free Software Foundation. If the Library as you +received it does not specify a version number of the GNU Lesser +General Public License, you may choose any version of the GNU Lesser +General Public License ever published by the Free Software Foundation. + + If the Library as you received it specifies that a proxy can decide +whether future versions of the GNU Lesser General Public License shall +apply, that proxy's public statement of acceptance of any version is +permanent authorization for you to choose that version for the +Library. + +------------------------------------------------------------------------------ +bzip2 is redistributed within all opencv-python Linux packages. +This license applies to libbz2 binary in the directory cv2/. + +This program, "bzip2", the associated library "libbzip2", and all +documentation, are copyright (C) 1996-2010 Julian R Seward. All +rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. The origin of this software must not be misrepresented; you must + not claim that you wrote the original software. If you use this + software in a product, an acknowledgment in the product + documentation would be appreciated but is not required. + +3. Altered source versions must be plainly marked as such, and must + not be misrepresented as being the original software. + +4. The name of the author may not be used to endorse or promote + products derived from this software without specific prior written + permission. + +THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS +OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE +GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, +WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Julian Seward, jseward@bzip.org +bzip2/libbzip2 version 1.0.6 of 6 September 2010 + +------------------------------------------------------------------------------ +libcrypto and libssl are redistributed within all opencv-python Linux and macOS packages. +libopencore-amrnb and libopencore-amrwb are redistributed within all opencv-python Linux and macOS packages. +This license applies to above binaries in the directory cv2/. + + LICENSE ISSUES + ============== + + The OpenSSL toolkit stays under a double license, i.e. both the conditions of + the OpenSSL License and the original SSLeay license apply to the toolkit. + See below for the actual license texts. + + OpenSSL License + --------------- + +/* ==================================================================== + * Copyright (c) 1998-2019 The OpenSSL Project. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * 3. All advertising materials mentioning features or use of this + * software must display the following acknowledgment: + * "This product includes software developed by the OpenSSL Project + * for use in the OpenSSL Toolkit. (http://www.openssl.org/)" + * + * 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to + * endorse or promote products derived from this software without + * prior written permission. For written permission, please contact + * openssl-core@openssl.org. + * + * 5. Products derived from this software may not be called "OpenSSL" + * nor may "OpenSSL" appear in their names without prior written + * permission of the OpenSSL Project. + * + * 6. Redistributions of any form whatsoever must retain the following + * acknowledgment: + * "This product includes software developed by the OpenSSL Project + * for use in the OpenSSL Toolkit (http://www.openssl.org/)" + * + * THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY + * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR + * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED + * OF THE POSSIBILITY OF SUCH DAMAGE. + * ==================================================================== + * + * This product includes cryptographic software written by Eric Young + * (eay@cryptsoft.com). This product includes software written by Tim + * Hudson (tjh@cryptsoft.com). + * + */ + + Original SSLeay License + ----------------------- + +/* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com) + * All rights reserved. + * + * This package is an SSL implementation written + * by Eric Young (eay@cryptsoft.com). + * The implementation was written so as to conform with Netscapes SSL. + * + * This library is free for commercial and non-commercial use as long as + * the following conditions are adhered to. The following conditions + * apply to all code found in this distribution, be it the RC4, RSA, + * lhash, DES, etc., code; not just the SSL code. The SSL documentation + * included with this distribution is covered by the same copyright terms + * except that the holder is Tim Hudson (tjh@cryptsoft.com). + * + * Copyright remains Eric Young's, and as such any Copyright notices in + * the code are not to be removed. + * If this package is used in a product, Eric Young should be given attribution + * as the author of the parts of the library used. + * This can be in the form of a textual message at program startup or + * in documentation (online or textual) provided with the package. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. All advertising materials mentioning features or use of this software + * must display the following acknowledgement: + * "This product includes cryptographic software written by + * Eric Young (eay@cryptsoft.com)" + * The word 'cryptographic' can be left out if the routines from the library + * being used are not cryptographic related :-). + * 4. If you include any Windows specific code (or a derivative thereof) from + * the apps directory (application code) you must include an acknowledgement: + * "This product includes software written by Tim Hudson (tjh@cryptsoft.com)" + * + * THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * The licence and distribution terms for any publicly available version or + * derivative of this code cannot be changed. i.e. this code cannot simply be + * copied and put under another distribution licence + * [including the GNU Public Licence.] + */ + +------------------------------------------------------------------------------ +libfontconfig is redistributed within all opencv-python macOS packages. +This license applies to libfontconfig binary in the directory cv2/. + +Copyright © 2000,2001,2002,2003,2004,2006,2007 Keith Packard +Copyright © 2005 Patrick Lam +Copyright © 2009 Roozbeh Pournader +Copyright © 2008,2009 Red Hat, Inc. +Copyright © 2008 Danilo Šegan +Copyright © 2012 Google, Inc. + + +Permission to use, copy, modify, distribute, and sell this software and its +documentation for any purpose is hereby granted without fee, provided that +the above copyright notice appear in all copies and that both that +copyright notice and this permission notice appear in supporting +documentation, and that the name of the author(s) not be used in +advertising or publicity pertaining to distribution of the software without +specific, written prior permission. The authors make no +representations about the suitability of this software for any purpose. It +is provided "as is" without express or implied warranty. + +THE AUTHOR(S) DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, +INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO +EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY SPECIAL, INDIRECT OR +CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, +DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER +TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR +PERFORMANCE OF THIS SOFTWARE. + +------------------------------------------------------------------------------ +libfreetype is redistributed within opencv-python Linux and macOS packages. +This license applies to libfreetype binary in the directory cv2/. + + The FreeType Project LICENSE + ---------------------------- + + 2006-Jan-27 + + Copyright 1996-2002, 2006 by + David Turner, Robert Wilhelm, and Werner Lemberg + + + +Introduction +============ + + The FreeType Project is distributed in several archive packages; + some of them may contain, in addition to the FreeType font engine, + various tools and contributions which rely on, or relate to, the + FreeType Project. + + This license applies to all files found in such packages, and + which do not fall under their own explicit license. The license + affects thus the FreeType font engine, the test programs, + documentation and makefiles, at the very least. + + This license was inspired by the BSD, Artistic, and IJG + (Independent JPEG Group) licenses, which all encourage inclusion + and use of free software in commercial and freeware products + alike. As a consequence, its main points are that: + + o We don't promise that this software works. However, we will be + interested in any kind of bug reports. (`as is' distribution) + + o You can use this software for whatever you want, in parts or + full form, without having to pay us. (`royalty-free' usage) + + o You may not pretend that you wrote this software. If you use + it, or only parts of it, in a program, you must acknowledge + somewhere in your documentation that you have used the + FreeType code. (`credits') + + We specifically permit and encourage the inclusion of this + software, with or without modifications, in commercial products. + We disclaim all warranties covering The FreeType Project and + assume no liability related to The FreeType Project. + + + Finally, many people asked us for a preferred form for a + credit/disclaimer to use in compliance with this license. We thus + encourage you to use the following text: + + """ + Portions of this software are copyright © The FreeType + Project (www.freetype.org). All rights reserved. + """ + + Please replace with the value from the FreeType version you + actually use. + + +Legal Terms +=========== + +0. Definitions +-------------- + + Throughout this license, the terms `package', `FreeType Project', + and `FreeType archive' refer to the set of files originally + distributed by the authors (David Turner, Robert Wilhelm, and + Werner Lemberg) as the `FreeType Project', be they named as alpha, + beta or final release. + + `You' refers to the licensee, or person using the project, where + `using' is a generic term including compiling the project's source + code as well as linking it to form a `program' or `executable'. + This program is referred to as `a program using the FreeType + engine'. + + This license applies to all files distributed in the original + FreeType Project, including all source code, binaries and + documentation, unless otherwise stated in the file in its + original, unmodified form as distributed in the original archive. + If you are unsure whether or not a particular file is covered by + this license, you must contact us to verify this. + + The FreeType Project is copyright (C) 1996-2000 by David Turner, + Robert Wilhelm, and Werner Lemberg. All rights reserved except as + specified below. + +1. No Warranty +-------------- + + THE FREETYPE PROJECT IS PROVIDED `AS IS' WITHOUT WARRANTY OF ANY + KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, + WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + PURPOSE. IN NO EVENT WILL ANY OF THE AUTHORS OR COPYRIGHT HOLDERS + BE LIABLE FOR ANY DAMAGES CAUSED BY THE USE OR THE INABILITY TO + USE, OF THE FREETYPE PROJECT. + +2. Redistribution +----------------- + + This license grants a worldwide, royalty-free, perpetual and + irrevocable right and license to use, execute, perform, compile, + display, copy, create derivative works of, distribute and + sublicense the FreeType Project (in both source and object code + forms) and derivative works thereof for any purpose; and to + authorize others to exercise some or all of the rights granted + herein, subject to the following conditions: + + o Redistribution of source code must retain this license file + (`FTL.TXT') unaltered; any additions, deletions or changes to + the original files must be clearly indicated in accompanying + documentation. The copyright notices of the unaltered, + original files must be preserved in all copies of source + files. + + o Redistribution in binary form must provide a disclaimer that + states that the software is based in part of the work of the + FreeType Team, in the distribution documentation. We also + encourage you to put an URL to the FreeType web page in your + documentation, though this isn't mandatory. + + These conditions apply to any software derived from or based on + the FreeType Project, not just the unmodified files. If you use + our work, you must acknowledge us. However, no fee need be paid + to us. + +3. Advertising +-------------- + + Neither the FreeType authors and contributors nor you shall use + the name of the other for commercial, advertising, or promotional + purposes without specific prior written permission. + + We suggest, but do not require, that you use one or more of the + following phrases to refer to this software in your documentation + or advertising materials: `FreeType Project', `FreeType Engine', + `FreeType library', or `FreeType Distribution'. + + As you have not signed this license, you are not required to + accept it. However, as the FreeType Project is copyrighted + material, only this license, or another one contracted with the + authors, grants you the right to use, distribute, and modify it. + Therefore, by using, distributing, or modifying the FreeType + Project, you indicate that you understand and accept all the terms + of this license. + +4. Contacts +----------- + + There are two mailing lists related to FreeType: + + o freetype@nongnu.org + + Discusses general use and applications of FreeType, as well as + future and wanted additions to the library and distribution. + If you are looking for support, start in this list if you + haven't found anything to help you in the documentation. + + o freetype-devel@nongnu.org + + Discusses bugs, as well as engine internals, design issues, + specific licenses, porting, etc. + + Our home page can be found at + + https://www.freetype.org + +------------------------------------------------------------------------------ +libpng is redistributed within all opencv-python Linux and macOS packages. +This license applies to libpng binary in the directory cv2/. + +PNG Reference Library License version 2 +--------------------------------------- + + * Copyright (c) 1995-2019 The PNG Reference Library Authors. + * Copyright (c) 2018-2019 Cosmin Truta. + * Copyright (c) 2000-2002, 2004, 2006-2018 Glenn Randers-Pehrson. + * Copyright (c) 1996-1997 Andreas Dilger. + * Copyright (c) 1995-1996 Guy Eric Schalnat, Group 42, Inc. + +The software is supplied "as is", without warranty of any kind, +express or implied, including, without limitation, the warranties +of merchantability, fitness for a particular purpose, title, and +non-infringement. In no event shall the Copyright owners, or +anyone distributing the software, be liable for any damages or +other liability, whether in contract, tort or otherwise, arising +from, out of, or in connection with the software, or the use or +other dealings in the software, even if advised of the possibility +of such damage. + +Permission is hereby granted to use, copy, modify, and distribute +this software, or portions hereof, for any purpose, without fee, +subject to the following restrictions: + + 1. The origin of this software must not be misrepresented; you + must not claim that you wrote the original software. If you + use this software in a product, an acknowledgment in the product + documentation would be appreciated, but is not required. + + 2. Altered source versions must be plainly marked as such, and must + not be misrepresented as being the original software. + + 3. This Copyright notice may not be removed or altered from any + source or altered source distribution. + + +PNG Reference Library License version 1 (for libpng 0.5 through 1.6.35) +----------------------------------------------------------------------- + +libpng versions 1.0.7, July 1, 2000, through 1.6.35, July 15, 2018 are +Copyright (c) 2000-2002, 2004, 2006-2018 Glenn Randers-Pehrson, are +derived from libpng-1.0.6, and are distributed according to the same +disclaimer and license as libpng-1.0.6 with the following individuals +added to the list of Contributing Authors: + + Simon-Pierre Cadieux + Eric S. Raymond + Mans Rullgard + Cosmin Truta + Gilles Vollant + James Yu + Mandar Sahastrabuddhe + Google Inc. + Vadim Barkov + +and with the following additions to the disclaimer: + + There is no warranty against interference with your enjoyment of + the library or against infringement. There is no warranty that our + efforts or the library will fulfill any of your particular purposes + or needs. This library is provided with all faults, and the entire + risk of satisfactory quality, performance, accuracy, and effort is + with the user. + +Some files in the "contrib" directory and some configure-generated +files that are distributed with libpng have other copyright owners, and +are released under other open source licenses. + +libpng versions 0.97, January 1998, through 1.0.6, March 20, 2000, are +Copyright (c) 1998-2000 Glenn Randers-Pehrson, are derived from +libpng-0.96, and are distributed according to the same disclaimer and +license as libpng-0.96, with the following individuals added to the +list of Contributing Authors: + + Tom Lane + Glenn Randers-Pehrson + Willem van Schaik + +libpng versions 0.89, June 1996, through 0.96, May 1997, are +Copyright (c) 1996-1997 Andreas Dilger, are derived from libpng-0.88, +and are distributed according to the same disclaimer and license as +libpng-0.88, with the following individuals added to the list of +Contributing Authors: + + John Bowler + Kevin Bracey + Sam Bushell + Magnus Holmgren + Greg Roelofs + Tom Tanner + +Some files in the "scripts" directory have other copyright owners, +but are released under this license. + +libpng versions 0.5, May 1995, through 0.88, January 1996, are +Copyright (c) 1995-1996 Guy Eric Schalnat, Group 42, Inc. + +For the purposes of this copyright and license, "Contributing Authors" +is defined as the following set of individuals: + + Andreas Dilger + Dave Martindale + Guy Eric Schalnat + Paul Schmidt + Tim Wegner + +The PNG Reference Library is supplied "AS IS". The Contributing +Authors and Group 42, Inc. disclaim all warranties, expressed or +implied, including, without limitation, the warranties of +merchantability and of fitness for any purpose. The Contributing +Authors and Group 42, Inc. assume no liability for direct, indirect, +incidental, special, exemplary, or consequential damages, which may +result from the use of the PNG Reference Library, even if advised of +the possibility of such damage. + +Permission is hereby granted to use, copy, modify, and distribute this +source code, or portions hereof, for any purpose, without fee, subject +to the following restrictions: + + 1. The origin of this source code must not be misrepresented. + + 2. Altered versions must be plainly marked as such and must not + be misrepresented as being the original source. + + 3. This Copyright notice may not be removed or altered from any + source or altered source distribution. + +The Contributing Authors and Group 42, Inc. specifically permit, +without fee, and encourage the use of this source code as a component +to supporting the PNG file format in commercial products. If you use +this source code in a product, acknowledgment is not required but would +be appreciated. + +------------------------------------------------------------------------------ +libz is redistributed within all opencv-python Linux packages. +This license applies to libz binary in the directory cv2/. + + Copyright (C) 1995-2017 Jean-loup Gailly and Mark Adler + + This software is provided 'as-is', without any express or implied + warranty. In no event will the authors be held liable for any damages + arising from the use of this software. + + Permission is granted to anyone to use this software for any purpose, + including commercial applications, and to alter it and redistribute it + freely, subject to the following restrictions: + + 1. The origin of this software must not be misrepresented; you must not + claim that you wrote the original software. If you use this software + in a product, an acknowledgment in the product documentation would be + appreciated but is not required. + 2. Altered source versions must be plainly marked as such, and must not be + misrepresented as being the original software. + 3. This notice may not be removed or altered from any source distribution. + + Jean-loup Gailly Mark Adler + jloup@gzip.org madler@alumni.caltech.edu + +------------------------------------------------------------------------------ +libdav1d is redistributed within opencv-python macOS packages. +This license applies to libdav1d binary in the directory cv2/. + +Copyright © 2018-2019, VideoLAN and dav1d authors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +libffi is redistributed within opencv-python macOS packages. +This license applies to libffi binary in the directory cv2/. + +libffi - Copyright (c) 1996-2020 Anthony Green, Red Hat, Inc and others. +See source files for details. + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +``Software''), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, +TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +------------------------------------------------------------------------------ +libogg is redistributed within opencv-python macOS packages. +This license applies to libogg binary in the directory cv2/. + +Copyright (c) 2002, Xiph.org Foundation + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +- Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + +- Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +- Neither the name of the Xiph.org Foundation nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION +OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +libopenjp2 is redistributed within opencv-python macOS packages. +This license applies to libopenjp2 binary in the directory cv2/. + +The copyright in this software is being made available under the 2-clauses +BSD License, included below. This software may be subject to other third +party and contributor rights, including patent rights, and no such rights +are granted under this license. + +Copyright (c) 2002-2014, Universite catholique de Louvain (UCL), Belgium +Copyright (c) 2002-2014, Professor Benoit Macq +Copyright (c) 2003-2014, Antonin Descampe +Copyright (c) 2003-2009, Francois-Olivier Devaux +Copyright (c) 2005, Herve Drolon, FreeImage Team +Copyright (c) 2002-2003, Yannick Verschueren +Copyright (c) 2001-2003, David Janssens +Copyright (c) 2011-2012, Centre National d'Etudes Spatiales (CNES), France +Copyright (c) 2012, CS Systemes d'Information, France + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS `AS IS' +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +libopus is redistributed within opencv-python macOS packages. +This license applies to libopus binary in the directory cv2/. + +Copyright 2001-2011 Xiph.Org, Skype Limited, Octasic, + Jean-Marc Valin, Timothy B. Terriberry, + CSIRO, Gregory Maxwell, Mark Borgerding, + Erik de Castro Lopo + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +- Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + +- Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +- Neither the name of Internet Society, IETF or IETF Trust, nor the +names of specific contributors, may be used to endorse or promote +products derived from this software without specific prior written +permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER +OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Opus is subject to the royalty-free patent licenses which are +specified at: + +Xiph.Org Foundation: +https://datatracker.ietf.org/ipr/1524/ + +Microsoft Corporation: +https://datatracker.ietf.org/ipr/1914/ + +Broadcom Corporation: +https://datatracker.ietf.org/ipr/1526/ + +------------------------------------------------------------------------------ +librav1e is redistributed within opencv-python macOS packages. +This license applies to librav1e binary in the directory cv2/. + +BSD 2-Clause License + +Copyright (c) 2017-2020, the rav1e contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +libsnappy is redistributed within opencv-python macOS packages. +This license applies to libsnappy binary in the directory cv2/. + +Copyright 2011, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +libspeex is redistributed within opencv-python macOS packages. +This license applies to libspeex binary in the directory cv2/. + +Copyright 2002-2008 Xiph.org Foundation +Copyright 2002-2008 Jean-Marc Valin +Copyright 2005-2007 Analog Devices Inc. +Copyright 2005-2008 Commonwealth Scientific and Industrial Research + Organisation (CSIRO) +Copyright 1993, 2002, 2006 David Rowe +Copyright 2003 EpicGames +Copyright 1992-1994 Jutta Degener, Carsten Bormann + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +- Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + +- Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +- Neither the name of the Xiph.org Foundation nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR +CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +libsrt is redistributed within opencv-python macOS packages. +This license applies to libsrt binary in the directory cv2/. + +/* + * + * Copyright (c) 2001-2017 Cisco Systems, Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * Neither the name of the Cisco Systems, Inc. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS + * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE + * COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED + * OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + + + Mozilla Public License Version 2.0 +================================== + +1. Definitions +-------------- + +1.1. "Contributor" + means each individual or legal entity that creates, contributes to + the creation of, or owns Covered Software. + +1.2. "Contributor Version" + means the combination of the Contributions of others (if any) used + by a Contributor and that particular Contributor's Contribution. + +1.3. "Contribution" + means Covered Software of a particular Contributor. + +1.4. "Covered Software" + means Source Code Form to which the initial Contributor has attached + the notice in Exhibit A, the Executable Form of such Source Code + Form, and Modifications of such Source Code Form, in each case + including portions thereof. + +1.5. "Incompatible With Secondary Licenses" + means + + (a) that the initial Contributor has attached the notice described + in Exhibit B to the Covered Software; or + + (b) that the Covered Software was made available under the terms of + version 1.1 or earlier of the License, but not also under the + terms of a Secondary License. + +1.6. "Executable Form" + means any form of the work other than Source Code Form. + +1.7. "Larger Work" + means a work that combines Covered Software with other material, in + a separate file or files, that is not Covered Software. + +1.8. "License" + means this document. + +1.9. "Licensable" + means having the right to grant, to the maximum extent possible, + whether at the time of the initial grant or subsequently, any and + all of the rights conveyed by this License. + +1.10. "Modifications" + means any of the following: + + (a) any file in Source Code Form that results from an addition to, + deletion from, or modification of the contents of Covered + Software; or + + (b) any new file in Source Code Form that contains any Covered + Software. + +1.11. "Patent Claims" of a Contributor + means any patent claim(s), including without limitation, method, + process, and apparatus claims, in any patent Licensable by such + Contributor that would be infringed, but for the grant of the + License, by the making, using, selling, offering for sale, having + made, import, or transfer of either its Contributions or its + Contributor Version. + +1.12. "Secondary License" + means either the GNU General Public License, Version 2.0, the GNU + Lesser General Public License, Version 2.1, the GNU Affero General + Public License, Version 3.0, or any later versions of those + licenses. + +1.13. "Source Code Form" + means the form of the work preferred for making modifications. + +1.14. "You" (or "Your") + means an individual or a legal entity exercising rights under this + License. For legal entities, "You" includes any entity that + controls, is controlled by, or is under common control with You. For + purposes of this definition, "control" means (a) the power, direct + or indirect, to cause the direction or management of such entity, + whether by contract or otherwise, or (b) ownership of more than + fifty percent (50%) of the outstanding shares or beneficial + ownership of such entity. + +2. License Grants and Conditions +-------------------------------- + +2.1. Grants + +Each Contributor hereby grants You a world-wide, royalty-free, +non-exclusive license: + +(a) under intellectual property rights (other than patent or trademark) + Licensable by such Contributor to use, reproduce, make available, + modify, display, perform, distribute, and otherwise exploit its + Contributions, either on an unmodified basis, with Modifications, or + as part of a Larger Work; and + +(b) under Patent Claims of such Contributor to make, use, sell, offer + for sale, have made, import, and otherwise transfer either its + Contributions or its Contributor Version. + +2.2. Effective Date + +The licenses granted in Section 2.1 with respect to any Contribution +become effective for each Contribution on the date the Contributor first +distributes such Contribution. + +2.3. Limitations on Grant Scope + +The licenses granted in this Section 2 are the only rights granted under +this License. No additional rights or licenses will be implied from the +distribution or licensing of Covered Software under this License. +Notwithstanding Section 2.1(b) above, no patent license is granted by a +Contributor: + +(a) for any code that a Contributor has removed from Covered Software; + or + +(b) for infringements caused by: (i) Your and any other third party's + modifications of Covered Software, or (ii) the combination of its + Contributions with other software (except as part of its Contributor + Version); or + +(c) under Patent Claims infringed by Covered Software in the absence of + its Contributions. + +This License does not grant any rights in the trademarks, service marks, +or logos of any Contributor (except as may be necessary to comply with +the notice requirements in Section 3.4). + +2.4. Subsequent Licenses + +No Contributor makes additional grants as a result of Your choice to +distribute the Covered Software under a subsequent version of this +License (see Section 10.2) or under the terms of a Secondary License (if +permitted under the terms of Section 3.3). + +2.5. Representation + +Each Contributor represents that the Contributor believes its +Contributions are its original creation(s) or it has sufficient rights +to grant the rights to its Contributions conveyed by this License. + +2.6. Fair Use + +This License is not intended to limit any rights You have under +applicable copyright doctrines of fair use, fair dealing, or other +equivalents. + +2.7. Conditions + +Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted +in Section 2.1. + +3. Responsibilities +------------------- + +3.1. Distribution of Source Form + +All distribution of Covered Software in Source Code Form, including any +Modifications that You create or to which You contribute, must be under +the terms of this License. You must inform recipients that the Source +Code Form of the Covered Software is governed by the terms of this +License, and how they can obtain a copy of this License. You may not +attempt to alter or restrict the recipients' rights in the Source Code +Form. + +3.2. Distribution of Executable Form + +If You distribute Covered Software in Executable Form then: + +(a) such Covered Software must also be made available in Source Code + Form, as described in Section 3.1, and You must inform recipients of + the Executable Form how they can obtain a copy of such Source Code + Form by reasonable means in a timely manner, at a charge no more + than the cost of distribution to the recipient; and + +(b) You may distribute such Executable Form under the terms of this + License, or sublicense it under different terms, provided that the + license for the Executable Form does not attempt to limit or alter + the recipients' rights in the Source Code Form under this License. + +3.3. Distribution of a Larger Work + +You may create and distribute a Larger Work under terms of Your choice, +provided that You also comply with the requirements of this License for +the Covered Software. If the Larger Work is a combination of Covered +Software with a work governed by one or more Secondary Licenses, and the +Covered Software is not Incompatible With Secondary Licenses, this +License permits You to additionally distribute such Covered Software +under the terms of such Secondary License(s), so that the recipient of +the Larger Work may, at their option, further distribute the Covered +Software under the terms of either this License or such Secondary +License(s). + +3.4. Notices + +You may not remove or alter the substance of any license notices +(including copyright notices, patent notices, disclaimers of warranty, +or limitations of liability) contained within the Source Code Form of +the Covered Software, except that You may alter any license notices to +the extent required to remedy known factual inaccuracies. + +3.5. Application of Additional Terms + +You may choose to offer, and to charge a fee for, warranty, support, +indemnity or liability obligations to one or more recipients of Covered +Software. However, You may do so only on Your own behalf, and not on +behalf of any Contributor. You must make it absolutely clear that any +such warranty, support, indemnity, or liability obligation is offered by +You alone, and You hereby agree to indemnify every Contributor for any +liability incurred by such Contributor as a result of warranty, support, +indemnity or liability terms You offer. You may include additional +disclaimers of warranty and limitations of liability specific to any +jurisdiction. + +4. Inability to Comply Due to Statute or Regulation +--------------------------------------------------- + +If it is impossible for You to comply with any of the terms of this +License with respect to some or all of the Covered Software due to +statute, judicial order, or regulation then You must: (a) comply with +the terms of this License to the maximum extent possible; and (b) +describe the limitations and the code they affect. Such description must +be placed in a text file included with all distributions of the Covered +Software under this License. Except to the extent prohibited by statute +or regulation, such description must be sufficiently detailed for a +recipient of ordinary skill to be able to understand it. + +5. Termination +-------------- + +5.1. The rights granted under this License will terminate automatically +if You fail to comply with any of its terms. However, if You become +compliant, then the rights granted under this License from a particular +Contributor are reinstated (a) provisionally, unless and until such +Contributor explicitly and finally terminates Your grants, and (b) on an +ongoing basis, if such Contributor fails to notify You of the +non-compliance by some reasonable means prior to 60 days after You have +come back into compliance. Moreover, Your grants from a particular +Contributor are reinstated on an ongoing basis if such Contributor +notifies You of the non-compliance by some reasonable means, this is the +first time You have received notice of non-compliance with this License +from such Contributor, and You become compliant prior to 30 days after +Your receipt of the notice. + +5.2. If You initiate litigation against any entity by asserting a patent +infringement claim (excluding declaratory judgment actions, +counter-claims, and cross-claims) alleging that a Contributor Version +directly or indirectly infringes any patent, then the rights granted to +You by any and all Contributors for the Covered Software under Section +2.1 of this License shall terminate. + +5.3. In the event of termination under Sections 5.1 or 5.2 above, all +end user license agreements (excluding distributors and resellers) which +have been validly granted by You or Your distributors under this License +prior to termination shall survive termination. + +************************************************************************ +* * +* 6. Disclaimer of Warranty * +* ------------------------- * +* * +* Covered Software is provided under this License on an "as is" * +* basis, without warranty of any kind, either expressed, implied, or * +* statutory, including, without limitation, warranties that the * +* Covered Software is free of defects, merchantable, fit for a * +* particular purpose or non-infringing. The entire risk as to the * +* quality and performance of the Covered Software is with You. * +* Should any Covered Software prove defective in any respect, You * +* (not any Contributor) assume the cost of any necessary servicing, * +* repair, or correction. This disclaimer of warranty constitutes an * +* essential part of this License. No use of any Covered Software is * +* authorized under this License except under this disclaimer. * +* * +************************************************************************ + +************************************************************************ +* * +* 7. Limitation of Liability * +* -------------------------- * +* * +* Under no circumstances and under no legal theory, whether tort * +* (including negligence), contract, or otherwise, shall any * +* Contributor, or anyone who distributes Covered Software as * +* permitted above, be liable to You for any direct, indirect, * +* special, incidental, or consequential damages of any character * +* including, without limitation, damages for lost profits, loss of * +* goodwill, work stoppage, computer failure or malfunction, or any * +* and all other commercial damages or losses, even if such party * +* shall have been informed of the possibility of such damages. This * +* limitation of liability shall not apply to liability for death or * +* personal injury resulting from such party's negligence to the * +* extent applicable law prohibits such limitation. Some * +* jurisdictions do not allow the exclusion or limitation of * +* incidental or consequential damages, so this exclusion and * +* limitation may not apply to You. * +* * +************************************************************************ + +8. Litigation +------------- + +Any litigation relating to this License may be brought only in the +courts of a jurisdiction where the defendant maintains its principal +place of business and such litigation shall be governed by laws of that +jurisdiction, without reference to its conflict-of-law provisions. +Nothing in this Section shall prevent a party's ability to bring +cross-claims or counter-claims. + +9. Miscellaneous +---------------- + +This License represents the complete agreement concerning the subject +matter hereof. If any provision of this License is held to be +unenforceable, such provision shall be reformed only to the extent +necessary to make it enforceable. Any law or regulation which provides +that the language of a contract shall be construed against the drafter +shall not be used to construe this License against a Contributor. + +10. Versions of the License +--------------------------- + +10.1. New Versions + +Mozilla Foundation is the license steward. Except as provided in Section +10.3, no one other than the license steward has the right to modify or +publish new versions of this License. Each version will be given a +distinguishing version number. + +10.2. Effect of New Versions + +You may distribute the Covered Software under the terms of the version +of the License under which You originally received the Covered Software, +or under the terms of any subsequent version published by the license +steward. + +10.3. Modified Versions + +If you create software not governed by this License, and you want to +create a new license for such software, you may create and use a +modified version of this License if you rename the license and remove +any references to the name of the license steward (except to note that +such modified license differs from this License). + +10.4. Distributing Source Code Form that is Incompatible With Secondary +Licenses + +If You choose to distribute Source Code Form that is Incompatible With +Secondary Licenses under the terms of this version of the License, the +notice described in Exhibit B of this License must be attached. + +Exhibit A - Source Code Form License Notice +------------------------------------------- + + This Source Code Form is subject to the terms of the Mozilla Public + License, v. 2.0. If a copy of the MPL was not distributed with this + file, You can obtain one at http://mozilla.org/MPL/2.0/. + +If it is not possible or desirable to put the notice in a particular +file, then You may include the notice in a location (such as a LICENSE +file in a relevant directory) where a recipient would be likely to look +for such a notice. + +You may add additional accurate notices of copyright ownership. + +Exhibit B - "Incompatible With Secondary Licenses" Notice +--------------------------------------------------------- + + This Source Code Form is "Incompatible With Secondary Licenses", as + defined by the Mozilla Public License, v. 2.0. + +------------------------------------------------------------------------------ +libtheoradec and libtheoraenc are redistributed within opencv-python macOS packages. +This license applies to libtheoradec and libtheoraenc binaries in the directory cv2/. + + Copyright (C) 2002-2009 Xiph.org Foundation + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +- Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + +- Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +- Neither the name of the Xiph.org Foundation nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION +OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +libwebp and libwebpmux are redistributed within all opencv-python packages. +This license applies to libwebp and libwebpmux binaries in the directory cv2/. + +Copyright (c) 2010, Google Inc. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + + * Neither the name of Google nor the names of its contributors may + be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +libvorbis and libvorbisenc are redistributed within opencv-python macOS packages. +This license applies to libvorbis and libvorbisenc binaries in the directory cv2/. + +Copyright (c) 2002-2020 Xiph.org Foundation + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +- Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + +- Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +- Neither the name of the Xiph.org Foundation nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION +OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +Libxcb utility libraries are redistributed within opencv-python non-headless Linux packages. +This license applies to libxcb related binaries in the directory cv2/. + +Copyright (C) 2001-2006 Bart Massey, Jamey Sharp, and Josh Triplett. +All Rights Reserved. + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated +documentation files (the "Software"), to deal in the +Software without restriction, including without limitation +the rights to use, copy, modify, merge, publish, distribute, +sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall +be included in all copies or substantial portions of the +Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY +KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE +WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR +PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS +BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the names of the authors +or their institutions shall not be used in advertising or +otherwise to promote the sale, use or other dealings in this +Software without prior written authorization from the +authors. + +------------------------------------------------------------------------------ +Libxcb-image is redistributed within opencv-python non-headless Linux packages. +This license applies to libxcb-image binary in the directory cv2/. + +Copyright © 2007-2008 Bart Massey +Copyright © 2008 Julien Danjou +Copyright © 2008 Keith Packard + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated documentation +files (the "Software"), to deal in the Software without +restriction, including without limitation the rights to use, copy, +modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF +CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the names of the authors or +their institutions shall not be used in advertising or otherwise to +promote the sale, use or other dealings in this Software without +prior written authorization from the authors. + +------------------------------------------------------------------------------ +Libxcb-util is redistributed within opencv-python non-headless Linux packages. +This license applies to libxcb-util binary in the directory cv2/. + +Copyright © 2008 Bart Massey +Copyright © 2008 Ian Osgood +Copyright © 2008 Jamey Sharp +Copyright © 2008 Josh Triplett +Copyright © 2008-2009 Julien Danjou + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated documentation +files (the "Software"), to deal in the Software without +restriction, including without limitation the rights to use, copy, +modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF +CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the names of the authors or +their institutions shall not be used in advertising or otherwise to +promote the sale, use or other dealings in this Software without +prior written authorization from the authors. + +------------------------------------------------------------------------------ +Libxcb-render-util is redistributed within opencv-python non-headless Linux packages. +This license applies to libxcb-render-util binary in the directory cv2/. + +Copyright © 2000 Keith Packard + +Permission to use, copy, modify, distribute, and sell this software and its +documentation for any purpose is hereby granted without fee, provided that +the above copyright notice appear in all copies and that both that +copyright notice and this permission notice appear in supporting +documentation, and that the name of Keith Packard not be used in +advertising or publicity pertaining to distribution of the software without +specific, written prior permission. Keith Packard makes no +representations about the suitability of this software for any purpose. It +is provided "as is" without express or implied warranty. + +KEITH PACKARD DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, +INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO +EVENT SHALL KEITH PACKARD BE LIABLE FOR ANY SPECIAL, INDIRECT OR +CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, +DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER +TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR +PERFORMANCE OF THIS SOFTWARE. + +Copyright © 2006 Jamey Sharp. + +Permission is hereby granted, free of charge, to any person obtaining a +copy of this software and associated documentation files (the "Software"), +to deal in the Software without restriction, including without limitation +the rights to use, copy, modify, merge, publish, distribute, sublicense, +and/or sell copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN +ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the names of the authors or their +institutions shall not be used in advertising or otherwise to promote the +sale, use or other dealings in this Software without prior written +authorization from the authors. + +Copyright © 2006 Ian Osgood + +Permission is hereby granted, free of charge, to any person obtaining a +copy of this software and associated documentation files (the "Software"), +to deal in the Software without restriction, including without limitation +the rights to use, copy, modify, merge, publish, distribute, sublicense, +and/or sell copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN +ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the names of the authors or their +institutions shall not be used in advertising or otherwise to promote the +sale, use or other dealings in this Software without prior written +authorization from the authors. + +------------------------------------------------------------------------------ +Libxcb-icccm is redistributed within opencv-python non-headless Linux packages. +This license applies to Libxcb-icccm binary in the directory cv2/. + +Copyright © 2008-2011 Arnaud Fontaine +Copyright © 2007-2008 Vincent Torri + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated documentation +files (the "Software"), to deal in the Software without +restriction, including without limitation the rights to use, copy, +modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF +CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the names of the authors or +their institutions shall not be used in advertising or otherwise to +promote the sale, use or other dealings in this Software without +prior written authorization from the authors. + +------------------------------------------------------------------------------ +libXau is redistributed within opencv-python non-headless Linux packages. +This license applies to libXau binary in the directory cv2/. + +Copyright 1988, 1993, 1994, 1998 The Open Group + +Permission to use, copy, modify, distribute, and sell this software and its +documentation for any purpose is hereby granted without fee, provided that +the above copyright notice appear in all copies and that both that +copyright notice and this permission notice appear in supporting +documentation. + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +OPEN GROUP BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN +AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the name of The Open Group shall not be +used in advertising or otherwise to promote the sale, use or other dealings +in this Software without prior written authorization from The Open Group. + +------------------------------------------------------------------------------ +Vulkan headers are redistributed within all opencv-python packages. +This license applies to Vulkan headers in the directory 3rdparty/include/vulkan. + +Copyright (c) 2015-2018 The Khronos Group Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +------------------------------------------------------------------------------ +Libjpeg-turbo is redistributed within all opencv-python packages as build option. + +libjpeg-turbo Licenses +====================== + +libjpeg-turbo is covered by three compatible BSD-style open source licenses: + +- The IJG (Independent JPEG Group) License, which is listed in + [README.ijg](README.ijg) + + This license applies to the libjpeg API library and associated programs + (any code inherited from libjpeg, and any modifications to that code.) + +- The Modified (3-clause) BSD License, which is listed below + + This license covers the TurboJPEG API library and associated programs, as + well as the build system. + +- The [zlib License](https://opensource.org/licenses/Zlib) + + This license is a subset of the other two, and it covers the libjpeg-turbo + SIMD extensions. + + +Complying with the libjpeg-turbo Licenses +========================================= + +This section provides a roll-up of the libjpeg-turbo licensing terms, to the +best of our understanding. + +1. If you are distributing a modified version of the libjpeg-turbo source, + then: + + 1. You cannot alter or remove any existing copyright or license notices + from the source. + + **Origin** + - Clause 1 of the IJG License + - Clause 1 of the Modified BSD License + - Clauses 1 and 3 of the zlib License + + 2. You must add your own copyright notice to the header of each source + file you modified, so others can tell that you modified that file (if + there is not an existing copyright header in that file, then you can + simply add a notice stating that you modified the file.) + + **Origin** + - Clause 1 of the IJG License + - Clause 2 of the zlib License + + 3. You must include the IJG README file, and you must not alter any of the + copyright or license text in that file. + + **Origin** + - Clause 1 of the IJG License + +2. If you are distributing only libjpeg-turbo binaries without the source, or + if you are distributing an application that statically links with + libjpeg-turbo, then: + + 1. Your product documentation must include a message stating: + + This software is based in part on the work of the Independent JPEG + Group. + + **Origin** + - Clause 2 of the IJG license + + 2. If your binary distribution includes or uses the TurboJPEG API, then + your product documentation must include the text of the Modified BSD + License (see below.) + + **Origin** + - Clause 2 of the Modified BSD License + +3. You cannot use the name of the IJG or The libjpeg-turbo Project or the + contributors thereof in advertising, publicity, etc. + + **Origin** + - IJG License + - Clause 3 of the Modified BSD License + +4. The IJG and The libjpeg-turbo Project do not warrant libjpeg-turbo to be + free of defects, nor do we accept any liability for undesirable + consequences resulting from your use of the software. + + **Origin** + - IJG License + - Modified BSD License + - zlib License + + +The Modified (3-clause) BSD License +=================================== + +Copyright (C)2009-2022 D. R. Commander. All Rights Reserved.
+Copyright (C)2015 Viktor Szathmáry. All Rights Reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +- Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. +- Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. +- Neither the name of the libjpeg-turbo Project nor the names of its + contributors may be used to endorse or promote products derived from this + software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS", +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + + +Why Three Licenses? +=================== + +The zlib License could have been used instead of the Modified (3-clause) BSD +License, and since the IJG License effectively subsumes the distribution +conditions of the zlib License, this would have effectively placed +libjpeg-turbo binary distributions under the IJG License. However, the IJG +License specifically refers to the Independent JPEG Group and does not extend +attribution and endorsement protections to other entities. Thus, it was +desirable to choose a license that granted us the same protections for new code +that were granted to the IJG for code derived from their software. + +------------------------------------------------------------------------------ +Libspng is redistributed within all opencv-python packages as build option. + +BSD 2-Clause License + +Copyright (c) 2018-2022, Randy +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +QUIRC library is redistributed within all opencv-python packages. + +quirc -- QR-code recognition library +Copyright (C) 2010-2012 Daniel Beer + +Permission to use, copy, modify, and/or distribute this software for +any purpose with or without fee is hereby granted, provided that the +above copyright notice and this permission notice appear in all +copies. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL +WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE +AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL +DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR +PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER +TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR +PERFORMANCE OF THIS SOFTWARE. + +------------------------------------------------------------------------------ +Flatbuffers library is redistributed within all opencv-python packages. + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +------------------------------------------------------------------------------ +Protobuf library is redistributed within all opencv-python packages. + +Copyright 2008 Google Inc. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Code generated by the Protocol Buffer compiler is owned by the owner +of the input file used when generating it. This code is not +standalone and requires a support library to be linked with it. This +support library is itself covered by the above license. + +------------------------------------------------------------------------------ +OpenJPEG library is redistributed within all opencv-python packages. + +/* + * The copyright in this software is being made available under the 2-clauses + * BSD License, included below. This software may be subject to other third + * party and contributor rights, including patent rights, and no such rights + * are granted under this license. + * + * Copyright (c) 2002-2014, Universite catholique de Louvain (UCL), Belgium + * Copyright (c) 2002-2014, Professor Benoit Macq + * Copyright (c) 2003-2014, Antonin Descampe + * Copyright (c) 2003-2009, Francois-Olivier Devaux + * Copyright (c) 2005, Herve Drolon, FreeImage Team + * Copyright (c) 2002-2003, Yannick Verschueren + * Copyright (c) 2001-2003, David Janssens + * Copyright (c) 2011-2012, Centre National d'Etudes Spatiales (CNES), France + * Copyright (c) 2012, CS Systemes d'Information, France + * + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS `AS IS' + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ + +------------------------------------------------------------------------------ +TIFF library is redistributed within all opencv-python packages. + +Copyright (c) 1988-1997 Sam Leffler +Copyright (c) 1991-1997 Silicon Graphics, Inc. + +Permission to use, copy, modify, distribute, and sell this software and +its documentation for any purpose is hereby granted without fee, provided +that (i) the above copyright notices and this permission notice appear in +all copies of the software and related documentation, and (ii) the names of +Sam Leffler and Silicon Graphics may not be used in any advertising or +publicity relating to the software without the specific, prior written +permission of Sam Leffler and Silicon Graphics. + +THE SOFTWARE IS PROVIDED "AS-IS" AND WITHOUT WARRANTY OF ANY KIND, +EXPRESS, IMPLIED OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY +WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. + +IN NO EVENT SHALL SAM LEFFLER OR SILICON GRAPHICS BE LIABLE FOR +ANY SPECIAL, INCIDENTAL, INDIRECT OR CONSEQUENTIAL DAMAGES OF ANY KIND, +OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, +WHETHER OR NOT ADVISED OF THE POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF +LIABILITY, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE +OF THIS SOFTWARE. + +------------------------------------------------------------------------------ +OpenEXR library is redistributed within all opencv-python packages. + +Copyright (c) 2006, Industrial Light & Magic, a division of Lucasfilm +Entertainment Company Ltd. Portions contributed and copyright held by +others as indicated. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above + copyright notice, this list of conditions and the following + disclaimer. + + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided with + the distribution. + + * Neither the name of Industrial Light & Magic nor the names of + any other contributors to this software may be used to endorse or + promote products derived from this software without specific prior + written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS +IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +Intel(R) IPP ICV library statically linked within x86 and x86_64 opencv-python packages. + +Intel(R) Integrated Performance Primitives 2021 Update 10 + +Intel Simplified Software License (Version October 2022) + +Intel(R) Integrated Performance Primitives (Intel(R) IPP) : Copyright (C) 1997 Intel Corporation + +Use and Redistribution. You may use and redistribute the software, which is +provided in binary form only, (the "Software"), without modification, +provided the following conditions are met: + +* Redistributions must reproduce the above copyright notice and these + terms of use in the Software and in the documentation and/or other materials + provided with the distribution. +* Neither the name of Intel nor the names of its suppliers may be used to + endorse or promote products derived from this Software without specific + prior written permission. +* No reverse engineering, decompilation, or disassembly of the Software is + permitted, nor any modification or alteration of the Software or its operation + at any time, including during execution. + +No other licenses. Except as provided in the preceding section, Intel grants no +licenses or other rights by implication, estoppel or otherwise to, patent, +copyright, trademark, trade name, service mark or other intellectual property +licenses or rights of Intel. + +Third party software. "Third Party Software" means the files (if any) listed +in the "third-party-software.txt" or other similarly-named text file that may +be included with the Software. Third Party Software, even if included with the +distribution of the Software, may be governed by separate license terms, including +without limitation, third party license terms, open source software notices and +terms, and/or other Intel software license terms. These separate license terms +solely govern Your use of the Third Party Software. + +DISCLAIMER. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY EXPRESS OR IMPLIED +WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT ARE +DISCLAIMED. THIS SOFTWARE IS NOT INTENDED FOR USE IN SYSTEMS OR APPLICATIONS +WHERE FAILURE OF THE SOFTWARE MAY CAUSE PERSONAL INJURY OR DEATH AND YOU AGREE +THAT YOU ARE FULLY RESPONSIBLE FOR ANY CLAIMS, COSTS, DAMAGES, EXPENSES, AND +ATTORNEYS' FEES ARISING OUT OF ANY SUCH USE, EVEN IF ANY CLAIM ALLEGES THAT +INTEL WAS NEGLIGENT REGARDING THE DESIGN OR MANUFACTURE OF THE SOFTWARE. + +LIMITATION OF LIABILITY. IN NO EVENT WILL INTEL BE LIABLE FOR ANY DIRECT, +INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, +BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE +OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF +ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +No support. Intel may make changes to the Software, at any time without notice, +and is not obligated to support, update or provide training for the Software. + +Termination. Your right to use the Software is terminated in the event of your +breach of this license. + +Feedback. Should you provide Intel with comments, modifications, corrections, +enhancements or other input ("Feedback") related to the Software, Intel will be +free to use, disclose, reproduce, license or otherwise distribute or exploit the +Feedback in its sole discretion without any obligations or restrictions of any +kind, including without limitation, intellectual property rights or licensing +obligations. + +Compliance with laws. You agree to comply with all relevant laws and regulations +governing your use, transfer, import or export (or prohibition thereof) of the +Software. + +Governing law. All disputes will be governed by the laws of the United States of +America and the State of Delaware without reference to conflict of law +principles and subject to the exclusive jurisdiction of the state or federal +courts sitting in the State of Delaware, and each party agrees that it submits +to the personal jurisdiction and venue of those courts and waives any +objections. THE UNITED NATIONS CONVENTION ON CONTRACTS FOR THE INTERNATIONAL +SALE OF GOODS (1980) IS SPECIFICALLY EXCLUDED AND WILL NOT APPLY TO THE SOFTWARE. + +------------------------------------------------------------------------------ +Orbbec SDK distributed with arm64 MacOS packages. + +MIT License + +Copyright (c) 2023 OrbbecDeveloper + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +------------------------------------------------------------------------------ + +libavif library and it's dependnecies are redistributed within all opencv-python packages. + +Copyright 2019 Joe Drago. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this +list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, +this list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ + +Files: src/obu.c + +Copyright © 2018-2019, VideoLAN and dav1d authors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ + +Files: third_party/iccjpeg/* + +In plain English: + +1. We don't promise that this software works. (But if you find any bugs, + please let us know!) +2. You can use this software for whatever you want. You don't have to pay us. +3. You may not pretend that you wrote this software. If you use it in a + program, you must acknowledge somewhere in your documentation that + you've used the IJG code. + +In legalese: + +The authors make NO WARRANTY or representation, either express or implied, +with respect to this software, its quality, accuracy, merchantability, or +fitness for a particular purpose. This software is provided "AS IS", and you, +its user, assume the entire risk as to its quality and accuracy. + +This software is copyright (C) 1991-2013, Thomas G. Lane, Guido Vollbeding. +All Rights Reserved except as specified below. + +Permission is hereby granted to use, copy, modify, and distribute this +software (or portions thereof) for any purpose, without fee, subject to these +conditions: +(1) If any part of the source code for this software is distributed, then this +README file must be included, with this copyright and no-warranty notice +unaltered; and any additions, deletions, or changes to the original files +must be clearly indicated in accompanying documentation. +(2) If only executable code is distributed, then the accompanying +documentation must state that "this software is based in part on the work of +the Independent JPEG Group". +(3) Permission for use of this software is granted only if the user accepts +full responsibility for any undesirable consequences; the authors accept +NO LIABILITY for damages of any kind. + +These conditions apply to any software derived from or based on the IJG code, +not just to the unmodified library. If you use our work, you ought to +acknowledge us. + +Permission is NOT granted for the use of any IJG author's name or company name +in advertising or publicity relating to this software or products derived from +it. This software may be referred to only as "the Independent JPEG Group's +software". + +We specifically permit and encourage the use of this software as the basis of +commercial products, provided that all warranty or liability claims are +assumed by the product vendor. + + +The Unix configuration script "configure" was produced with GNU Autoconf. +It is copyright by the Free Software Foundation but is freely distributable. +The same holds for its supporting scripts (config.guess, config.sub, +ltmain.sh). Another support script, install-sh, is copyright by X Consortium +but is also freely distributable. + +The IJG distribution formerly included code to read and write GIF files. +To avoid entanglement with the Unisys LZW patent, GIF reading support has +been removed altogether, and the GIF writer has been simplified to produce +"uncompressed GIFs". This technique does not use the LZW algorithm; the +resulting GIF files are larger than usual, but are readable by all standard +GIF decoders. + +We are required to state that + "The Graphics Interchange Format(c) is the Copyright property of + CompuServe Incorporated. GIF(sm) is a Service Mark property of + CompuServe Incorporated." + +------------------------------------------------------------------------------ + +Files: contrib/gdk-pixbuf/* + +Copyright 2020 Emmanuel Gil Peyrot. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this +list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, +this list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ + +Files: android_jni/gradlew* + + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +------------------------------------------------------------------------------ + +Files: third_party/libyuv/* + +Copyright 2011 The LibYuv Project Authors. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + + * Neither the name of Google nor the names of its contributors may + be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ + +aom library and it's dependnecies are redistributed within all opencv-python packages. + +Copyright (c) 2016, Alliance for Open Media. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS +FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE +COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, +BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN +ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ + +KAZE Features library is redistributed within all opencv-python packages. + +Copyright (c) 2012, Pablo Fernández Alcantarilla +All Rights Reserved + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + * Neither the name of the copyright holders nor the names of its contributors + may be used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY +EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES +OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT +SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR +BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY +WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ + +AKAZE Features library is redistributed within all opencv-python packages. + +Copyright (c) 2014, Pablo Fernandez Alcantarilla, Jesus Nuevo +All Rights Reserved + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + * Neither the name of the copyright holders nor the names of its contributors + may be used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY +EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES +OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT +SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR +BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY +WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +opencv-python +Apache Software License +https://github.com/opencv/opencv-python +OpenCV library is redistributed within opencv-python package. +This license applies to OpenCV binary in the directory cv2/. + + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +------------------------------------------------------------------------------ +libvpx is redistributed within all opencv-python Linux packages. +This license applies to libvpx binary in the directory cv2/. + +Copyright (c) 2010, The WebM Project authors. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + + * Neither the name of Google, nor the WebM Project, nor the names + of its contributors may be used to endorse or promote products + derived from this software without specific prior written + permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +FFmpeg is redistributed within all opencv-python packages. + +Libbluray, libgnutls, libnettle, libhogweed, libintl, libmp3lame, libp11, +librtmp, libsoxr and libtasn1 are redistributed within all opencv-python macOS packages. + +This license applies to the above library binaries in the directory cv2/. + + GNU LESSER GENERAL PUBLIC LICENSE + Version 2.1, February 1999 + + Copyright (C) 1991, 1999 Free Software Foundation, Inc. + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + +[This is the first released version of the Lesser GPL. It also counts + as the successor of the GNU Library Public License, version 2, hence + the version number 2.1.] + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +Licenses are intended to guarantee your freedom to share and change +free software--to make sure the software is free for all its users. + + This license, the Lesser General Public License, applies to some +specially designated software packages--typically libraries--of the +Free Software Foundation and other authors who decide to use it. You +can use it too, but we suggest you first think carefully about whether +this license or the ordinary General Public License is the better +strategy to use in any particular case, based on the explanations below. + + When we speak of free software, we are referring to freedom of use, +not price. Our General Public Licenses are designed to make sure that +you have the freedom to distribute copies of free software (and charge +for this service if you wish); that you receive source code or can get +it if you want it; that you can change the software and use pieces of +it in new free programs; and that you are informed that you can do +these things. + + To protect your rights, we need to make restrictions that forbid +distributors to deny you these rights or to ask you to surrender these +rights. These restrictions translate to certain responsibilities for +you if you distribute copies of the library or if you modify it. + + For example, if you distribute copies of the library, whether gratis +or for a fee, you must give the recipients all the rights that we gave +you. You must make sure that they, too, receive or can get the source +code. If you link other code with the library, you must provide +complete object files to the recipients, so that they can relink them +with the library after making changes to the library and recompiling +it. And you must show them these terms so they know their rights. + + We protect your rights with a two-step method: (1) we copyright the +library, and (2) we offer you this license, which gives you legal +permission to copy, distribute and/or modify the library. + + To protect each distributor, we want to make it very clear that +there is no warranty for the free library. Also, if the library is +modified by someone else and passed on, the recipients should know +that what they have is not the original version, so that the original +author's reputation will not be affected by problems that might be +introduced by others. + + Finally, software patents pose a constant threat to the existence of +any free program. We wish to make sure that a company cannot +effectively restrict the users of a free program by obtaining a +restrictive license from a patent holder. Therefore, we insist that +any patent license obtained for a version of the library must be +consistent with the full freedom of use specified in this license. + + Most GNU software, including some libraries, is covered by the +ordinary GNU General Public License. This license, the GNU Lesser +General Public License, applies to certain designated libraries, and +is quite different from the ordinary General Public License. We use +this license for certain libraries in order to permit linking those +libraries into non-free programs. + + When a program is linked with a library, whether statically or using +a shared library, the combination of the two is legally speaking a +combined work, a derivative of the original library. The ordinary +General Public License therefore permits such linking only if the +entire combination fits its criteria of freedom. The Lesser General +Public License permits more lax criteria for linking other code with +the library. + + We call this license the "Lesser" General Public License because it +does Less to protect the user's freedom than the ordinary General +Public License. It also provides other free software developers Less +of an advantage over competing non-free programs. These disadvantages +are the reason we use the ordinary General Public License for many +libraries. However, the Lesser license provides advantages in certain +special circumstances. + + For example, on rare occasions, there may be a special need to +encourage the widest possible use of a certain library, so that it becomes +a de-facto standard. To achieve this, non-free programs must be +allowed to use the library. A more frequent case is that a free +library does the same job as widely used non-free libraries. In this +case, there is little to gain by limiting the free library to free +software only, so we use the Lesser General Public License. + + In other cases, permission to use a particular library in non-free +programs enables a greater number of people to use a large body of +free software. For example, permission to use the GNU C Library in +non-free programs enables many more people to use the whole GNU +operating system, as well as its variant, the GNU/Linux operating +system. + + Although the Lesser General Public License is Less protective of the +users' freedom, it does ensure that the user of a program that is +linked with the Library has the freedom and the wherewithal to run +that program using a modified version of the Library. + + The precise terms and conditions for copying, distribution and +modification follow. Pay close attention to the difference between a +"work based on the library" and a "work that uses the library". The +former contains code derived from the library, whereas the latter must +be combined with the library in order to run. + + GNU LESSER GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License Agreement applies to any software library or other +program which contains a notice placed by the copyright holder or +other authorized party saying it may be distributed under the terms of +this Lesser General Public License (also called "this License"). +Each licensee is addressed as "you". + + A "library" means a collection of software functions and/or data +prepared so as to be conveniently linked with application programs +(which use some of those functions and data) to form executables. + + The "Library", below, refers to any such software library or work +which has been distributed under these terms. A "work based on the +Library" means either the Library or any derivative work under +copyright law: that is to say, a work containing the Library or a +portion of it, either verbatim or with modifications and/or translated +straightforwardly into another language. (Hereinafter, translation is +included without limitation in the term "modification".) + + "Source code" for a work means the preferred form of the work for +making modifications to it. For a library, complete source code means +all the source code for all modules it contains, plus any associated +interface definition files, plus the scripts used to control compilation +and installation of the library. + + Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running a program using the Library is not restricted, and output from +such a program is covered only if its contents constitute a work based +on the Library (independent of the use of the Library in a tool for +writing it). Whether that is true depends on what the Library does +and what the program that uses the Library does. + + 1. You may copy and distribute verbatim copies of the Library's +complete source code as you receive it, in any medium, provided that +you conspicuously and appropriately publish on each copy an +appropriate copyright notice and disclaimer of warranty; keep intact +all the notices that refer to this License and to the absence of any +warranty; and distribute a copy of this License along with the +Library. + + You may charge a fee for the physical act of transferring a copy, +and you may at your option offer warranty protection in exchange for a +fee. + + 2. You may modify your copy or copies of the Library or any portion +of it, thus forming a work based on the Library, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) The modified work must itself be a software library. + + b) You must cause the files modified to carry prominent notices + stating that you changed the files and the date of any change. + + c) You must cause the whole of the work to be licensed at no + charge to all third parties under the terms of this License. + + d) If a facility in the modified Library refers to a function or a + table of data to be supplied by an application program that uses + the facility, other than as an argument passed when the facility + is invoked, then you must make a good faith effort to ensure that, + in the event an application does not supply such function or + table, the facility still operates, and performs whatever part of + its purpose remains meaningful. + + (For example, a function in a library to compute square roots has + a purpose that is entirely well-defined independent of the + application. Therefore, Subsection 2d requires that any + application-supplied function or table used by this function must + be optional: if the application does not supply it, the square + root function must still compute square roots.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Library, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Library, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote +it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Library. + +In addition, mere aggregation of another work not based on the Library +with the Library (or with a work based on the Library) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may opt to apply the terms of the ordinary GNU General Public +License instead of this License to a given copy of the Library. To do +this, you must alter all the notices that refer to this License, so +that they refer to the ordinary GNU General Public License, version 2, +instead of to this License. (If a newer version than version 2 of the +ordinary GNU General Public License has appeared, then you can specify +that version instead if you wish.) Do not make any other change in +these notices. + + Once this change is made in a given copy, it is irreversible for +that copy, so the ordinary GNU General Public License applies to all +subsequent copies and derivative works made from that copy. + + This option is useful when you wish to copy part of the code of +the Library into a program that is not a library. + + 4. You may copy and distribute the Library (or a portion or +derivative of it, under Section 2) in object code or executable form +under the terms of Sections 1 and 2 above provided that you accompany +it with the complete corresponding machine-readable source code, which +must be distributed under the terms of Sections 1 and 2 above on a +medium customarily used for software interchange. + + If distribution of object code is made by offering access to copy +from a designated place, then offering equivalent access to copy the +source code from the same place satisfies the requirement to +distribute the source code, even though third parties are not +compelled to copy the source along with the object code. + + 5. A program that contains no derivative of any portion of the +Library, but is designed to work with the Library by being compiled or +linked with it, is called a "work that uses the Library". Such a +work, in isolation, is not a derivative work of the Library, and +therefore falls outside the scope of this License. + + However, linking a "work that uses the Library" with the Library +creates an executable that is a derivative of the Library (because it +contains portions of the Library), rather than a "work that uses the +library". The executable is therefore covered by this License. +Section 6 states terms for distribution of such executables. + + When a "work that uses the Library" uses material from a header file +that is part of the Library, the object code for the work may be a +derivative work of the Library even though the source code is not. +Whether this is true is especially significant if the work can be +linked without the Library, or if the work is itself a library. The +threshold for this to be true is not precisely defined by law. + + If such an object file uses only numerical parameters, data +structure layouts and accessors, and small macros and small inline +functions (ten lines or less in length), then the use of the object +file is unrestricted, regardless of whether it is legally a derivative +work. (Executables containing this object code plus portions of the +Library will still fall under Section 6.) + + Otherwise, if the work is a derivative of the Library, you may +distribute the object code for the work under the terms of Section 6. +Any executables containing that work also fall under Section 6, +whether or not they are linked directly with the Library itself. + + 6. As an exception to the Sections above, you may also combine or +link a "work that uses the Library" with the Library to produce a +work containing portions of the Library, and distribute that work +under terms of your choice, provided that the terms permit +modification of the work for the customer's own use and reverse +engineering for debugging such modifications. + + You must give prominent notice with each copy of the work that the +Library is used in it and that the Library and its use are covered by +this License. You must supply a copy of this License. If the work +during execution displays copyright notices, you must include the +copyright notice for the Library among them, as well as a reference +directing the user to the copy of this License. Also, you must do one +of these things: + + a) Accompany the work with the complete corresponding + machine-readable source code for the Library including whatever + changes were used in the work (which must be distributed under + Sections 1 and 2 above); and, if the work is an executable linked + with the Library, with the complete machine-readable "work that + uses the Library", as object code and/or source code, so that the + user can modify the Library and then relink to produce a modified + executable containing the modified Library. (It is understood + that the user who changes the contents of definitions files in the + Library will not necessarily be able to recompile the application + to use the modified definitions.) + + b) Use a suitable shared library mechanism for linking with the + Library. A suitable mechanism is one that (1) uses at run time a + copy of the library already present on the user's computer system, + rather than copying library functions into the executable, and (2) + will operate properly with a modified version of the library, if + the user installs one, as long as the modified version is + interface-compatible with the version that the work was made with. + + c) Accompany the work with a written offer, valid for at + least three years, to give the same user the materials + specified in Subsection 6a, above, for a charge no more + than the cost of performing this distribution. + + d) If distribution of the work is made by offering access to copy + from a designated place, offer equivalent access to copy the above + specified materials from the same place. + + e) Verify that the user has already received a copy of these + materials or that you have already sent this user a copy. + + For an executable, the required form of the "work that uses the +Library" must include any data and utility programs needed for +reproducing the executable from it. However, as a special exception, +the materials to be distributed need not include anything that is +normally distributed (in either source or binary form) with the major +components (compiler, kernel, and so on) of the operating system on +which the executable runs, unless that component itself accompanies +the executable. + + It may happen that this requirement contradicts the license +restrictions of other proprietary libraries that do not normally +accompany the operating system. Such a contradiction means you cannot +use both them and the Library together in an executable that you +distribute. + + 7. You may place library facilities that are a work based on the +Library side-by-side in a single library together with other library +facilities not covered by this License, and distribute such a combined +library, provided that the separate distribution of the work based on +the Library and of the other library facilities is otherwise +permitted, and provided that you do these two things: + + a) Accompany the combined library with a copy of the same work + based on the Library, uncombined with any other library + facilities. This must be distributed under the terms of the + Sections above. + + b) Give prominent notice with the combined library of the fact + that part of it is a work based on the Library, and explaining + where to find the accompanying uncombined form of the same work. + + 8. You may not copy, modify, sublicense, link with, or distribute +the Library except as expressly provided under this License. Any +attempt otherwise to copy, modify, sublicense, link with, or +distribute the Library is void, and will automatically terminate your +rights under this License. However, parties who have received copies, +or rights, from you under this License will not have their licenses +terminated so long as such parties remain in full compliance. + + 9. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Library or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Library (or any work based on the +Library), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Library or works based on it. + + 10. Each time you redistribute the Library (or any work based on the +Library), the recipient automatically receives a license from the +original licensor to copy, distribute, link with or modify the Library +subject to these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties with +this License. + + 11. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Library at all. For example, if a patent +license would not permit royalty-free redistribution of the Library by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Library. + +If any portion of this section is held invalid or unenforceable under any +particular circumstance, the balance of the section is intended to apply, +and the section as a whole is intended to apply in other circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 12. If the distribution and/or use of the Library is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Library under this License may add +an explicit geographical distribution limitation excluding those countries, +so that distribution is permitted only in or among countries not thus +excluded. In such case, this License incorporates the limitation as if +written in the body of this License. + + 13. The Free Software Foundation may publish revised and/or new +versions of the Lesser General Public License from time to time. +Such new versions will be similar in spirit to the present version, +but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Library +specifies a version number of this License which applies to it and +"any later version", you have the option of following the terms and +conditions either of that version or of any later version published by +the Free Software Foundation. If the Library does not specify a +license version number, you may choose any version ever published by +the Free Software Foundation. + + 14. If you wish to incorporate parts of the Library into other free +programs whose distribution conditions are incompatible with these, +write to the author to ask for permission. For software which is +copyrighted by the Free Software Foundation, write to the Free +Software Foundation; we sometimes make exceptions for this. Our +decision will be guided by the two goals of preserving the free status +of all derivatives of our free software and of promoting the sharing +and reuse of software generally. + + NO WARRANTY + + 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO +WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. +EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR +OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY +KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE +LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME +THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN +WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY +AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU +FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR +CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE +LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING +RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A +FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF +SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH +DAMAGES. + + END OF TERMS AND CONDITIONS + +------------------------------------------------------------------------------ +Qt 5 is redistributed within non-headless opencv-python Linux and macOS packages. +libgmp is redistributed within opencv-python macOS packages. +libidn2 is redistributed within opencv-python macOS packages. +libunistring is redistributed within opencv-python macOS packages. +This license applies to the above binaries in the directory cv2/. + + GNU LESSER GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + This version of the GNU Lesser General Public License incorporates +the terms and conditions of version 3 of the GNU General Public +License, supplemented by the additional permissions listed below. + + 0. Additional Definitions. + + As used herein, "this License" refers to version 3 of the GNU Lesser +General Public License, and the "GNU GPL" refers to version 3 of the GNU +General Public License. + + "The Library" refers to a covered work governed by this License, +other than an Application or a Combined Work as defined below. + + An "Application" is any work that makes use of an interface provided +by the Library, but which is not otherwise based on the Library. +Defining a subclass of a class defined by the Library is deemed a mode +of using an interface provided by the Library. + + A "Combined Work" is a work produced by combining or linking an +Application with the Library. The particular version of the Library +with which the Combined Work was made is also called the "Linked +Version". + + The "Minimal Corresponding Source" for a Combined Work means the +Corresponding Source for the Combined Work, excluding any source code +for portions of the Combined Work that, considered in isolation, are +based on the Application, and not on the Linked Version. + + The "Corresponding Application Code" for a Combined Work means the +object code and/or source code for the Application, including any data +and utility programs needed for reproducing the Combined Work from the +Application, but excluding the System Libraries of the Combined Work. + + 1. Exception to Section 3 of the GNU GPL. + + You may convey a covered work under sections 3 and 4 of this License +without being bound by section 3 of the GNU GPL. + + 2. Conveying Modified Versions. + + If you modify a copy of the Library, and, in your modifications, a +facility refers to a function or data to be supplied by an Application +that uses the facility (other than as an argument passed when the +facility is invoked), then you may convey a copy of the modified +version: + + a) under this License, provided that you make a good faith effort to + ensure that, in the event an Application does not supply the + function or data, the facility still operates, and performs + whatever part of its purpose remains meaningful, or + + b) under the GNU GPL, with none of the additional permissions of + this License applicable to that copy. + + 3. Object Code Incorporating Material from Library Header Files. + + The object code form of an Application may incorporate material from +a header file that is part of the Library. You may convey such object +code under terms of your choice, provided that, if the incorporated +material is not limited to numerical parameters, data structure +layouts and accessors, or small macros, inline functions and templates +(ten or fewer lines in length), you do both of the following: + + a) Give prominent notice with each copy of the object code that the + Library is used in it and that the Library and its use are + covered by this License. + + b) Accompany the object code with a copy of the GNU GPL and this license + document. + + 4. Combined Works. + + You may convey a Combined Work under terms of your choice that, +taken together, effectively do not restrict modification of the +portions of the Library contained in the Combined Work and reverse +engineering for debugging such modifications, if you also do each of +the following: + + a) Give prominent notice with each copy of the Combined Work that + the Library is used in it and that the Library and its use are + covered by this License. + + b) Accompany the Combined Work with a copy of the GNU GPL and this license + document. + + c) For a Combined Work that displays copyright notices during + execution, include the copyright notice for the Library among + these notices, as well as a reference directing the user to the + copies of the GNU GPL and this license document. + + d) Do one of the following: + + 0) Convey the Minimal Corresponding Source under the terms of this + License, and the Corresponding Application Code in a form + suitable for, and under terms that permit, the user to + recombine or relink the Application with a modified version of + the Linked Version to produce a modified Combined Work, in the + manner specified by section 6 of the GNU GPL for conveying + Corresponding Source. + + 1) Use a suitable shared library mechanism for linking with the + Library. A suitable mechanism is one that (a) uses at run time + a copy of the Library already present on the user's computer + system, and (b) will operate properly with a modified version + of the Library that is interface-compatible with the Linked + Version. + + e) Provide Installation Information, but only if you would otherwise + be required to provide such information under section 6 of the + GNU GPL, and only to the extent that such information is + necessary to install and execute a modified version of the + Combined Work produced by recombining or relinking the + Application with a modified version of the Linked Version. (If + you use option 4d0, the Installation Information must accompany + the Minimal Corresponding Source and Corresponding Application + Code. If you use option 4d1, you must provide the Installation + Information in the manner specified by section 6 of the GNU GPL + for conveying Corresponding Source.) + + 5. Combined Libraries. + + You may place library facilities that are a work based on the +Library side by side in a single library together with other library +facilities that are not Applications and are not covered by this +License, and convey such a combined library under terms of your +choice, if you do both of the following: + + a) Accompany the combined library with a copy of the same work based + on the Library, uncombined with any other library facilities, + conveyed under the terms of this License. + + b) Give prominent notice with the combined library that part of it + is a work based on the Library, and explaining where to find the + accompanying uncombined form of the same work. + + 6. Revised Versions of the GNU Lesser General Public License. + + The Free Software Foundation may publish revised and/or new versions +of the GNU Lesser General Public License from time to time. Such new +versions will be similar in spirit to the present version, but may +differ in detail to address new problems or concerns. + + Each version is given a distinguishing version number. If the +Library as you received it specifies that a certain numbered version +of the GNU Lesser General Public License "or any later version" +applies to it, you have the option of following the terms and +conditions either of that published version or of any later version +published by the Free Software Foundation. If the Library as you +received it does not specify a version number of the GNU Lesser +General Public License, you may choose any version of the GNU Lesser +General Public License ever published by the Free Software Foundation. + + If the Library as you received it specifies that a proxy can decide +whether future versions of the GNU Lesser General Public License shall +apply, that proxy's public statement of acceptance of any version is +permanent authorization for you to choose that version for the +Library. + +------------------------------------------------------------------------------ +bzip2 is redistributed within all opencv-python Linux packages. +This license applies to libbz2 binary in the directory cv2/. + +This program, "bzip2", the associated library "libbzip2", and all +documentation, are copyright (C) 1996-2010 Julian R Seward. All +rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. The origin of this software must not be misrepresented; you must + not claim that you wrote the original software. If you use this + software in a product, an acknowledgment in the product + documentation would be appreciated but is not required. + +3. Altered source versions must be plainly marked as such, and must + not be misrepresented as being the original software. + +4. The name of the author may not be used to endorse or promote + products derived from this software without specific prior written + permission. + +THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS +OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE +GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, +WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Julian Seward, jseward@bzip.org +bzip2/libbzip2 version 1.0.6 of 6 September 2010 + +------------------------------------------------------------------------------ +libcrypto and libssl are redistributed within all opencv-python Linux and macOS packages. +libopencore-amrnb and libopencore-amrwb are redistributed within all opencv-python Linux and macOS packages. +This license applies to above binaries in the directory cv2/. + + LICENSE ISSUES + ============== + + The OpenSSL toolkit stays under a double license, i.e. both the conditions of + the OpenSSL License and the original SSLeay license apply to the toolkit. + See below for the actual license texts. + + OpenSSL License + --------------- + +/* ==================================================================== + * Copyright (c) 1998-2019 The OpenSSL Project. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * 3. All advertising materials mentioning features or use of this + * software must display the following acknowledgment: + * "This product includes software developed by the OpenSSL Project + * for use in the OpenSSL Toolkit. (http://www.openssl.org/)" + * + * 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to + * endorse or promote products derived from this software without + * prior written permission. For written permission, please contact + * openssl-core@openssl.org. + * + * 5. Products derived from this software may not be called "OpenSSL" + * nor may "OpenSSL" appear in their names without prior written + * permission of the OpenSSL Project. + * + * 6. Redistributions of any form whatsoever must retain the following + * acknowledgment: + * "This product includes software developed by the OpenSSL Project + * for use in the OpenSSL Toolkit (http://www.openssl.org/)" + * + * THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY + * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR + * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED + * OF THE POSSIBILITY OF SUCH DAMAGE. + * ==================================================================== + * + * This product includes cryptographic software written by Eric Young + * (eay@cryptsoft.com). This product includes software written by Tim + * Hudson (tjh@cryptsoft.com). + * + */ + + Original SSLeay License + ----------------------- + +/* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com) + * All rights reserved. + * + * This package is an SSL implementation written + * by Eric Young (eay@cryptsoft.com). + * The implementation was written so as to conform with Netscapes SSL. + * + * This library is free for commercial and non-commercial use as long as + * the following conditions are adhered to. The following conditions + * apply to all code found in this distribution, be it the RC4, RSA, + * lhash, DES, etc., code; not just the SSL code. The SSL documentation + * included with this distribution is covered by the same copyright terms + * except that the holder is Tim Hudson (tjh@cryptsoft.com). + * + * Copyright remains Eric Young's, and as such any Copyright notices in + * the code are not to be removed. + * If this package is used in a product, Eric Young should be given attribution + * as the author of the parts of the library used. + * This can be in the form of a textual message at program startup or + * in documentation (online or textual) provided with the package. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. All advertising materials mentioning features or use of this software + * must display the following acknowledgement: + * "This product includes cryptographic software written by + * Eric Young (eay@cryptsoft.com)" + * The word 'cryptographic' can be left out if the routines from the library + * being used are not cryptographic related :-). + * 4. If you include any Windows specific code (or a derivative thereof) from + * the apps directory (application code) you must include an acknowledgement: + * "This product includes software written by Tim Hudson (tjh@cryptsoft.com)" + * + * THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * The licence and distribution terms for any publicly available version or + * derivative of this code cannot be changed. i.e. this code cannot simply be + * copied and put under another distribution licence + * [including the GNU Public Licence.] + */ + +------------------------------------------------------------------------------ +libfontconfig is redistributed within all opencv-python macOS packages. +This license applies to libfontconfig binary in the directory cv2/. + +Copyright © 2000,2001,2002,2003,2004,2006,2007 Keith Packard +Copyright © 2005 Patrick Lam +Copyright © 2009 Roozbeh Pournader +Copyright © 2008,2009 Red Hat, Inc. +Copyright © 2008 Danilo Šegan +Copyright © 2012 Google, Inc. + + +Permission to use, copy, modify, distribute, and sell this software and its +documentation for any purpose is hereby granted without fee, provided that +the above copyright notice appear in all copies and that both that +copyright notice and this permission notice appear in supporting +documentation, and that the name of the author(s) not be used in +advertising or publicity pertaining to distribution of the software without +specific, written prior permission. The authors make no +representations about the suitability of this software for any purpose. It +is provided "as is" without express or implied warranty. + +THE AUTHOR(S) DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, +INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO +EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY SPECIAL, INDIRECT OR +CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, +DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER +TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR +PERFORMANCE OF THIS SOFTWARE. + +------------------------------------------------------------------------------ +libfreetype is redistributed within opencv-python Linux and macOS packages. +This license applies to libfreetype binary in the directory cv2/. + + The FreeType Project LICENSE + ---------------------------- + + 2006-Jan-27 + + Copyright 1996-2002, 2006 by + David Turner, Robert Wilhelm, and Werner Lemberg + + + +Introduction +============ + + The FreeType Project is distributed in several archive packages; + some of them may contain, in addition to the FreeType font engine, + various tools and contributions which rely on, or relate to, the + FreeType Project. + + This license applies to all files found in such packages, and + which do not fall under their own explicit license. The license + affects thus the FreeType font engine, the test programs, + documentation and makefiles, at the very least. + + This license was inspired by the BSD, Artistic, and IJG + (Independent JPEG Group) licenses, which all encourage inclusion + and use of free software in commercial and freeware products + alike. As a consequence, its main points are that: + + o We don't promise that this software works. However, we will be + interested in any kind of bug reports. (`as is' distribution) + + o You can use this software for whatever you want, in parts or + full form, without having to pay us. (`royalty-free' usage) + + o You may not pretend that you wrote this software. If you use + it, or only parts of it, in a program, you must acknowledge + somewhere in your documentation that you have used the + FreeType code. (`credits') + + We specifically permit and encourage the inclusion of this + software, with or without modifications, in commercial products. + We disclaim all warranties covering The FreeType Project and + assume no liability related to The FreeType Project. + + + Finally, many people asked us for a preferred form for a + credit/disclaimer to use in compliance with this license. We thus + encourage you to use the following text: + + """ + Portions of this software are copyright © The FreeType + Project (www.freetype.org). All rights reserved. + """ + + Please replace with the value from the FreeType version you + actually use. + + +Legal Terms +=========== + +0. Definitions +-------------- + + Throughout this license, the terms `package', `FreeType Project', + and `FreeType archive' refer to the set of files originally + distributed by the authors (David Turner, Robert Wilhelm, and + Werner Lemberg) as the `FreeType Project', be they named as alpha, + beta or final release. + + `You' refers to the licensee, or person using the project, where + `using' is a generic term including compiling the project's source + code as well as linking it to form a `program' or `executable'. + This program is referred to as `a program using the FreeType + engine'. + + This license applies to all files distributed in the original + FreeType Project, including all source code, binaries and + documentation, unless otherwise stated in the file in its + original, unmodified form as distributed in the original archive. + If you are unsure whether or not a particular file is covered by + this license, you must contact us to verify this. + + The FreeType Project is copyright (C) 1996-2000 by David Turner, + Robert Wilhelm, and Werner Lemberg. All rights reserved except as + specified below. + +1. No Warranty +-------------- + + THE FREETYPE PROJECT IS PROVIDED `AS IS' WITHOUT WARRANTY OF ANY + KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, + WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + PURPOSE. IN NO EVENT WILL ANY OF THE AUTHORS OR COPYRIGHT HOLDERS + BE LIABLE FOR ANY DAMAGES CAUSED BY THE USE OR THE INABILITY TO + USE, OF THE FREETYPE PROJECT. + +2. Redistribution +----------------- + + This license grants a worldwide, royalty-free, perpetual and + irrevocable right and license to use, execute, perform, compile, + display, copy, create derivative works of, distribute and + sublicense the FreeType Project (in both source and object code + forms) and derivative works thereof for any purpose; and to + authorize others to exercise some or all of the rights granted + herein, subject to the following conditions: + + o Redistribution of source code must retain this license file + (`FTL.TXT') unaltered; any additions, deletions or changes to + the original files must be clearly indicated in accompanying + documentation. The copyright notices of the unaltered, + original files must be preserved in all copies of source + files. + + o Redistribution in binary form must provide a disclaimer that + states that the software is based in part of the work of the + FreeType Team, in the distribution documentation. We also + encourage you to put an URL to the FreeType web page in your + documentation, though this isn't mandatory. + + These conditions apply to any software derived from or based on + the FreeType Project, not just the unmodified files. If you use + our work, you must acknowledge us. However, no fee need be paid + to us. + +3. Advertising +-------------- + + Neither the FreeType authors and contributors nor you shall use + the name of the other for commercial, advertising, or promotional + purposes without specific prior written permission. + + We suggest, but do not require, that you use one or more of the + following phrases to refer to this software in your documentation + or advertising materials: `FreeType Project', `FreeType Engine', + `FreeType library', or `FreeType Distribution'. + + As you have not signed this license, you are not required to + accept it. However, as the FreeType Project is copyrighted + material, only this license, or another one contracted with the + authors, grants you the right to use, distribute, and modify it. + Therefore, by using, distributing, or modifying the FreeType + Project, you indicate that you understand and accept all the terms + of this license. + +4. Contacts +----------- + + There are two mailing lists related to FreeType: + + o freetype@nongnu.org + + Discusses general use and applications of FreeType, as well as + future and wanted additions to the library and distribution. + If you are looking for support, start in this list if you + haven't found anything to help you in the documentation. + + o freetype-devel@nongnu.org + + Discusses bugs, as well as engine internals, design issues, + specific licenses, porting, etc. + + Our home page can be found at + + https://www.freetype.org + +------------------------------------------------------------------------------ +libpng is redistributed within all opencv-python Linux and macOS packages. +This license applies to libpng binary in the directory cv2/. + +PNG Reference Library License version 2 +--------------------------------------- + + * Copyright (c) 1995-2019 The PNG Reference Library Authors. + * Copyright (c) 2018-2019 Cosmin Truta. + * Copyright (c) 2000-2002, 2004, 2006-2018 Glenn Randers-Pehrson. + * Copyright (c) 1996-1997 Andreas Dilger. + * Copyright (c) 1995-1996 Guy Eric Schalnat, Group 42, Inc. + +The software is supplied "as is", without warranty of any kind, +express or implied, including, without limitation, the warranties +of merchantability, fitness for a particular purpose, title, and +non-infringement. In no event shall the Copyright owners, or +anyone distributing the software, be liable for any damages or +other liability, whether in contract, tort or otherwise, arising +from, out of, or in connection with the software, or the use or +other dealings in the software, even if advised of the possibility +of such damage. + +Permission is hereby granted to use, copy, modify, and distribute +this software, or portions hereof, for any purpose, without fee, +subject to the following restrictions: + + 1. The origin of this software must not be misrepresented; you + must not claim that you wrote the original software. If you + use this software in a product, an acknowledgment in the product + documentation would be appreciated, but is not required. + + 2. Altered source versions must be plainly marked as such, and must + not be misrepresented as being the original software. + + 3. This Copyright notice may not be removed or altered from any + source or altered source distribution. + + +PNG Reference Library License version 1 (for libpng 0.5 through 1.6.35) +----------------------------------------------------------------------- + +libpng versions 1.0.7, July 1, 2000, through 1.6.35, July 15, 2018 are +Copyright (c) 2000-2002, 2004, 2006-2018 Glenn Randers-Pehrson, are +derived from libpng-1.0.6, and are distributed according to the same +disclaimer and license as libpng-1.0.6 with the following individuals +added to the list of Contributing Authors: + + Simon-Pierre Cadieux + Eric S. Raymond + Mans Rullgard + Cosmin Truta + Gilles Vollant + James Yu + Mandar Sahastrabuddhe + Google Inc. + Vadim Barkov + +and with the following additions to the disclaimer: + + There is no warranty against interference with your enjoyment of + the library or against infringement. There is no warranty that our + efforts or the library will fulfill any of your particular purposes + or needs. This library is provided with all faults, and the entire + risk of satisfactory quality, performance, accuracy, and effort is + with the user. + +Some files in the "contrib" directory and some configure-generated +files that are distributed with libpng have other copyright owners, and +are released under other open source licenses. + +libpng versions 0.97, January 1998, through 1.0.6, March 20, 2000, are +Copyright (c) 1998-2000 Glenn Randers-Pehrson, are derived from +libpng-0.96, and are distributed according to the same disclaimer and +license as libpng-0.96, with the following individuals added to the +list of Contributing Authors: + + Tom Lane + Glenn Randers-Pehrson + Willem van Schaik + +libpng versions 0.89, June 1996, through 0.96, May 1997, are +Copyright (c) 1996-1997 Andreas Dilger, are derived from libpng-0.88, +and are distributed according to the same disclaimer and license as +libpng-0.88, with the following individuals added to the list of +Contributing Authors: + + John Bowler + Kevin Bracey + Sam Bushell + Magnus Holmgren + Greg Roelofs + Tom Tanner + +Some files in the "scripts" directory have other copyright owners, +but are released under this license. + +libpng versions 0.5, May 1995, through 0.88, January 1996, are +Copyright (c) 1995-1996 Guy Eric Schalnat, Group 42, Inc. + +For the purposes of this copyright and license, "Contributing Authors" +is defined as the following set of individuals: + + Andreas Dilger + Dave Martindale + Guy Eric Schalnat + Paul Schmidt + Tim Wegner + +The PNG Reference Library is supplied "AS IS". The Contributing +Authors and Group 42, Inc. disclaim all warranties, expressed or +implied, including, without limitation, the warranties of +merchantability and of fitness for any purpose. The Contributing +Authors and Group 42, Inc. assume no liability for direct, indirect, +incidental, special, exemplary, or consequential damages, which may +result from the use of the PNG Reference Library, even if advised of +the possibility of such damage. + +Permission is hereby granted to use, copy, modify, and distribute this +source code, or portions hereof, for any purpose, without fee, subject +to the following restrictions: + + 1. The origin of this source code must not be misrepresented. + + 2. Altered versions must be plainly marked as such and must not + be misrepresented as being the original source. + + 3. This Copyright notice may not be removed or altered from any + source or altered source distribution. + +The Contributing Authors and Group 42, Inc. specifically permit, +without fee, and encourage the use of this source code as a component +to supporting the PNG file format in commercial products. If you use +this source code in a product, acknowledgment is not required but would +be appreciated. + +------------------------------------------------------------------------------ +libz is redistributed within all opencv-python Linux packages. +This license applies to libz binary in the directory cv2/. + + Copyright (C) 1995-2017 Jean-loup Gailly and Mark Adler + + This software is provided 'as-is', without any express or implied + warranty. In no event will the authors be held liable for any damages + arising from the use of this software. + + Permission is granted to anyone to use this software for any purpose, + including commercial applications, and to alter it and redistribute it + freely, subject to the following restrictions: + + 1. The origin of this software must not be misrepresented; you must not + claim that you wrote the original software. If you use this software + in a product, an acknowledgment in the product documentation would be + appreciated but is not required. + 2. Altered source versions must be plainly marked as such, and must not be + misrepresented as being the original software. + 3. This notice may not be removed or altered from any source distribution. + + Jean-loup Gailly Mark Adler + jloup@gzip.org madler@alumni.caltech.edu + +------------------------------------------------------------------------------ +libdav1d is redistributed within opencv-python macOS packages. +This license applies to libdav1d binary in the directory cv2/. + +Copyright © 2018-2019, VideoLAN and dav1d authors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +libffi is redistributed within opencv-python macOS packages. +This license applies to libffi binary in the directory cv2/. + +libffi - Copyright (c) 1996-2020 Anthony Green, Red Hat, Inc and others. +See source files for details. + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +``Software''), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, +TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +------------------------------------------------------------------------------ +libogg is redistributed within opencv-python macOS packages. +This license applies to libogg binary in the directory cv2/. + +Copyright (c) 2002, Xiph.org Foundation + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +- Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + +- Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +- Neither the name of the Xiph.org Foundation nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION +OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +libopenjp2 is redistributed within opencv-python macOS packages. +This license applies to libopenjp2 binary in the directory cv2/. + +The copyright in this software is being made available under the 2-clauses +BSD License, included below. This software may be subject to other third +party and contributor rights, including patent rights, and no such rights +are granted under this license. + +Copyright (c) 2002-2014, Universite catholique de Louvain (UCL), Belgium +Copyright (c) 2002-2014, Professor Benoit Macq +Copyright (c) 2003-2014, Antonin Descampe +Copyright (c) 2003-2009, Francois-Olivier Devaux +Copyright (c) 2005, Herve Drolon, FreeImage Team +Copyright (c) 2002-2003, Yannick Verschueren +Copyright (c) 2001-2003, David Janssens +Copyright (c) 2011-2012, Centre National d'Etudes Spatiales (CNES), France +Copyright (c) 2012, CS Systemes d'Information, France + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS `AS IS' +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +libopus is redistributed within opencv-python macOS packages. +This license applies to libopus binary in the directory cv2/. + +Copyright 2001-2011 Xiph.Org, Skype Limited, Octasic, + Jean-Marc Valin, Timothy B. Terriberry, + CSIRO, Gregory Maxwell, Mark Borgerding, + Erik de Castro Lopo + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +- Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + +- Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +- Neither the name of Internet Society, IETF or IETF Trust, nor the +names of specific contributors, may be used to endorse or promote +products derived from this software without specific prior written +permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER +OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Opus is subject to the royalty-free patent licenses which are +specified at: + +Xiph.Org Foundation: +https://datatracker.ietf.org/ipr/1524/ + +Microsoft Corporation: +https://datatracker.ietf.org/ipr/1914/ + +Broadcom Corporation: +https://datatracker.ietf.org/ipr/1526/ + +------------------------------------------------------------------------------ +librav1e is redistributed within opencv-python macOS packages. +This license applies to librav1e binary in the directory cv2/. + +BSD 2-Clause License + +Copyright (c) 2017-2020, the rav1e contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +libsnappy is redistributed within opencv-python macOS packages. +This license applies to libsnappy binary in the directory cv2/. + +Copyright 2011, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +libspeex is redistributed within opencv-python macOS packages. +This license applies to libspeex binary in the directory cv2/. + +Copyright 2002-2008 Xiph.org Foundation +Copyright 2002-2008 Jean-Marc Valin +Copyright 2005-2007 Analog Devices Inc. +Copyright 2005-2008 Commonwealth Scientific and Industrial Research + Organisation (CSIRO) +Copyright 1993, 2002, 2006 David Rowe +Copyright 2003 EpicGames +Copyright 1992-1994 Jutta Degener, Carsten Bormann + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +- Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + +- Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +- Neither the name of the Xiph.org Foundation nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR +CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +libsrt is redistributed within opencv-python macOS packages. +This license applies to libsrt binary in the directory cv2/. + +/* + * + * Copyright (c) 2001-2017 Cisco Systems, Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * Neither the name of the Cisco Systems, Inc. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS + * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE + * COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED + * OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + + + Mozilla Public License Version 2.0 +================================== + +1. Definitions +-------------- + +1.1. "Contributor" + means each individual or legal entity that creates, contributes to + the creation of, or owns Covered Software. + +1.2. "Contributor Version" + means the combination of the Contributions of others (if any) used + by a Contributor and that particular Contributor's Contribution. + +1.3. "Contribution" + means Covered Software of a particular Contributor. + +1.4. "Covered Software" + means Source Code Form to which the initial Contributor has attached + the notice in Exhibit A, the Executable Form of such Source Code + Form, and Modifications of such Source Code Form, in each case + including portions thereof. + +1.5. "Incompatible With Secondary Licenses" + means + + (a) that the initial Contributor has attached the notice described + in Exhibit B to the Covered Software; or + + (b) that the Covered Software was made available under the terms of + version 1.1 or earlier of the License, but not also under the + terms of a Secondary License. + +1.6. "Executable Form" + means any form of the work other than Source Code Form. + +1.7. "Larger Work" + means a work that combines Covered Software with other material, in + a separate file or files, that is not Covered Software. + +1.8. "License" + means this document. + +1.9. "Licensable" + means having the right to grant, to the maximum extent possible, + whether at the time of the initial grant or subsequently, any and + all of the rights conveyed by this License. + +1.10. "Modifications" + means any of the following: + + (a) any file in Source Code Form that results from an addition to, + deletion from, or modification of the contents of Covered + Software; or + + (b) any new file in Source Code Form that contains any Covered + Software. + +1.11. "Patent Claims" of a Contributor + means any patent claim(s), including without limitation, method, + process, and apparatus claims, in any patent Licensable by such + Contributor that would be infringed, but for the grant of the + License, by the making, using, selling, offering for sale, having + made, import, or transfer of either its Contributions or its + Contributor Version. + +1.12. "Secondary License" + means either the GNU General Public License, Version 2.0, the GNU + Lesser General Public License, Version 2.1, the GNU Affero General + Public License, Version 3.0, or any later versions of those + licenses. + +1.13. "Source Code Form" + means the form of the work preferred for making modifications. + +1.14. "You" (or "Your") + means an individual or a legal entity exercising rights under this + License. For legal entities, "You" includes any entity that + controls, is controlled by, or is under common control with You. For + purposes of this definition, "control" means (a) the power, direct + or indirect, to cause the direction or management of such entity, + whether by contract or otherwise, or (b) ownership of more than + fifty percent (50%) of the outstanding shares or beneficial + ownership of such entity. + +2. License Grants and Conditions +-------------------------------- + +2.1. Grants + +Each Contributor hereby grants You a world-wide, royalty-free, +non-exclusive license: + +(a) under intellectual property rights (other than patent or trademark) + Licensable by such Contributor to use, reproduce, make available, + modify, display, perform, distribute, and otherwise exploit its + Contributions, either on an unmodified basis, with Modifications, or + as part of a Larger Work; and + +(b) under Patent Claims of such Contributor to make, use, sell, offer + for sale, have made, import, and otherwise transfer either its + Contributions or its Contributor Version. + +2.2. Effective Date + +The licenses granted in Section 2.1 with respect to any Contribution +become effective for each Contribution on the date the Contributor first +distributes such Contribution. + +2.3. Limitations on Grant Scope + +The licenses granted in this Section 2 are the only rights granted under +this License. No additional rights or licenses will be implied from the +distribution or licensing of Covered Software under this License. +Notwithstanding Section 2.1(b) above, no patent license is granted by a +Contributor: + +(a) for any code that a Contributor has removed from Covered Software; + or + +(b) for infringements caused by: (i) Your and any other third party's + modifications of Covered Software, or (ii) the combination of its + Contributions with other software (except as part of its Contributor + Version); or + +(c) under Patent Claims infringed by Covered Software in the absence of + its Contributions. + +This License does not grant any rights in the trademarks, service marks, +or logos of any Contributor (except as may be necessary to comply with +the notice requirements in Section 3.4). + +2.4. Subsequent Licenses + +No Contributor makes additional grants as a result of Your choice to +distribute the Covered Software under a subsequent version of this +License (see Section 10.2) or under the terms of a Secondary License (if +permitted under the terms of Section 3.3). + +2.5. Representation + +Each Contributor represents that the Contributor believes its +Contributions are its original creation(s) or it has sufficient rights +to grant the rights to its Contributions conveyed by this License. + +2.6. Fair Use + +This License is not intended to limit any rights You have under +applicable copyright doctrines of fair use, fair dealing, or other +equivalents. + +2.7. Conditions + +Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted +in Section 2.1. + +3. Responsibilities +------------------- + +3.1. Distribution of Source Form + +All distribution of Covered Software in Source Code Form, including any +Modifications that You create or to which You contribute, must be under +the terms of this License. You must inform recipients that the Source +Code Form of the Covered Software is governed by the terms of this +License, and how they can obtain a copy of this License. You may not +attempt to alter or restrict the recipients' rights in the Source Code +Form. + +3.2. Distribution of Executable Form + +If You distribute Covered Software in Executable Form then: + +(a) such Covered Software must also be made available in Source Code + Form, as described in Section 3.1, and You must inform recipients of + the Executable Form how they can obtain a copy of such Source Code + Form by reasonable means in a timely manner, at a charge no more + than the cost of distribution to the recipient; and + +(b) You may distribute such Executable Form under the terms of this + License, or sublicense it under different terms, provided that the + license for the Executable Form does not attempt to limit or alter + the recipients' rights in the Source Code Form under this License. + +3.3. Distribution of a Larger Work + +You may create and distribute a Larger Work under terms of Your choice, +provided that You also comply with the requirements of this License for +the Covered Software. If the Larger Work is a combination of Covered +Software with a work governed by one or more Secondary Licenses, and the +Covered Software is not Incompatible With Secondary Licenses, this +License permits You to additionally distribute such Covered Software +under the terms of such Secondary License(s), so that the recipient of +the Larger Work may, at their option, further distribute the Covered +Software under the terms of either this License or such Secondary +License(s). + +3.4. Notices + +You may not remove or alter the substance of any license notices +(including copyright notices, patent notices, disclaimers of warranty, +or limitations of liability) contained within the Source Code Form of +the Covered Software, except that You may alter any license notices to +the extent required to remedy known factual inaccuracies. + +3.5. Application of Additional Terms + +You may choose to offer, and to charge a fee for, warranty, support, +indemnity or liability obligations to one or more recipients of Covered +Software. However, You may do so only on Your own behalf, and not on +behalf of any Contributor. You must make it absolutely clear that any +such warranty, support, indemnity, or liability obligation is offered by +You alone, and You hereby agree to indemnify every Contributor for any +liability incurred by such Contributor as a result of warranty, support, +indemnity or liability terms You offer. You may include additional +disclaimers of warranty and limitations of liability specific to any +jurisdiction. + +4. Inability to Comply Due to Statute or Regulation +--------------------------------------------------- + +If it is impossible for You to comply with any of the terms of this +License with respect to some or all of the Covered Software due to +statute, judicial order, or regulation then You must: (a) comply with +the terms of this License to the maximum extent possible; and (b) +describe the limitations and the code they affect. Such description must +be placed in a text file included with all distributions of the Covered +Software under this License. Except to the extent prohibited by statute +or regulation, such description must be sufficiently detailed for a +recipient of ordinary skill to be able to understand it. + +5. Termination +-------------- + +5.1. The rights granted under this License will terminate automatically +if You fail to comply with any of its terms. However, if You become +compliant, then the rights granted under this License from a particular +Contributor are reinstated (a) provisionally, unless and until such +Contributor explicitly and finally terminates Your grants, and (b) on an +ongoing basis, if such Contributor fails to notify You of the +non-compliance by some reasonable means prior to 60 days after You have +come back into compliance. Moreover, Your grants from a particular +Contributor are reinstated on an ongoing basis if such Contributor +notifies You of the non-compliance by some reasonable means, this is the +first time You have received notice of non-compliance with this License +from such Contributor, and You become compliant prior to 30 days after +Your receipt of the notice. + +5.2. If You initiate litigation against any entity by asserting a patent +infringement claim (excluding declaratory judgment actions, +counter-claims, and cross-claims) alleging that a Contributor Version +directly or indirectly infringes any patent, then the rights granted to +You by any and all Contributors for the Covered Software under Section +2.1 of this License shall terminate. + +5.3. In the event of termination under Sections 5.1 or 5.2 above, all +end user license agreements (excluding distributors and resellers) which +have been validly granted by You or Your distributors under this License +prior to termination shall survive termination. + +************************************************************************ +* * +* 6. Disclaimer of Warranty * +* ------------------------- * +* * +* Covered Software is provided under this License on an "as is" * +* basis, without warranty of any kind, either expressed, implied, or * +* statutory, including, without limitation, warranties that the * +* Covered Software is free of defects, merchantable, fit for a * +* particular purpose or non-infringing. The entire risk as to the * +* quality and performance of the Covered Software is with You. * +* Should any Covered Software prove defective in any respect, You * +* (not any Contributor) assume the cost of any necessary servicing, * +* repair, or correction. This disclaimer of warranty constitutes an * +* essential part of this License. No use of any Covered Software is * +* authorized under this License except under this disclaimer. * +* * +************************************************************************ + +************************************************************************ +* * +* 7. Limitation of Liability * +* -------------------------- * +* * +* Under no circumstances and under no legal theory, whether tort * +* (including negligence), contract, or otherwise, shall any * +* Contributor, or anyone who distributes Covered Software as * +* permitted above, be liable to You for any direct, indirect, * +* special, incidental, or consequential damages of any character * +* including, without limitation, damages for lost profits, loss of * +* goodwill, work stoppage, computer failure or malfunction, or any * +* and all other commercial damages or losses, even if such party * +* shall have been informed of the possibility of such damages. This * +* limitation of liability shall not apply to liability for death or * +* personal injury resulting from such party's negligence to the * +* extent applicable law prohibits such limitation. Some * +* jurisdictions do not allow the exclusion or limitation of * +* incidental or consequential damages, so this exclusion and * +* limitation may not apply to You. * +* * +************************************************************************ + +8. Litigation +------------- + +Any litigation relating to this License may be brought only in the +courts of a jurisdiction where the defendant maintains its principal +place of business and such litigation shall be governed by laws of that +jurisdiction, without reference to its conflict-of-law provisions. +Nothing in this Section shall prevent a party's ability to bring +cross-claims or counter-claims. + +9. Miscellaneous +---------------- + +This License represents the complete agreement concerning the subject +matter hereof. If any provision of this License is held to be +unenforceable, such provision shall be reformed only to the extent +necessary to make it enforceable. Any law or regulation which provides +that the language of a contract shall be construed against the drafter +shall not be used to construe this License against a Contributor. + +10. Versions of the License +--------------------------- + +10.1. New Versions + +Mozilla Foundation is the license steward. Except as provided in Section +10.3, no one other than the license steward has the right to modify or +publish new versions of this License. Each version will be given a +distinguishing version number. + +10.2. Effect of New Versions + +You may distribute the Covered Software under the terms of the version +of the License under which You originally received the Covered Software, +or under the terms of any subsequent version published by the license +steward. + +10.3. Modified Versions + +If you create software not governed by this License, and you want to +create a new license for such software, you may create and use a +modified version of this License if you rename the license and remove +any references to the name of the license steward (except to note that +such modified license differs from this License). + +10.4. Distributing Source Code Form that is Incompatible With Secondary +Licenses + +If You choose to distribute Source Code Form that is Incompatible With +Secondary Licenses under the terms of this version of the License, the +notice described in Exhibit B of this License must be attached. + +Exhibit A - Source Code Form License Notice +------------------------------------------- + + This Source Code Form is subject to the terms of the Mozilla Public + License, v. 2.0. If a copy of the MPL was not distributed with this + file, You can obtain one at http://mozilla.org/MPL/2.0/. + +If it is not possible or desirable to put the notice in a particular +file, then You may include the notice in a location (such as a LICENSE +file in a relevant directory) where a recipient would be likely to look +for such a notice. + +You may add additional accurate notices of copyright ownership. + +Exhibit B - "Incompatible With Secondary Licenses" Notice +--------------------------------------------------------- + + This Source Code Form is "Incompatible With Secondary Licenses", as + defined by the Mozilla Public License, v. 2.0. + +------------------------------------------------------------------------------ +libtheoradec and libtheoraenc are redistributed within opencv-python macOS packages. +This license applies to libtheoradec and libtheoraenc binaries in the directory cv2/. + + Copyright (C) 2002-2009 Xiph.org Foundation + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +- Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + +- Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +- Neither the name of the Xiph.org Foundation nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION +OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +libwebp and libwebpmux are redistributed within all opencv-python packages. +This license applies to libwebp and libwebpmux binaries in the directory cv2/. + +Copyright (c) 2010, Google Inc. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + + * Neither the name of Google nor the names of its contributors may + be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +libvorbis and libvorbisenc are redistributed within opencv-python macOS packages. +This license applies to libvorbis and libvorbisenc binaries in the directory cv2/. + +Copyright (c) 2002-2020 Xiph.org Foundation + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +- Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + +- Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +- Neither the name of the Xiph.org Foundation nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION +OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +Libxcb utility libraries are redistributed within opencv-python non-headless Linux packages. +This license applies to libxcb related binaries in the directory cv2/. + +Copyright (C) 2001-2006 Bart Massey, Jamey Sharp, and Josh Triplett. +All Rights Reserved. + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated +documentation files (the "Software"), to deal in the +Software without restriction, including without limitation +the rights to use, copy, modify, merge, publish, distribute, +sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall +be included in all copies or substantial portions of the +Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY +KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE +WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR +PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS +BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the names of the authors +or their institutions shall not be used in advertising or +otherwise to promote the sale, use or other dealings in this +Software without prior written authorization from the +authors. + +------------------------------------------------------------------------------ +Libxcb-image is redistributed within opencv-python non-headless Linux packages. +This license applies to libxcb-image binary in the directory cv2/. + +Copyright © 2007-2008 Bart Massey +Copyright © 2008 Julien Danjou +Copyright © 2008 Keith Packard + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated documentation +files (the "Software"), to deal in the Software without +restriction, including without limitation the rights to use, copy, +modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF +CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the names of the authors or +their institutions shall not be used in advertising or otherwise to +promote the sale, use or other dealings in this Software without +prior written authorization from the authors. + +------------------------------------------------------------------------------ +Libxcb-util is redistributed within opencv-python non-headless Linux packages. +This license applies to libxcb-util binary in the directory cv2/. + +Copyright © 2008 Bart Massey +Copyright © 2008 Ian Osgood +Copyright © 2008 Jamey Sharp +Copyright © 2008 Josh Triplett +Copyright © 2008-2009 Julien Danjou + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated documentation +files (the "Software"), to deal in the Software without +restriction, including without limitation the rights to use, copy, +modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF +CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the names of the authors or +their institutions shall not be used in advertising or otherwise to +promote the sale, use or other dealings in this Software without +prior written authorization from the authors. + +------------------------------------------------------------------------------ +Libxcb-render-util is redistributed within opencv-python non-headless Linux packages. +This license applies to libxcb-render-util binary in the directory cv2/. + +Copyright © 2000 Keith Packard + +Permission to use, copy, modify, distribute, and sell this software and its +documentation for any purpose is hereby granted without fee, provided that +the above copyright notice appear in all copies and that both that +copyright notice and this permission notice appear in supporting +documentation, and that the name of Keith Packard not be used in +advertising or publicity pertaining to distribution of the software without +specific, written prior permission. Keith Packard makes no +representations about the suitability of this software for any purpose. It +is provided "as is" without express or implied warranty. + +KEITH PACKARD DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, +INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO +EVENT SHALL KEITH PACKARD BE LIABLE FOR ANY SPECIAL, INDIRECT OR +CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, +DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER +TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR +PERFORMANCE OF THIS SOFTWARE. + +Copyright © 2006 Jamey Sharp. + +Permission is hereby granted, free of charge, to any person obtaining a +copy of this software and associated documentation files (the "Software"), +to deal in the Software without restriction, including without limitation +the rights to use, copy, modify, merge, publish, distribute, sublicense, +and/or sell copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN +ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the names of the authors or their +institutions shall not be used in advertising or otherwise to promote the +sale, use or other dealings in this Software without prior written +authorization from the authors. + +Copyright © 2006 Ian Osgood + +Permission is hereby granted, free of charge, to any person obtaining a +copy of this software and associated documentation files (the "Software"), +to deal in the Software without restriction, including without limitation +the rights to use, copy, modify, merge, publish, distribute, sublicense, +and/or sell copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN +ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the names of the authors or their +institutions shall not be used in advertising or otherwise to promote the +sale, use or other dealings in this Software without prior written +authorization from the authors. + +------------------------------------------------------------------------------ +Libxcb-icccm is redistributed within opencv-python non-headless Linux packages. +This license applies to Libxcb-icccm binary in the directory cv2/. + +Copyright © 2008-2011 Arnaud Fontaine +Copyright © 2007-2008 Vincent Torri + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated documentation +files (the "Software"), to deal in the Software without +restriction, including without limitation the rights to use, copy, +modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF +CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the names of the authors or +their institutions shall not be used in advertising or otherwise to +promote the sale, use or other dealings in this Software without +prior written authorization from the authors. + +------------------------------------------------------------------------------ +libXau is redistributed within opencv-python non-headless Linux packages. +This license applies to libXau binary in the directory cv2/. + +Copyright 1988, 1993, 1994, 1998 The Open Group + +Permission to use, copy, modify, distribute, and sell this software and its +documentation for any purpose is hereby granted without fee, provided that +the above copyright notice appear in all copies and that both that +copyright notice and this permission notice appear in supporting +documentation. + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +OPEN GROUP BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN +AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the name of The Open Group shall not be +used in advertising or otherwise to promote the sale, use or other dealings +in this Software without prior written authorization from The Open Group. + +------------------------------------------------------------------------------ +Vulkan headers are redistributed within all opencv-python packages. +This license applies to Vulkan headers in the directory 3rdparty/include/vulkan. + +Copyright (c) 2015-2018 The Khronos Group Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +------------------------------------------------------------------------------ +Libjpeg-turbo is redistributed within all opencv-python packages as build option. + +libjpeg-turbo Licenses +====================== + +libjpeg-turbo is covered by three compatible BSD-style open source licenses: + +- The IJG (Independent JPEG Group) License, which is listed in + [README.ijg](README.ijg) + + This license applies to the libjpeg API library and associated programs + (any code inherited from libjpeg, and any modifications to that code.) + +- The Modified (3-clause) BSD License, which is listed below + + This license covers the TurboJPEG API library and associated programs, as + well as the build system. + +- The [zlib License](https://opensource.org/licenses/Zlib) + + This license is a subset of the other two, and it covers the libjpeg-turbo + SIMD extensions. + + +Complying with the libjpeg-turbo Licenses +========================================= + +This section provides a roll-up of the libjpeg-turbo licensing terms, to the +best of our understanding. + +1. If you are distributing a modified version of the libjpeg-turbo source, + then: + + 1. You cannot alter or remove any existing copyright or license notices + from the source. + + **Origin** + - Clause 1 of the IJG License + - Clause 1 of the Modified BSD License + - Clauses 1 and 3 of the zlib License + + 2. You must add your own copyright notice to the header of each source + file you modified, so others can tell that you modified that file (if + there is not an existing copyright header in that file, then you can + simply add a notice stating that you modified the file.) + + **Origin** + - Clause 1 of the IJG License + - Clause 2 of the zlib License + + 3. You must include the IJG README file, and you must not alter any of the + copyright or license text in that file. + + **Origin** + - Clause 1 of the IJG License + +2. If you are distributing only libjpeg-turbo binaries without the source, or + if you are distributing an application that statically links with + libjpeg-turbo, then: + + 1. Your product documentation must include a message stating: + + This software is based in part on the work of the Independent JPEG + Group. + + **Origin** + - Clause 2 of the IJG license + + 2. If your binary distribution includes or uses the TurboJPEG API, then + your product documentation must include the text of the Modified BSD + License (see below.) + + **Origin** + - Clause 2 of the Modified BSD License + +3. You cannot use the name of the IJG or The libjpeg-turbo Project or the + contributors thereof in advertising, publicity, etc. + + **Origin** + - IJG License + - Clause 3 of the Modified BSD License + +4. The IJG and The libjpeg-turbo Project do not warrant libjpeg-turbo to be + free of defects, nor do we accept any liability for undesirable + consequences resulting from your use of the software. + + **Origin** + - IJG License + - Modified BSD License + - zlib License + + +The Modified (3-clause) BSD License +=================================== + +Copyright (C)2009-2022 D. R. Commander. All Rights Reserved.
+Copyright (C)2015 Viktor Szathmáry. All Rights Reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +- Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. +- Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. +- Neither the name of the libjpeg-turbo Project nor the names of its + contributors may be used to endorse or promote products derived from this + software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS", +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + + +Why Three Licenses? +=================== + +The zlib License could have been used instead of the Modified (3-clause) BSD +License, and since the IJG License effectively subsumes the distribution +conditions of the zlib License, this would have effectively placed +libjpeg-turbo binary distributions under the IJG License. However, the IJG +License specifically refers to the Independent JPEG Group and does not extend +attribution and endorsement protections to other entities. Thus, it was +desirable to choose a license that granted us the same protections for new code +that were granted to the IJG for code derived from their software. + +------------------------------------------------------------------------------ +Libspng is redistributed within all opencv-python packages as build option. + +BSD 2-Clause License + +Copyright (c) 2018-2022, Randy +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +QUIRC library is redistributed within all opencv-python packages. + +quirc -- QR-code recognition library +Copyright (C) 2010-2012 Daniel Beer + +Permission to use, copy, modify, and/or distribute this software for +any purpose with or without fee is hereby granted, provided that the +above copyright notice and this permission notice appear in all +copies. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL +WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE +AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL +DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR +PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER +TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR +PERFORMANCE OF THIS SOFTWARE. + +------------------------------------------------------------------------------ +Flatbuffers library is redistributed within all opencv-python packages. + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +------------------------------------------------------------------------------ +Protobuf library is redistributed within all opencv-python packages. + +Copyright 2008 Google Inc. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Code generated by the Protocol Buffer compiler is owned by the owner +of the input file used when generating it. This code is not +standalone and requires a support library to be linked with it. This +support library is itself covered by the above license. + +------------------------------------------------------------------------------ +OpenJPEG library is redistributed within all opencv-python packages. + +/* + * The copyright in this software is being made available under the 2-clauses + * BSD License, included below. This software may be subject to other third + * party and contributor rights, including patent rights, and no such rights + * are granted under this license. + * + * Copyright (c) 2002-2014, Universite catholique de Louvain (UCL), Belgium + * Copyright (c) 2002-2014, Professor Benoit Macq + * Copyright (c) 2003-2014, Antonin Descampe + * Copyright (c) 2003-2009, Francois-Olivier Devaux + * Copyright (c) 2005, Herve Drolon, FreeImage Team + * Copyright (c) 2002-2003, Yannick Verschueren + * Copyright (c) 2001-2003, David Janssens + * Copyright (c) 2011-2012, Centre National d'Etudes Spatiales (CNES), France + * Copyright (c) 2012, CS Systemes d'Information, France + * + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS `AS IS' + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ + +------------------------------------------------------------------------------ +TIFF library is redistributed within all opencv-python packages. + +Copyright (c) 1988-1997 Sam Leffler +Copyright (c) 1991-1997 Silicon Graphics, Inc. + +Permission to use, copy, modify, distribute, and sell this software and +its documentation for any purpose is hereby granted without fee, provided +that (i) the above copyright notices and this permission notice appear in +all copies of the software and related documentation, and (ii) the names of +Sam Leffler and Silicon Graphics may not be used in any advertising or +publicity relating to the software without the specific, prior written +permission of Sam Leffler and Silicon Graphics. + +THE SOFTWARE IS PROVIDED "AS-IS" AND WITHOUT WARRANTY OF ANY KIND, +EXPRESS, IMPLIED OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY +WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. + +IN NO EVENT SHALL SAM LEFFLER OR SILICON GRAPHICS BE LIABLE FOR +ANY SPECIAL, INCIDENTAL, INDIRECT OR CONSEQUENTIAL DAMAGES OF ANY KIND, +OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, +WHETHER OR NOT ADVISED OF THE POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF +LIABILITY, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE +OF THIS SOFTWARE. + +------------------------------------------------------------------------------ +OpenEXR library is redistributed within all opencv-python packages. + +Copyright (c) 2006, Industrial Light & Magic, a division of Lucasfilm +Entertainment Company Ltd. Portions contributed and copyright held by +others as indicated. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above + copyright notice, this list of conditions and the following + disclaimer. + + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided with + the distribution. + + * Neither the name of Industrial Light & Magic nor the names of + any other contributors to this software may be used to endorse or + promote products derived from this software without specific prior + written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS +IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ +Intel(R) IPP ICV library statically linked within x86 and x86_64 opencv-python packages. + +Intel(R) Integrated Performance Primitives 2021 Update 10 + +Intel Simplified Software License (Version October 2022) + +Intel(R) Integrated Performance Primitives (Intel(R) IPP) : Copyright (C) 1997 Intel Corporation + +Use and Redistribution. You may use and redistribute the software, which is +provided in binary form only, (the "Software"), without modification, +provided the following conditions are met: + +* Redistributions must reproduce the above copyright notice and these + terms of use in the Software and in the documentation and/or other materials + provided with the distribution. +* Neither the name of Intel nor the names of its suppliers may be used to + endorse or promote products derived from this Software without specific + prior written permission. +* No reverse engineering, decompilation, or disassembly of the Software is + permitted, nor any modification or alteration of the Software or its operation + at any time, including during execution. + +No other licenses. Except as provided in the preceding section, Intel grants no +licenses or other rights by implication, estoppel or otherwise to, patent, +copyright, trademark, trade name, service mark or other intellectual property +licenses or rights of Intel. + +Third party software. "Third Party Software" means the files (if any) listed +in the "third-party-software.txt" or other similarly-named text file that may +be included with the Software. Third Party Software, even if included with the +distribution of the Software, may be governed by separate license terms, including +without limitation, third party license terms, open source software notices and +terms, and/or other Intel software license terms. These separate license terms +solely govern Your use of the Third Party Software. + +DISCLAIMER. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY EXPRESS OR IMPLIED +WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT ARE +DISCLAIMED. THIS SOFTWARE IS NOT INTENDED FOR USE IN SYSTEMS OR APPLICATIONS +WHERE FAILURE OF THE SOFTWARE MAY CAUSE PERSONAL INJURY OR DEATH AND YOU AGREE +THAT YOU ARE FULLY RESPONSIBLE FOR ANY CLAIMS, COSTS, DAMAGES, EXPENSES, AND +ATTORNEYS' FEES ARISING OUT OF ANY SUCH USE, EVEN IF ANY CLAIM ALLEGES THAT +INTEL WAS NEGLIGENT REGARDING THE DESIGN OR MANUFACTURE OF THE SOFTWARE. + +LIMITATION OF LIABILITY. IN NO EVENT WILL INTEL BE LIABLE FOR ANY DIRECT, +INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, +BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE +OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF +ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +No support. Intel may make changes to the Software, at any time without notice, +and is not obligated to support, update or provide training for the Software. + +Termination. Your right to use the Software is terminated in the event of your +breach of this license. + +Feedback. Should you provide Intel with comments, modifications, corrections, +enhancements or other input ("Feedback") related to the Software, Intel will be +free to use, disclose, reproduce, license or otherwise distribute or exploit the +Feedback in its sole discretion without any obligations or restrictions of any +kind, including without limitation, intellectual property rights or licensing +obligations. + +Compliance with laws. You agree to comply with all relevant laws and regulations +governing your use, transfer, import or export (or prohibition thereof) of the +Software. + +Governing law. All disputes will be governed by the laws of the United States of +America and the State of Delaware without reference to conflict of law +principles and subject to the exclusive jurisdiction of the state or federal +courts sitting in the State of Delaware, and each party agrees that it submits +to the personal jurisdiction and venue of those courts and waives any +objections. THE UNITED NATIONS CONVENTION ON CONTRACTS FOR THE INTERNATIONAL +SALE OF GOODS (1980) IS SPECIFICALLY EXCLUDED AND WILL NOT APPLY TO THE SOFTWARE. + +------------------------------------------------------------------------------ +Orbbec SDK distributed with arm64 MacOS packages. + +MIT License + +Copyright (c) 2023 OrbbecDeveloper + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +------------------------------------------------------------------------------ + +libavif library and it's dependnecies are redistributed within all opencv-python packages. + +Copyright 2019 Joe Drago. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this +list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, +this list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ + +Files: src/obu.c + +Copyright © 2018-2019, VideoLAN and dav1d authors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ + +Files: third_party/iccjpeg/* + +In plain English: + +1. We don't promise that this software works. (But if you find any bugs, + please let us know!) +2. You can use this software for whatever you want. You don't have to pay us. +3. You may not pretend that you wrote this software. If you use it in a + program, you must acknowledge somewhere in your documentation that + you've used the IJG code. + +In legalese: + +The authors make NO WARRANTY or representation, either express or implied, +with respect to this software, its quality, accuracy, merchantability, or +fitness for a particular purpose. This software is provided "AS IS", and you, +its user, assume the entire risk as to its quality and accuracy. + +This software is copyright (C) 1991-2013, Thomas G. Lane, Guido Vollbeding. +All Rights Reserved except as specified below. + +Permission is hereby granted to use, copy, modify, and distribute this +software (or portions thereof) for any purpose, without fee, subject to these +conditions: +(1) If any part of the source code for this software is distributed, then this +README file must be included, with this copyright and no-warranty notice +unaltered; and any additions, deletions, or changes to the original files +must be clearly indicated in accompanying documentation. +(2) If only executable code is distributed, then the accompanying +documentation must state that "this software is based in part on the work of +the Independent JPEG Group". +(3) Permission for use of this software is granted only if the user accepts +full responsibility for any undesirable consequences; the authors accept +NO LIABILITY for damages of any kind. + +These conditions apply to any software derived from or based on the IJG code, +not just to the unmodified library. If you use our work, you ought to +acknowledge us. + +Permission is NOT granted for the use of any IJG author's name or company name +in advertising or publicity relating to this software or products derived from +it. This software may be referred to only as "the Independent JPEG Group's +software". + +We specifically permit and encourage the use of this software as the basis of +commercial products, provided that all warranty or liability claims are +assumed by the product vendor. + + +The Unix configuration script "configure" was produced with GNU Autoconf. +It is copyright by the Free Software Foundation but is freely distributable. +The same holds for its supporting scripts (config.guess, config.sub, +ltmain.sh). Another support script, install-sh, is copyright by X Consortium +but is also freely distributable. + +The IJG distribution formerly included code to read and write GIF files. +To avoid entanglement with the Unisys LZW patent, GIF reading support has +been removed altogether, and the GIF writer has been simplified to produce +"uncompressed GIFs". This technique does not use the LZW algorithm; the +resulting GIF files are larger than usual, but are readable by all standard +GIF decoders. + +We are required to state that + "The Graphics Interchange Format(c) is the Copyright property of + CompuServe Incorporated. GIF(sm) is a Service Mark property of + CompuServe Incorporated." + +------------------------------------------------------------------------------ + +Files: contrib/gdk-pixbuf/* + +Copyright 2020 Emmanuel Gil Peyrot. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this +list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, +this list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ + +Files: android_jni/gradlew* + + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +------------------------------------------------------------------------------ + +Files: third_party/libyuv/* + +Copyright 2011 The LibYuv Project Authors. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + + * Neither the name of Google nor the names of its contributors may + be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ + +aom library and it's dependnecies are redistributed within all opencv-python packages. + +Copyright (c) 2016, Alliance for Open Media. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS +FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE +COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, +BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN +ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ + +KAZE Features library is redistributed within all opencv-python packages. + +Copyright (c) 2012, Pablo Fernández Alcantarilla +All Rights Reserved + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + * Neither the name of the copyright holders nor the names of its contributors + may be used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY +EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES +OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT +SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR +BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY +WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ + +AKAZE Features library is redistributed within all opencv-python packages. + +Copyright (c) 2014, Pablo Fernandez Alcantarilla, Jesus Nuevo +All Rights Reserved + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + * Neither the name of the copyright holders nor the names of its contributors + may be used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY +EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES +OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT +SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR +BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY +WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +opentelemetry-api +Apache-2.0 +https://github.com/open-telemetry/opentelemetry-python/tree/main/opentelemetry-api + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +opentelemetry-exporter-otlp-proto-common +Apache-2.0 +https://github.com/open-telemetry/opentelemetry-python/tree/main/exporter/opentelemetry-exporter-otlp-proto-common + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +opentelemetry-exporter-otlp-proto-http +Apache-2.0 +https://github.com/open-telemetry/opentelemetry-python/tree/main/exporter/opentelemetry-exporter-otlp-proto-http + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +opentelemetry-proto +Apache-2.0 +https://github.com/open-telemetry/opentelemetry-python/tree/main/opentelemetry-proto + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +opentelemetry-sdk +Apache-2.0 +https://github.com/open-telemetry/opentelemetry-python/tree/main/opentelemetry-sdk + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +opentelemetry-semantic-conventions +Apache-2.0 +https://github.com/open-telemetry/opentelemetry-python/tree/main/opentelemetry-semantic-conventions + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +orderly-set +MIT License +https://github.com/seperman/orderly-set +UNKNOWN + +orjson +MPL-2.0 AND (Apache-2.0 OR MIT) +https://github.com/ijl/orjson + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + +3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + +4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + +5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + +6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + +8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + +END OF TERMS AND CONDITIONS + +APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + +Copyright [yyyy] [name of copyright owner] + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + + +packaging +Apache Software License; BSD License +https://github.com/pypa/packaging +This software is made available under the terms of *either* of the licenses +found in LICENSE.APACHE or LICENSE.BSD. Contributions to this software is made +under the terms of *both* these licenses. + + +pandas +BSD License +https://pandas.pydata.org +BSD 3-Clause License + +Copyright (c) 2008-2011, AQR Capital Management, LLC, Lambda Foundry, Inc. and PyData Development Team +All rights reserved. + +Copyright (c) 2011-2023, Open source contributors. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Copyright (c) 2010-2019 Keith Goodman +Copyright (c) 2019 Bottleneck Developers +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE.Copyright 2017- Paul Ganssle +Copyright 2017- dateutil contributors (see AUTHORS file) + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +The above license applies to all contributions after 2017-12-01, as well as +all contributions that have been re-licensed (see AUTHORS file for the list of +contributors who have re-licensed their code). +-------------------------------------------------------------------------------- +dateutil - Extensions to the standard Python datetime module. + +Copyright (c) 2003-2011 - Gustavo Niemeyer +Copyright (c) 2012-2014 - Tomi Pieviläinen +Copyright (c) 2014-2016 - Yaron de Leeuw +Copyright (c) 2015- - Paul Ganssle +Copyright (c) 2015- - dateutil contributors (see AUTHORS file) + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + * Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +The above BSD License Applies to all code, even that also covered by Apache 2.0.# MIT License + +Copyright (c) 2019 Hadley Wickham; RStudio; and Evan Miller + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +Based on http://opensource.org/licenses/MIT + +This is a template. Complete and ship as file LICENSE the following 2 +lines (only) + +YEAR: +COPYRIGHT HOLDER: + +and specify as + +License: MIT + file LICENSE + +Copyright (c) , + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +The MIT License + +Copyright (c) 2008- Attractive Chaos + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS +BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN +ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE.musl as a whole is licensed under the following standard MIT license: + +---------------------------------------------------------------------- +Copyright © 2005-2020 Rich Felker, et al. + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, +TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +---------------------------------------------------------------------- + +Authors/contributors include: + +A. Wilcox +Ada Worcester +Alex Dowad +Alex Suykov +Alexander Monakov +Andre McCurdy +Andrew Kelley +Anthony G. Basile +Aric Belsito +Arvid Picciani +Bartosz Brachaczek +Benjamin Peterson +Bobby Bingham +Boris Brezillon +Brent Cook +Chris Spiegel +Clément Vasseur +Daniel Micay +Daniel Sabogal +Daurnimator +David Carlier +David Edelsohn +Denys Vlasenko +Dmitry Ivanov +Dmitry V. Levin +Drew DeVault +Emil Renner Berthing +Fangrui Song +Felix Fietkau +Felix Janda +Gianluca Anzolin +Hauke Mehrtens +He X +Hiltjo Posthuma +Isaac Dunham +Jaydeep Patil +Jens Gustedt +Jeremy Huntwork +Jo-Philipp Wich +Joakim Sindholt +John Spencer +Julien Ramseier +Justin Cormack +Kaarle Ritvanen +Khem Raj +Kylie McClain +Leah Neukirchen +Luca Barbato +Luka Perkov +M Farkas-Dyck (Strake) +Mahesh Bodapati +Markus Wichmann +Masanori Ogino +Michael Clark +Michael Forney +Mikhail Kremnyov +Natanael Copa +Nicholas J. Kain +orc +Pascal Cuoq +Patrick Oppenlander +Petr Hosek +Petr Skocik +Pierre Carrier +Reini Urban +Rich Felker +Richard Pennington +Ryan Fairfax +Samuel Holland +Segev Finer +Shiz +sin +Solar Designer +Stefan Kristiansson +Stefan O'Rear +Szabolcs Nagy +Timo Teräs +Trutz Behn +Valentin Ochs +Will Dietz +William Haddon +William Pitcock + +Portions of this software are derived from third-party works licensed +under terms compatible with the above MIT license: + +The TRE regular expression implementation (src/regex/reg* and +src/regex/tre*) is Copyright © 2001-2008 Ville Laurikari and licensed +under a 2-clause BSD license (license text in the source files). The +included version has been heavily modified by Rich Felker in 2012, in +the interests of size, simplicity, and namespace cleanliness. + +Much of the math library code (src/math/* and src/complex/*) is +Copyright © 1993,2004 Sun Microsystems or +Copyright © 2003-2011 David Schultz or +Copyright © 2003-2009 Steven G. Kargl or +Copyright © 2003-2009 Bruce D. Evans or +Copyright © 2008 Stephen L. Moshier or +Copyright © 2017-2018 Arm Limited +and labelled as such in comments in the individual source files. All +have been licensed under extremely permissive terms. + +The ARM memcpy code (src/string/arm/memcpy.S) is Copyright © 2008 +The Android Open Source Project and is licensed under a two-clause BSD +license. It was taken from Bionic libc, used on Android. + +The AArch64 memcpy and memset code (src/string/aarch64/*) are +Copyright © 1999-2019, Arm Limited. + +The implementation of DES for crypt (src/crypt/crypt_des.c) is +Copyright © 1994 David Burren. It is licensed under a BSD license. + +The implementation of blowfish crypt (src/crypt/crypt_blowfish.c) was +originally written by Solar Designer and placed into the public +domain. The code also comes with a fallback permissive license for use +in jurisdictions that may not recognize the public domain. + +The smoothsort implementation (src/stdlib/qsort.c) is Copyright © 2011 +Valentin Ochs and is licensed under an MIT-style license. + +The x86_64 port was written by Nicholas J. Kain and is licensed under +the standard MIT terms. + +The mips and microblaze ports were originally written by Richard +Pennington for use in the ellcc project. The original code was adapted +by Rich Felker for build system and code conventions during upstream +integration. It is licensed under the standard MIT terms. + +The mips64 port was contributed by Imagination Technologies and is +licensed under the standard MIT terms. + +The powerpc port was also originally written by Richard Pennington, +and later supplemented and integrated by John Spencer. It is licensed +under the standard MIT terms. + +All other files which have no copyright comments are original works +produced specifically for use as part of this library, written either +by Rich Felker, the main author of the library, or by one or more +contibutors listed above. Details on authorship of individual files +can be found in the git version control history of the project. The +omission of copyright and license comments in each file is in the +interest of source tree size. + +In addition, permission is hereby granted for all public header files +(include/* and arch/*/bits/*) and crt files intended to be linked into +applications (crt/*, ldso/dlstart.c, and arch/*/crt_arch.h) to omit +the copyright notice and permission notice otherwise required by the +license, and to use these files without any requirement of +attribution. These files include substantial contributions from: + +Bobby Bingham +John Spencer +Nicholas J. Kain +Rich Felker +Richard Pennington +Stefan Kristiansson +Szabolcs Nagy + +all of whom have explicitly granted such permission. + +This file previously contained text expressing a belief that most of +the files covered by the above exception were sufficiently trivial not +to be subject to copyright, resulting in confusion over whether it +negated the permissions granted in the license. In the spirit of +permissive licensing, and of not having licensing issues being an +obstacle to adoption, that text has been removed.Copyright (c) 2005-2023, NumPy Developers. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided + with the distribution. + + * Neither the name of the NumPy Developers nor the names of any + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + +Copyright (c) Donald Stufft and individual contributors. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + 1. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.A. HISTORY OF THE SOFTWARE +========================== + +Python was created in the early 1990s by Guido van Rossum at Stichting +Mathematisch Centrum (CWI, see https://www.cwi.nl) in the Netherlands +as a successor of a language called ABC. Guido remains Python's +principal author, although it includes many contributions from others. + +In 1995, Guido continued his work on Python at the Corporation for +National Research Initiatives (CNRI, see https://www.cnri.reston.va.us) +in Reston, Virginia where he released several versions of the +software. + +In May 2000, Guido and the Python core development team moved to +BeOpen.com to form the BeOpen PythonLabs team. In October of the same +year, the PythonLabs team moved to Digital Creations, which became +Zope Corporation. In 2001, the Python Software Foundation (PSF, see +https://www.python.org/psf/) was formed, a non-profit organization +created specifically to own Python-related Intellectual Property. +Zope Corporation was a sponsoring member of the PSF. + +All Python releases are Open Source (see https://opensource.org for +the Open Source Definition). Historically, most, but not all, Python +releases have also been GPL-compatible; the table below summarizes +the various releases. + + Release Derived Year Owner GPL- + from compatible? (1) + + 0.9.0 thru 1.2 1991-1995 CWI yes + 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes + 1.6 1.5.2 2000 CNRI no + 2.0 1.6 2000 BeOpen.com no + 1.6.1 1.6 2001 CNRI yes (2) + 2.1 2.0+1.6.1 2001 PSF no + 2.0.1 2.0+1.6.1 2001 PSF yes + 2.1.1 2.1+2.0.1 2001 PSF yes + 2.1.2 2.1.1 2002 PSF yes + 2.1.3 2.1.2 2002 PSF yes + 2.2 and above 2.1.1 2001-now PSF yes + +Footnotes: + +(1) GPL-compatible doesn't mean that we're distributing Python under + the GPL. All Python licenses, unlike the GPL, let you distribute + a modified version without making your changes open source. The + GPL-compatible licenses make it possible to combine Python with + other software that is released under the GPL; the others don't. + +(2) According to Richard Stallman, 1.6.1 is not GPL-compatible, + because its license has a choice of law clause. According to + CNRI, however, Stallman's lawyer has told CNRI's lawyer that 1.6.1 + is "not incompatible" with the GPL. + +Thanks to the many outside volunteers who have worked under Guido's +direction to make these releases possible. + + +B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON +=============================================================== + +Python software and documentation are licensed under the +Python Software Foundation License Version 2. + +Starting with Python 3.8.6, examples, recipes, and other code in +the documentation are dual licensed under the PSF License Version 2 +and the Zero-Clause BSD license. + +Some software incorporated into Python is under different licenses. +The licenses are listed with code falling under that license. + + +PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 +-------------------------------------------- + +1. This LICENSE AGREEMENT is between the Python Software Foundation +("PSF"), and the Individual or Organization ("Licensee") accessing and +otherwise using this software ("Python") in source or binary form and +its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, PSF hereby +grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, +analyze, test, perform and/or display publicly, prepare derivative works, +distribute, and otherwise use Python alone or in any derivative version, +provided, however, that PSF's License Agreement and PSF's notice of copyright, +i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, +2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023 Python Software Foundation; +All Rights Reserved" are retained in Python alone or in any derivative version +prepared by Licensee. + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python. + +4. PSF is making Python available to Licensee on an "AS IS" +basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. Nothing in this License Agreement shall be deemed to create any +relationship of agency, partnership, or joint venture between PSF and +Licensee. This License Agreement does not grant permission to use PSF +trademarks or trade name in a trademark sense to endorse or promote +products or services of Licensee, or any third party. + +8. By copying, installing or otherwise using Python, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0 +------------------------------------------- + +BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1 + +1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an +office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the +Individual or Organization ("Licensee") accessing and otherwise using +this software in source or binary form and its associated +documentation ("the Software"). + +2. Subject to the terms and conditions of this BeOpen Python License +Agreement, BeOpen hereby grants Licensee a non-exclusive, +royalty-free, world-wide license to reproduce, analyze, test, perform +and/or display publicly, prepare derivative works, distribute, and +otherwise use the Software alone or in any derivative version, +provided, however, that the BeOpen Python License is retained in the +Software, alone or in any derivative version prepared by Licensee. + +3. BeOpen is making the Software available to Licensee on an "AS IS" +basis. BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE +SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS +AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY +DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +5. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +6. This License Agreement shall be governed by and interpreted in all +respects by the law of the State of California, excluding conflict of +law provisions. Nothing in this License Agreement shall be deemed to +create any relationship of agency, partnership, or joint venture +between BeOpen and Licensee. This License Agreement does not grant +permission to use BeOpen trademarks or trade names in a trademark +sense to endorse or promote products or services of Licensee, or any +third party. As an exception, the "BeOpen Python" logos available at +http://www.pythonlabs.com/logos.html may be used according to the +permissions granted on that web page. + +7. By copying, installing or otherwise using the software, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1 +--------------------------------------- + +1. This LICENSE AGREEMENT is between the Corporation for National +Research Initiatives, having an office at 1895 Preston White Drive, +Reston, VA 20191 ("CNRI"), and the Individual or Organization +("Licensee") accessing and otherwise using Python 1.6.1 software in +source or binary form and its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, CNRI +hereby grants Licensee a nonexclusive, royalty-free, world-wide +license to reproduce, analyze, test, perform and/or display publicly, +prepare derivative works, distribute, and otherwise use Python 1.6.1 +alone or in any derivative version, provided, however, that CNRI's +License Agreement and CNRI's notice of copyright, i.e., "Copyright (c) +1995-2001 Corporation for National Research Initiatives; All Rights +Reserved" are retained in Python 1.6.1 alone or in any derivative +version prepared by Licensee. Alternately, in lieu of CNRI's License +Agreement, Licensee may substitute the following text (omitting the +quotes): "Python 1.6.1 is made available subject to the terms and +conditions in CNRI's License Agreement. This Agreement together with +Python 1.6.1 may be located on the internet using the following +unique, persistent identifier (known as a handle): 1895.22/1013. This +Agreement may also be obtained from a proxy server on the internet +using the following URL: http://hdl.handle.net/1895.22/1013". + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python 1.6.1 or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python 1.6.1. + +4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS" +basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1 WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. This License Agreement shall be governed by the federal +intellectual property law of the United States, including without +limitation the federal copyright law, and, to the extent such +U.S. federal law does not apply, by the law of the Commonwealth of +Virginia, excluding Virginia's conflict of law provisions. +Notwithstanding the foregoing, with regard to derivative works based +on Python 1.6.1 that incorporate non-separable material that was +previously distributed under the GNU General Public License (GPL), the +law of the Commonwealth of Virginia shall govern this License +Agreement only as to issues arising under or with respect to +Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this +License Agreement shall be deemed to create any relationship of +agency, partnership, or joint venture between CNRI and Licensee. This +License Agreement does not grant permission to use CNRI trademarks or +trade name in a trademark sense to endorse or promote products or +services of Licensee, or any third party. + +8. By clicking on the "ACCEPT" button where indicated, or by copying, +installing or otherwise using Python 1.6.1, Licensee agrees to be +bound by the terms and conditions of this License Agreement. + + ACCEPT + + +CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2 +-------------------------------------------------- + +Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, +The Netherlands. All rights reserved. + +Permission to use, copy, modify, and distribute this software and its +documentation for any purpose and without fee is hereby granted, +provided that the above copyright notice appear in all copies and that +both that copyright notice and this permission notice appear in +supporting documentation, and that the name of Stichting Mathematisch +Centrum or CWI not be used in advertising or publicity pertaining to +distribution of the software without specific, written prior +permission. + +STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO +THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND +FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE +FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT +OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + +ZERO-CLAUSE BSD LICENSE FOR CODE IN THE PYTHON DOCUMENTATION +---------------------------------------------------------------------- + +Permission to use, copy, modify, and/or distribute this software for any +purpose with or without fee is hereby granted. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH +REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY +AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, +INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM +LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR +OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR +PERFORMANCE OF THIS SOFTWARE. +Copyright (c) 2014, Al Sweigart +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the {organization} nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.Copyright (c) 2017 Anthony Sottile + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE.Copyright (c) 2015-2019 Jared Hobbs + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE.Developed by ESN, an Electronic Arts Inc. studio. +Copyright (c) 2014, Electronic Arts Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: +* Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. +* Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. +* Neither the name of ESN, Electronic Arts Inc. nor the +names of its contributors may be used to endorse or promote products +derived from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL ELECTRONIC ARTS INC. BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +---- + +Portions of code from MODP_ASCII - Ascii transformations (upper/lower, etc) +https://github.com/client9/stringencoders + + Copyright 2005, 2006, 2007 + Nick Galbreath -- nickg [at] modp [dot] com + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are + met: + + Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + + Neither the name of the modp.com nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + This is the standard "new" BSD license: + http://www.opensource.org/licenses/bsd-license.php + +https://github.com/client9/stringencoders/blob/cfd5c1507325ae497ea9bacdacba12c0ffd79d30/COPYING + +---- + +Numeric decoder derived from from TCL library +https://opensource.apple.com/source/tcl/tcl-14/tcl/license.terms + * Copyright (c) 1988-1993 The Regents of the University of California. + * Copyright (c) 1994 Sun Microsystems, Inc. + + This software is copyrighted by the Regents of the University of + California, Sun Microsystems, Inc., Scriptics Corporation, ActiveState + Corporation and other parties. The following terms apply to all files + associated with the software unless explicitly disclaimed in + individual files. + + The authors hereby grant permission to use, copy, modify, distribute, + and license this software and its documentation for any purpose, provided + that existing copyright notices are retained in all copies and that this + notice is included verbatim in any distributions. No written agreement, + license, or royalty fee is required for any of the authorized uses. + Modifications to this software may be copyrighted by their authors + and need not follow the licensing terms described here, provided that + the new terms are clearly indicated on the first page of each file where + they apply. + + IN NO EVENT SHALL THE AUTHORS OR DISTRIBUTORS BE LIABLE TO ANY PARTY + FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES + ARISING OUT OF THE USE OF THIS SOFTWARE, ITS DOCUMENTATION, OR ANY + DERIVATIVES THEREOF, EVEN IF THE AUTHORS HAVE BEEN ADVISED OF THE + POSSIBILITY OF SUCH DAMAGE. + + THE AUTHORS AND DISTRIBUTORS SPECIFICALLY DISCLAIM ANY WARRANTIES, + INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT. THIS SOFTWARE + IS PROVIDED ON AN "AS IS" BASIS, AND THE AUTHORS AND DISTRIBUTORS HAVE + NO OBLIGATION TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR + MODIFICATIONS. + + GOVERNMENT USE: If you are acquiring this software on behalf of the + U.S. government, the Government shall have only "Restricted Rights" + in the software and related documentation as defined in the Federal + Acquisition Regulations (FARs) in Clause 52.227.19 (c) (2). If you + are acquiring the software on behalf of the Department of Defense, the + software shall be classified as "Commercial Computer Software" and the + Government shall have only "Restricted Rights" as defined in Clause + 252.227-7013 (c) (1) of DFARs. Notwithstanding the foregoing, the + authors grant the U.S. Government and others acting in its behalf + permission to use and distribute the software in accordance with the + terms specified in this license.Apache License +Version 2.0, January 2004 +http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + +"License" shall mean the terms and conditions for use, reproduction, and +distribution as defined by Sections 1 through 9 of this document. + +"Licensor" shall mean the copyright owner or entity authorized by the copyright +owner that is granting the License. + +"Legal Entity" shall mean the union of the acting entity and all other entities +that control, are controlled by, or are under common control with that entity. +For the purposes of this definition, "control" means (i) the power, direct or +indirect, to cause the direction or management of such entity, whether by +contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the +outstanding shares, or (iii) beneficial ownership of such entity. + +"You" (or "Your") shall mean an individual or Legal Entity exercising +permissions granted by this License. + +"Source" form shall mean the preferred form for making modifications, including +but not limited to software source code, documentation source, and configuration +files. + +"Object" form shall mean any form resulting from mechanical transformation or +translation of a Source form, including but not limited to compiled object code, +generated documentation, and conversions to other media types. + +"Work" shall mean the work of authorship, whether in Source or Object form, made +available under the License, as indicated by a copyright notice that is included +in or attached to the work (an example is provided in the Appendix below). + +"Derivative Works" shall mean any work, whether in Source or Object form, that +is based on (or derived from) the Work and for which the editorial revisions, +annotations, elaborations, or other modifications represent, as a whole, an +original work of authorship. For the purposes of this License, Derivative Works +shall not include works that remain separable from, or merely link (or bind by +name) to the interfaces of, the Work and Derivative Works thereof. + +"Contribution" shall mean any work of authorship, including the original version +of the Work and any modifications or additions to that Work or Derivative Works +thereof, that is intentionally submitted to Licensor for inclusion in the Work +by the copyright owner or by an individual or Legal Entity authorized to submit +on behalf of the copyright owner. For the purposes of this definition, +"submitted" means any form of electronic, verbal, or written communication sent +to the Licensor or its representatives, including but not limited to +communication on electronic mailing lists, source code control systems, and +issue tracking systems that are managed by, or on behalf of, the Licensor for +the purpose of discussing and improving the Work, but excluding communication +that is conspicuously marked or otherwise designated in writing by the copyright +owner as "Not a Contribution." + +"Contributor" shall mean Licensor and any individual or Legal Entity on behalf +of whom a Contribution has been received by Licensor and subsequently +incorporated within the Work. + +2. Grant of Copyright License. + +Subject to the terms and conditions of this License, each Contributor hereby +grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, +irrevocable copyright license to reproduce, prepare Derivative Works of, +publicly display, publicly perform, sublicense, and distribute the Work and such +Derivative Works in Source or Object form. + +3. Grant of Patent License. + +Subject to the terms and conditions of this License, each Contributor hereby +grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, +irrevocable (except as stated in this section) patent license to make, have +made, use, offer to sell, sell, import, and otherwise transfer the Work, where +such license applies only to those patent claims licensable by such Contributor +that are necessarily infringed by their Contribution(s) alone or by combination +of their Contribution(s) with the Work to which such Contribution(s) was +submitted. If You institute patent litigation against any entity (including a +cross-claim or counterclaim in a lawsuit) alleging that the Work or a +Contribution incorporated within the Work constitutes direct or contributory +patent infringement, then any patent licenses granted to You under this License +for that Work shall terminate as of the date such litigation is filed. + +4. Redistribution. + +You may reproduce and distribute copies of the Work or Derivative Works thereof +in any medium, with or without modifications, and in Source or Object form, +provided that You meet the following conditions: + +You must give any other recipients of the Work or Derivative Works a copy of +this License; and +You must cause any modified files to carry prominent notices stating that You +changed the files; and +You must retain, in the Source form of any Derivative Works that You distribute, +all copyright, patent, trademark, and attribution notices from the Source form +of the Work, excluding those notices that do not pertain to any part of the +Derivative Works; and +If the Work includes a "NOTICE" text file as part of its distribution, then any +Derivative Works that You distribute must include a readable copy of the +attribution notices contained within such NOTICE file, excluding those notices +that do not pertain to any part of the Derivative Works, in at least one of the +following places: within a NOTICE text file distributed as part of the +Derivative Works; within the Source form or documentation, if provided along +with the Derivative Works; or, within a display generated by the Derivative +Works, if and wherever such third-party notices normally appear. The contents of +the NOTICE file are for informational purposes only and do not modify the +License. You may add Your own attribution notices within Derivative Works that +You distribute, alongside or as an addendum to the NOTICE text from the Work, +provided that such additional attribution notices cannot be construed as +modifying the License. +You may add Your own copyright statement to Your modifications and may provide +additional or different license terms and conditions for use, reproduction, or +distribution of Your modifications, or for any such Derivative Works as a whole, +provided Your use, reproduction, and distribution of the Work otherwise complies +with the conditions stated in this License. + +5. Submission of Contributions. + +Unless You explicitly state otherwise, any Contribution intentionally submitted +for inclusion in the Work by You to the Licensor shall be under the terms and +conditions of this License, without any additional terms or conditions. +Notwithstanding the above, nothing herein shall supersede or modify the terms of +any separate license agreement you may have executed with Licensor regarding +such Contributions. + +6. Trademarks. + +This License does not grant permission to use the trade names, trademarks, +service marks, or product names of the Licensor, except as required for +reasonable and customary use in describing the origin of the Work and +reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. + +Unless required by applicable law or agreed to in writing, Licensor provides the +Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, +including, without limitation, any warranties or conditions of TITLE, +NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are +solely responsible for determining the appropriateness of using or +redistributing the Work and assume any risks associated with Your exercise of +permissions under this License. + +8. Limitation of Liability. + +In no event and under no legal theory, whether in tort (including negligence), +contract, or otherwise, unless required by applicable law (such as deliberate +and grossly negligent acts) or agreed to in writing, shall any Contributor be +liable to You for damages, including any direct, indirect, special, incidental, +or consequential damages of any character arising as a result of this License or +out of the use or inability to use the Work (including but not limited to +damages for loss of goodwill, work stoppage, computer failure or malfunction, or +any and all other commercial damages or losses), even if such Contributor has +been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. + +While redistributing the Work or Derivative Works thereof, You may choose to +offer, and charge a fee for, acceptance of support, warranty, indemnity, or +other liability obligations and/or rights consistent with this License. However, +in accepting such obligations, You may act only on Your own behalf and on Your +sole responsibility, not on behalf of any other Contributor, and only if You +agree to indemnify, defend, and hold each Contributor harmless for any liability +incurred by, or claims asserted against, such Contributor by reason of your +accepting any such warranty or additional liability. + +END OF TERMS AND CONDITIONS + +APPENDIX: How to apply the Apache License to your work + +To apply the Apache License to your work, attach the following boilerplate +notice, with the fields enclosed by brackets "[]" replaced with your own +identifying information. (Don't include the brackets!) The text should be +enclosed in the appropriate comment syntax for the file format. We also +recommend that a file or class name and description of purpose be included on +the same "printed page" as the copyright notice for easier identification within +third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +pandocfilters +BSD License +http://github.com/jgm/pandocfilters +Copyright (c) 2013, John MacFarlane +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + - Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + - Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + - Neither the name of John Macfarlane nor the names of its contributors may + be used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +parse +MIT +https://github.com/r1chardj0n3s/parse +Copyright (c) 2012-2019 Richard Jones + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +parso +MIT License +https://github.com/davidhalter/parso +All contributions towards parso are MIT licensed. + +Some Python files have been taken from the standard library and are therefore +PSF licensed. Modifications on these files are dual licensed (both MIT and +PSF). These files are: + +- parso/pgen2/* +- parso/tokenize.py +- parso/token.py +- test/test_pgen2.py + +Also some test files under test/normalizer_issue_files have been copied from +https://github.com/PyCQA/pycodestyle (Expat License == MIT License). + +------------------------------------------------------------------------------- +The MIT License (MIT) + +Copyright (c) <2013-2017> + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +------------------------------------------------------------------------------- + +PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 +-------------------------------------------- + +1. This LICENSE AGREEMENT is between the Python Software Foundation +("PSF"), and the Individual or Organization ("Licensee") accessing and +otherwise using this software ("Python") in source or binary form and +its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, PSF hereby +grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, +analyze, test, perform and/or display publicly, prepare derivative works, +distribute, and otherwise use Python alone or in any derivative version, +provided, however, that PSF's License Agreement and PSF's notice of copyright, +i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, +2011, 2012, 2013, 2014, 2015 Python Software Foundation; All Rights Reserved" +are retained in Python alone or in any derivative version prepared by Licensee. + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python. + +4. PSF is making Python available to Licensee on an "AS IS" +basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. Nothing in this License Agreement shall be deemed to create any +relationship of agency, partnership, or joint venture between PSF and +Licensee. This License Agreement does not grant permission to use PSF +trademarks or trade name in a trademark sense to endorse or promote +products or services of Licensee, or any third party. + +8. By copying, installing or otherwise using Python, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +pathspec +Mozilla Public License 2.0 (MPL 2.0) +UNKNOWN +Mozilla Public License Version 2.0 +================================== + +1. Definitions +-------------- + +1.1. "Contributor" + means each individual or legal entity that creates, contributes to + the creation of, or owns Covered Software. + +1.2. "Contributor Version" + means the combination of the Contributions of others (if any) used + by a Contributor and that particular Contributor's Contribution. + +1.3. "Contribution" + means Covered Software of a particular Contributor. + +1.4. "Covered Software" + means Source Code Form to which the initial Contributor has attached + the notice in Exhibit A, the Executable Form of such Source Code + Form, and Modifications of such Source Code Form, in each case + including portions thereof. + +1.5. "Incompatible With Secondary Licenses" + means + + (a) that the initial Contributor has attached the notice described + in Exhibit B to the Covered Software; or + + (b) that the Covered Software was made available under the terms of + version 1.1 or earlier of the License, but not also under the + terms of a Secondary License. + +1.6. "Executable Form" + means any form of the work other than Source Code Form. + +1.7. "Larger Work" + means a work that combines Covered Software with other material, in + a separate file or files, that is not Covered Software. + +1.8. "License" + means this document. + +1.9. "Licensable" + means having the right to grant, to the maximum extent possible, + whether at the time of the initial grant or subsequently, any and + all of the rights conveyed by this License. + +1.10. "Modifications" + means any of the following: + + (a) any file in Source Code Form that results from an addition to, + deletion from, or modification of the contents of Covered + Software; or + + (b) any new file in Source Code Form that contains any Covered + Software. + +1.11. "Patent Claims" of a Contributor + means any patent claim(s), including without limitation, method, + process, and apparatus claims, in any patent Licensable by such + Contributor that would be infringed, but for the grant of the + License, by the making, using, selling, offering for sale, having + made, import, or transfer of either its Contributions or its + Contributor Version. + +1.12. "Secondary License" + means either the GNU General Public License, Version 2.0, the GNU + Lesser General Public License, Version 2.1, the GNU Affero General + Public License, Version 3.0, or any later versions of those + licenses. + +1.13. "Source Code Form" + means the form of the work preferred for making modifications. + +1.14. "You" (or "Your") + means an individual or a legal entity exercising rights under this + License. For legal entities, "You" includes any entity that + controls, is controlled by, or is under common control with You. For + purposes of this definition, "control" means (a) the power, direct + or indirect, to cause the direction or management of such entity, + whether by contract or otherwise, or (b) ownership of more than + fifty percent (50%) of the outstanding shares or beneficial + ownership of such entity. + +2. License Grants and Conditions +-------------------------------- + +2.1. Grants + +Each Contributor hereby grants You a world-wide, royalty-free, +non-exclusive license: + +(a) under intellectual property rights (other than patent or trademark) + Licensable by such Contributor to use, reproduce, make available, + modify, display, perform, distribute, and otherwise exploit its + Contributions, either on an unmodified basis, with Modifications, or + as part of a Larger Work; and + +(b) under Patent Claims of such Contributor to make, use, sell, offer + for sale, have made, import, and otherwise transfer either its + Contributions or its Contributor Version. + +2.2. Effective Date + +The licenses granted in Section 2.1 with respect to any Contribution +become effective for each Contribution on the date the Contributor first +distributes such Contribution. + +2.3. Limitations on Grant Scope + +The licenses granted in this Section 2 are the only rights granted under +this License. No additional rights or licenses will be implied from the +distribution or licensing of Covered Software under this License. +Notwithstanding Section 2.1(b) above, no patent license is granted by a +Contributor: + +(a) for any code that a Contributor has removed from Covered Software; + or + +(b) for infringements caused by: (i) Your and any other third party's + modifications of Covered Software, or (ii) the combination of its + Contributions with other software (except as part of its Contributor + Version); or + +(c) under Patent Claims infringed by Covered Software in the absence of + its Contributions. + +This License does not grant any rights in the trademarks, service marks, +or logos of any Contributor (except as may be necessary to comply with +the notice requirements in Section 3.4). + +2.4. Subsequent Licenses + +No Contributor makes additional grants as a result of Your choice to +distribute the Covered Software under a subsequent version of this +License (see Section 10.2) or under the terms of a Secondary License (if +permitted under the terms of Section 3.3). + +2.5. Representation + +Each Contributor represents that the Contributor believes its +Contributions are its original creation(s) or it has sufficient rights +to grant the rights to its Contributions conveyed by this License. + +2.6. Fair Use + +This License is not intended to limit any rights You have under +applicable copyright doctrines of fair use, fair dealing, or other +equivalents. + +2.7. Conditions + +Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted +in Section 2.1. + +3. Responsibilities +------------------- + +3.1. Distribution of Source Form + +All distribution of Covered Software in Source Code Form, including any +Modifications that You create or to which You contribute, must be under +the terms of this License. You must inform recipients that the Source +Code Form of the Covered Software is governed by the terms of this +License, and how they can obtain a copy of this License. You may not +attempt to alter or restrict the recipients' rights in the Source Code +Form. + +3.2. Distribution of Executable Form + +If You distribute Covered Software in Executable Form then: + +(a) such Covered Software must also be made available in Source Code + Form, as described in Section 3.1, and You must inform recipients of + the Executable Form how they can obtain a copy of such Source Code + Form by reasonable means in a timely manner, at a charge no more + than the cost of distribution to the recipient; and + +(b) You may distribute such Executable Form under the terms of this + License, or sublicense it under different terms, provided that the + license for the Executable Form does not attempt to limit or alter + the recipients' rights in the Source Code Form under this License. + +3.3. Distribution of a Larger Work + +You may create and distribute a Larger Work under terms of Your choice, +provided that You also comply with the requirements of this License for +the Covered Software. If the Larger Work is a combination of Covered +Software with a work governed by one or more Secondary Licenses, and the +Covered Software is not Incompatible With Secondary Licenses, this +License permits You to additionally distribute such Covered Software +under the terms of such Secondary License(s), so that the recipient of +the Larger Work may, at their option, further distribute the Covered +Software under the terms of either this License or such Secondary +License(s). + +3.4. Notices + +You may not remove or alter the substance of any license notices +(including copyright notices, patent notices, disclaimers of warranty, +or limitations of liability) contained within the Source Code Form of +the Covered Software, except that You may alter any license notices to +the extent required to remedy known factual inaccuracies. + +3.5. Application of Additional Terms + +You may choose to offer, and to charge a fee for, warranty, support, +indemnity or liability obligations to one or more recipients of Covered +Software. However, You may do so only on Your own behalf, and not on +behalf of any Contributor. You must make it absolutely clear that any +such warranty, support, indemnity, or liability obligation is offered by +You alone, and You hereby agree to indemnify every Contributor for any +liability incurred by such Contributor as a result of warranty, support, +indemnity or liability terms You offer. You may include additional +disclaimers of warranty and limitations of liability specific to any +jurisdiction. + +4. Inability to Comply Due to Statute or Regulation +--------------------------------------------------- + +If it is impossible for You to comply with any of the terms of this +License with respect to some or all of the Covered Software due to +statute, judicial order, or regulation then You must: (a) comply with +the terms of this License to the maximum extent possible; and (b) +describe the limitations and the code they affect. Such description must +be placed in a text file included with all distributions of the Covered +Software under this License. Except to the extent prohibited by statute +or regulation, such description must be sufficiently detailed for a +recipient of ordinary skill to be able to understand it. + +5. Termination +-------------- + +5.1. The rights granted under this License will terminate automatically +if You fail to comply with any of its terms. However, if You become +compliant, then the rights granted under this License from a particular +Contributor are reinstated (a) provisionally, unless and until such +Contributor explicitly and finally terminates Your grants, and (b) on an +ongoing basis, if such Contributor fails to notify You of the +non-compliance by some reasonable means prior to 60 days after You have +come back into compliance. Moreover, Your grants from a particular +Contributor are reinstated on an ongoing basis if such Contributor +notifies You of the non-compliance by some reasonable means, this is the +first time You have received notice of non-compliance with this License +from such Contributor, and You become compliant prior to 30 days after +Your receipt of the notice. + +5.2. If You initiate litigation against any entity by asserting a patent +infringement claim (excluding declaratory judgment actions, +counter-claims, and cross-claims) alleging that a Contributor Version +directly or indirectly infringes any patent, then the rights granted to +You by any and all Contributors for the Covered Software under Section +2.1 of this License shall terminate. + +5.3. In the event of termination under Sections 5.1 or 5.2 above, all +end user license agreements (excluding distributors and resellers) which +have been validly granted by You or Your distributors under this License +prior to termination shall survive termination. + +************************************************************************ +* * +* 6. Disclaimer of Warranty * +* ------------------------- * +* * +* Covered Software is provided under this License on an "as is" * +* basis, without warranty of any kind, either expressed, implied, or * +* statutory, including, without limitation, warranties that the * +* Covered Software is free of defects, merchantable, fit for a * +* particular purpose or non-infringing. The entire risk as to the * +* quality and performance of the Covered Software is with You. * +* Should any Covered Software prove defective in any respect, You * +* (not any Contributor) assume the cost of any necessary servicing, * +* repair, or correction. This disclaimer of warranty constitutes an * +* essential part of this License. No use of any Covered Software is * +* authorized under this License except under this disclaimer. * +* * +************************************************************************ + +************************************************************************ +* * +* 7. Limitation of Liability * +* -------------------------- * +* * +* Under no circumstances and under no legal theory, whether tort * +* (including negligence), contract, or otherwise, shall any * +* Contributor, or anyone who distributes Covered Software as * +* permitted above, be liable to You for any direct, indirect, * +* special, incidental, or consequential damages of any character * +* including, without limitation, damages for lost profits, loss of * +* goodwill, work stoppage, computer failure or malfunction, or any * +* and all other commercial damages or losses, even if such party * +* shall have been informed of the possibility of such damages. This * +* limitation of liability shall not apply to liability for death or * +* personal injury resulting from such party's negligence to the * +* extent applicable law prohibits such limitation. Some * +* jurisdictions do not allow the exclusion or limitation of * +* incidental or consequential damages, so this exclusion and * +* limitation may not apply to You. * +* * +************************************************************************ + +8. Litigation +------------- + +Any litigation relating to this License may be brought only in the +courts of a jurisdiction where the defendant maintains its principal +place of business and such litigation shall be governed by laws of that +jurisdiction, without reference to its conflict-of-law provisions. +Nothing in this Section shall prevent a party's ability to bring +cross-claims or counter-claims. + +9. Miscellaneous +---------------- + +This License represents the complete agreement concerning the subject +matter hereof. If any provision of this License is held to be +unenforceable, such provision shall be reformed only to the extent +necessary to make it enforceable. Any law or regulation which provides +that the language of a contract shall be construed against the drafter +shall not be used to construe this License against a Contributor. + +10. Versions of the License +--------------------------- + +10.1. New Versions + +Mozilla Foundation is the license steward. Except as provided in Section +10.3, no one other than the license steward has the right to modify or +publish new versions of this License. Each version will be given a +distinguishing version number. + +10.2. Effect of New Versions + +You may distribute the Covered Software under the terms of the version +of the License under which You originally received the Covered Software, +or under the terms of any subsequent version published by the license +steward. + +10.3. Modified Versions + +If you create software not governed by this License, and you want to +create a new license for such software, you may create and use a +modified version of this License if you rename the license and remove +any references to the name of the license steward (except to note that +such modified license differs from this License). + +10.4. Distributing Source Code Form that is Incompatible With Secondary +Licenses + +If You choose to distribute Source Code Form that is Incompatible With +Secondary Licenses under the terms of this version of the License, the +notice described in Exhibit B of this License must be attached. + +Exhibit A - Source Code Form License Notice +------------------------------------------- + + This Source Code Form is subject to the terms of the Mozilla Public + License, v. 2.0. If a copy of the MPL was not distributed with this + file, You can obtain one at http://mozilla.org/MPL/2.0/. + +If it is not possible or desirable to put the notice in a particular +file, then You may include the notice in a location (such as a LICENSE +file in a relevant directory) where a recipient would be likely to look +for such a notice. + +You may add additional accurate notices of copyright ownership. + +Exhibit B - "Incompatible With Secondary Licenses" Notice +--------------------------------------------------------- + + This Source Code Form is "Incompatible With Secondary Licenses", as + defined by the Mozilla Public License, v. 2.0. + + +peft +Apache Software License +https://github.com/huggingface/peft + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +pexpect +ISC License (ISCL) +https://pexpect.readthedocs.io/ +ISC LICENSE + + This license is approved by the OSI and FSF as GPL-compatible. + http://opensource.org/licenses/isc-license.txt + + Copyright (c) 2013-2014, Pexpect development team + Copyright (c) 2012, Noah Spurrier + + Permission to use, copy, modify, and/or distribute this software for any + purpose with or without fee is hereby granted, provided that the above + copyright notice and this permission notice appear in all copies. + + THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + + + +pillow +MIT-CMU +https://python-pillow.github.io +The Python Imaging Library (PIL) is + + Copyright © 1997-2011 by Secret Labs AB + Copyright © 1995-2011 by Fredrik Lundh and contributors + +Pillow is the friendly PIL fork. It is + + Copyright © 2010 by Jeffrey A. Clark and contributors + +Like PIL, Pillow is licensed under the open source MIT-CMU License: + +By obtaining, using, and/or copying this software and/or its associated +documentation, you agree that you have read, understood, and will comply +with the following terms and conditions: + +Permission to use, copy, modify and distribute this software and its +documentation for any purpose and without fee is hereby granted, +provided that the above copyright notice appears in all copies, and that +both that copyright notice and this permission notice appear in supporting +documentation, and that the name of Secret Labs AB or the author not be +used in advertising or publicity pertaining to distribution of the software +without specific, written prior permission. + +SECRET LABS AB AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS +SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. +IN NO EVENT SHALL SECRET LABS AB OR THE AUTHOR BE LIABLE FOR ANY SPECIAL, +INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM +LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE +OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR +PERFORMANCE OF THIS SOFTWARE. + + +---- + +AOM + +Copyright (c) 2016, Alliance for Open Media. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS +FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE +COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, +BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN +ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + + +---- + +BROTLI + +Copyright (c) 2009, 2010, 2013-2016 by the Brotli Authors. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +---- + +BZIP2 + + +-------------------------------------------------------------------------- + +This program, "bzip2", the associated library "libbzip2", and all +documentation, are copyright (C) 1996-2019 Julian R Seward. All +rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. The origin of this software must not be misrepresented; you must + not claim that you wrote the original software. If you use this + software in a product, an acknowledgment in the product + documentation would be appreciated but is not required. + +3. Altered source versions must be plainly marked as such, and must + not be misrepresented as being the original software. + +4. The name of the author may not be used to endorse or promote + products derived from this software without specific prior written + permission. + +THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS +OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE +GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, +WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Julian Seward, jseward@acm.org +bzip2/libbzip2 version 1.0.8 of 13 July 2019 + +-------------------------------------------------------------------------- + + +---- + +DAV1D + +Copyright © 2018-2019, VideoLAN and dav1d authors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +---- + +FREETYPE2 + +The FreeType 2 font engine is copyrighted work and cannot be used +legally without a software license. In order to make this project +usable to a vast majority of developers, we distribute it under two +mutually exclusive open-source licenses. + +This means that *you* must choose *one* of the two licenses described +below, then obey all its terms and conditions when using FreeType 2 in +any of your projects or products. + + - The FreeType License, found in the file `docs/FTL.TXT`, which is + similar to the original BSD license *with* an advertising clause + that forces you to explicitly cite the FreeType project in your + product's documentation. All details are in the license file. + This license is suited to products which don't use the GNU General + Public License. + + Note that this license is compatible to the GNU General Public + License version 3, but not version 2. + + - The GNU General Public License version 2, found in + `docs/GPLv2.TXT` (any later version can be used also), for + programs which already use the GPL. Note that the FTL is + incompatible with GPLv2 due to its advertisement clause. + +The contributed BDF and PCF drivers come with a license similar to +that of the X Window System. It is compatible to the above two +licenses (see files `src/bdf/README` and `src/pcf/README`). The same +holds for the source code files `src/base/fthash.c` and +`include/freetype/internal/fthash.h`; they were part of the BDF driver +in earlier FreeType versions. + +The gzip module uses the zlib license (see `src/gzip/zlib.h`) which +too is compatible to the above two licenses. + +The files `src/autofit/ft-hb.c` and `src/autofit/ft-hb.h` contain code +taken almost verbatim from the HarfBuzz file `hb-ft.cc`, which uses +the 'Old MIT' license, compatible to the above two licenses. + +The MD5 checksum support (only used for debugging in development +builds) is in the public domain. + +-------------------------------------------------------------------------- + + The FreeType Project LICENSE + ---------------------------- + + 2006-Jan-27 + + Copyright 1996-2002, 2006 by + David Turner, Robert Wilhelm, and Werner Lemberg + + + +Introduction +============ + + The FreeType Project is distributed in several archive packages; + some of them may contain, in addition to the FreeType font engine, + various tools and contributions which rely on, or relate to, the + FreeType Project. + + This license applies to all files found in such packages, and + which do not fall under their own explicit license. The license + affects thus the FreeType font engine, the test programs, + documentation and makefiles, at the very least. + + This license was inspired by the BSD, Artistic, and IJG + (Independent JPEG Group) licenses, which all encourage inclusion + and use of free software in commercial and freeware products + alike. As a consequence, its main points are that: + + o We don't promise that this software works. However, we will be + interested in any kind of bug reports. (`as is' distribution) + + o You can use this software for whatever you want, in parts or + full form, without having to pay us. (`royalty-free' usage) + + o You may not pretend that you wrote this software. If you use + it, or only parts of it, in a program, you must acknowledge + somewhere in your documentation that you have used the + FreeType code. (`credits') + + We specifically permit and encourage the inclusion of this + software, with or without modifications, in commercial products. + We disclaim all warranties covering The FreeType Project and + assume no liability related to The FreeType Project. + + + Finally, many people asked us for a preferred form for a + credit/disclaimer to use in compliance with this license. We thus + encourage you to use the following text: + + """ + Portions of this software are copyright © The FreeType + Project (www.freetype.org). All rights reserved. + """ + + Please replace with the value from the FreeType version you + actually use. + + +Legal Terms +=========== + +0. Definitions +-------------- + + Throughout this license, the terms `package', `FreeType Project', + and `FreeType archive' refer to the set of files originally + distributed by the authors (David Turner, Robert Wilhelm, and + Werner Lemberg) as the `FreeType Project', be they named as alpha, + beta or final release. + + `You' refers to the licensee, or person using the project, where + `using' is a generic term including compiling the project's source + code as well as linking it to form a `program' or `executable'. + This program is referred to as `a program using the FreeType + engine'. + + This license applies to all files distributed in the original + FreeType Project, including all source code, binaries and + documentation, unless otherwise stated in the file in its + original, unmodified form as distributed in the original archive. + If you are unsure whether or not a particular file is covered by + this license, you must contact us to verify this. + + The FreeType Project is copyright (C) 1996-2000 by David Turner, + Robert Wilhelm, and Werner Lemberg. All rights reserved except as + specified below. + +1. No Warranty +-------------- + + THE FREETYPE PROJECT IS PROVIDED `AS IS' WITHOUT WARRANTY OF ANY + KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, + WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + PURPOSE. IN NO EVENT WILL ANY OF THE AUTHORS OR COPYRIGHT HOLDERS + BE LIABLE FOR ANY DAMAGES CAUSED BY THE USE OR THE INABILITY TO + USE, OF THE FREETYPE PROJECT. + +2. Redistribution +----------------- + + This license grants a worldwide, royalty-free, perpetual and + irrevocable right and license to use, execute, perform, compile, + display, copy, create derivative works of, distribute and + sublicense the FreeType Project (in both source and object code + forms) and derivative works thereof for any purpose; and to + authorize others to exercise some or all of the rights granted + herein, subject to the following conditions: + + o Redistribution of source code must retain this license file + (`FTL.TXT') unaltered; any additions, deletions or changes to + the original files must be clearly indicated in accompanying + documentation. The copyright notices of the unaltered, + original files must be preserved in all copies of source + files. + + o Redistribution in binary form must provide a disclaimer that + states that the software is based in part of the work of the + FreeType Team, in the distribution documentation. We also + encourage you to put an URL to the FreeType web page in your + documentation, though this isn't mandatory. + + These conditions apply to any software derived from or based on + the FreeType Project, not just the unmodified files. If you use + our work, you must acknowledge us. However, no fee need be paid + to us. + +3. Advertising +-------------- + + Neither the FreeType authors and contributors nor you shall use + the name of the other for commercial, advertising, or promotional + purposes without specific prior written permission. + + We suggest, but do not require, that you use one or more of the + following phrases to refer to this software in your documentation + or advertising materials: `FreeType Project', `FreeType Engine', + `FreeType library', or `FreeType Distribution'. + + As you have not signed this license, you are not required to + accept it. However, as the FreeType Project is copyrighted + material, only this license, or another one contracted with the + authors, grants you the right to use, distribute, and modify it. + Therefore, by using, distributing, or modifying the FreeType + Project, you indicate that you understand and accept all the terms + of this license. + +4. Contacts +----------- + + There are two mailing lists related to FreeType: + + o freetype@nongnu.org + + Discusses general use and applications of FreeType, as well as + future and wanted additions to the library and distribution. + If you are looking for support, start in this list if you + haven't found anything to help you in the documentation. + + o freetype-devel@nongnu.org + + Discusses bugs, as well as engine internals, design issues, + specific licenses, porting, etc. + + Our home page can be found at + + https://www.freetype.org + + +--- end of FTL.TXT --- + +The following license details are part of `src/bdf/README`: + +``` +License +******* + +Copyright (C) 2001-2002 by Francesco Zappa Nardelli + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, +TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +*** Portions of the driver (that is, bdflib.c and bdf.h): + +Copyright 2000 Computing Research Labs, New Mexico State University +Copyright 2001-2002, 2011 Francesco Zappa Nardelli + +Permission is hereby granted, free of charge, to any person obtaining a +copy of this software and associated documentation files (the "Software"), +to deal in the Software without restriction, including without limitation +the rights to use, copy, modify, merge, publish, distribute, sublicense, +and/or sell copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +THE COMPUTING RESEARCH LAB OR NEW MEXICO STATE UNIVERSITY BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT +OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR +THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +Credits +******* + +This driver is based on excellent Mark Leisher's bdf library. If you +find something good in this driver you should probably thank him, not +me. +``` + +The following license details are part of `src/pcf/README`: + +``` +License +******* + +Copyright (C) 2000 by Francesco Zappa Nardelli + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, +TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +Credits +******* + +Keith Packard wrote the pcf driver found in XFree86. His work is at +the same time the specification and the sample implementation of the +PCF format. Undoubtedly, this driver is inspired from his work. +``` + + +---- + +HARFBUZZ + +HarfBuzz is licensed under the so-called "Old MIT" license. Details follow. +For parts of HarfBuzz that are licensed under different licenses see individual +files names COPYING in subdirectories where applicable. + +Copyright © 2010-2022 Google, Inc. +Copyright © 2015-2020 Ebrahim Byagowi +Copyright © 2019,2020 Facebook, Inc. +Copyright © 2012,2015 Mozilla Foundation +Copyright © 2011 Codethink Limited +Copyright © 2008,2010 Nokia Corporation and/or its subsidiary(-ies) +Copyright © 2009 Keith Stribley +Copyright © 2011 Martin Hosken and SIL International +Copyright © 2007 Chris Wilson +Copyright © 2005,2006,2020,2021,2022,2023 Behdad Esfahbod +Copyright © 2004,2007,2008,2009,2010,2013,2021,2022,2023 Red Hat, Inc. +Copyright © 1998-2005 David Turner and Werner Lemberg +Copyright © 2016 Igalia S.L. +Copyright © 2022 Matthias Clasen +Copyright © 2018,2021 Khaled Hosny +Copyright © 2018,2019,2020 Adobe, Inc +Copyright © 2013-2015 Alexei Podtelezhnikov + +For full copyright notices consult the individual files in the package. + + +Permission is hereby granted, without written agreement and without +license or royalty fees, to use, copy, modify, and distribute this +software and its documentation for any purpose, provided that the +above copyright notice and the following two paragraphs appear in +all copies of this software. + +IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE TO ANY PARTY FOR +DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES +ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS DOCUMENTATION, EVEN +IF THE COPYRIGHT HOLDER HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH +DAMAGE. + +THE COPYRIGHT HOLDER SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, +BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND +FITNESS FOR A PARTICULAR PURPOSE. THE SOFTWARE PROVIDED HEREUNDER IS +ON AN "AS IS" BASIS, AND THE COPYRIGHT HOLDER HAS NO OBLIGATION TO +PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. + + +---- + +LCMS2 + +Little CMS +Copyright (c) 1998-2020 Marti Maria Saguer + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +---- + +LIBAVIF + +Copyright 2019 Joe Drago. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this +list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, +this list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ + +Files: src/obu.c + +Copyright © 2018-2019, VideoLAN and dav1d authors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ + +Files: third_party/iccjpeg/* + +In plain English: + +1. We don't promise that this software works. (But if you find any bugs, + please let us know!) +2. You can use this software for whatever you want. You don't have to pay us. +3. You may not pretend that you wrote this software. If you use it in a + program, you must acknowledge somewhere in your documentation that + you've used the IJG code. + +In legalese: + +The authors make NO WARRANTY or representation, either express or implied, +with respect to this software, its quality, accuracy, merchantability, or +fitness for a particular purpose. This software is provided "AS IS", and you, +its user, assume the entire risk as to its quality and accuracy. + +This software is copyright (C) 1991-2013, Thomas G. Lane, Guido Vollbeding. +All Rights Reserved except as specified below. + +Permission is hereby granted to use, copy, modify, and distribute this +software (or portions thereof) for any purpose, without fee, subject to these +conditions: +(1) If any part of the source code for this software is distributed, then this +README file must be included, with this copyright and no-warranty notice +unaltered; and any additions, deletions, or changes to the original files +must be clearly indicated in accompanying documentation. +(2) If only executable code is distributed, then the accompanying +documentation must state that "this software is based in part on the work of +the Independent JPEG Group". +(3) Permission for use of this software is granted only if the user accepts +full responsibility for any undesirable consequences; the authors accept +NO LIABILITY for damages of any kind. + +These conditions apply to any software derived from or based on the IJG code, +not just to the unmodified library. If you use our work, you ought to +acknowledge us. + +Permission is NOT granted for the use of any IJG author's name or company name +in advertising or publicity relating to this software or products derived from +it. This software may be referred to only as "the Independent JPEG Group's +software". + +We specifically permit and encourage the use of this software as the basis of +commercial products, provided that all warranty or liability claims are +assumed by the product vendor. + + +The Unix configuration script "configure" was produced with GNU Autoconf. +It is copyright by the Free Software Foundation but is freely distributable. +The same holds for its supporting scripts (config.guess, config.sub, +ltmain.sh). Another support script, install-sh, is copyright by X Consortium +but is also freely distributable. + +The IJG distribution formerly included code to read and write GIF files. +To avoid entanglement with the Unisys LZW patent, GIF reading support has +been removed altogether, and the GIF writer has been simplified to produce +"uncompressed GIFs". This technique does not use the LZW algorithm; the +resulting GIF files are larger than usual, but are readable by all standard +GIF decoders. + +We are required to state that + "The Graphics Interchange Format(c) is the Copyright property of + CompuServe Incorporated. GIF(sm) is a Service Mark property of + CompuServe Incorporated." + +------------------------------------------------------------------------------ + +Files: contrib/gdk-pixbuf/* + +Copyright 2020 Emmanuel Gil Peyrot. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this +list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, +this list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ + +Files: android_jni/gradlew* + + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +------------------------------------------------------------------------------ + +Files: third_party/libyuv/* + +Copyright 2011 The LibYuv Project Authors. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + + * Neither the name of Google nor the names of its contributors may + be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +---- + +LIBJPEG + +1. We don't promise that this software works. (But if you find any bugs, + please let us know!) +2. You can use this software for whatever you want. You don't have to pay us. +3. You may not pretend that you wrote this software. If you use it in a + program, you must acknowledge somewhere in your documentation that + you've used the IJG code. + +In legalese: + +The authors make NO WARRANTY or representation, either express or implied, +with respect to this software, its quality, accuracy, merchantability, or +fitness for a particular purpose. This software is provided "AS IS", and you, +its user, assume the entire risk as to its quality and accuracy. + +This software is copyright (C) 1991-2020, Thomas G. Lane, Guido Vollbeding. +All Rights Reserved except as specified below. + +Permission is hereby granted to use, copy, modify, and distribute this +software (or portions thereof) for any purpose, without fee, subject to these +conditions: +(1) If any part of the source code for this software is distributed, then this +README file must be included, with this copyright and no-warranty notice +unaltered; and any additions, deletions, or changes to the original files +must be clearly indicated in accompanying documentation. +(2) If only executable code is distributed, then the accompanying +documentation must state that "this software is based in part on the work of +the Independent JPEG Group". +(3) Permission for use of this software is granted only if the user accepts +full responsibility for any undesirable consequences; the authors accept +NO LIABILITY for damages of any kind. + +These conditions apply to any software derived from or based on the IJG code, +not just to the unmodified library. If you use our work, you ought to +acknowledge us. + +Permission is NOT granted for the use of any IJG author's name or company name +in advertising or publicity relating to this software or products derived from +it. This software may be referred to only as "the Independent JPEG Group's +software". + +We specifically permit and encourage the use of this software as the basis of +commercial products, provided that all warranty or liability claims are +assumed by the product vendor. + + +---- + +LIBLZMA + +XZ Utils Licensing +================== + + Different licenses apply to different files in this package. Here + is a rough summary of which licenses apply to which parts of this + package (but check the individual files to be sure!): + + - liblzma is in the public domain. + + - xz, xzdec, and lzmadec command line tools are in the public + domain unless GNU getopt_long had to be compiled and linked + in from the lib directory. The getopt_long code is under + GNU LGPLv2.1+. + + - The scripts to grep, diff, and view compressed files have been + adapted from gzip. These scripts and their documentation are + under GNU GPLv2+. + + - All the documentation in the doc directory and most of the + XZ Utils specific documentation files in other directories + are in the public domain. + + - Translated messages are in the public domain. + + - The build system contains public domain files, and files that + are under GNU GPLv2+ or GNU GPLv3+. None of these files end up + in the binaries being built. + + - Test files and test code in the tests directory, and debugging + utilities in the debug directory are in the public domain. + + - The extra directory may contain public domain files, and files + that are under various free software licenses. + + You can do whatever you want with the files that have been put into + the public domain. If you find public domain legally problematic, + take the previous sentence as a license grant. If you still find + the lack of copyright legally problematic, you have too many + lawyers. + + As usual, this software is provided "as is", without any warranty. + + If you copy significant amounts of public domain code from XZ Utils + into your project, acknowledging this somewhere in your software is + polite (especially if it is proprietary, non-free software), but + naturally it is not legally required. Here is an example of a good + notice to put into "about box" or into documentation: + + This software includes code from XZ Utils . + + The following license texts are included in the following files: + - COPYING.LGPLv2.1: GNU Lesser General Public License version 2.1 + - COPYING.GPLv2: GNU General Public License version 2 + - COPYING.GPLv3: GNU General Public License version 3 + + Note that the toolchain (compiler, linker etc.) may add some code + pieces that are copyrighted. Thus, it is possible that e.g. liblzma + binary wouldn't actually be in the public domain in its entirety + even though it contains no copyrighted code from the XZ Utils source + package. + + If you have questions, don't hesitate to ask the author(s) for more + information. + + +---- + +LIBPNG + +COPYRIGHT NOTICE, DISCLAIMER, and LICENSE +========================================= + +PNG Reference Library License version 2 +--------------------------------------- + + * Copyright (c) 1995-2022 The PNG Reference Library Authors. + * Copyright (c) 2018-2022 Cosmin Truta. + * Copyright (c) 2000-2002, 2004, 2006-2018 Glenn Randers-Pehrson. + * Copyright (c) 1996-1997 Andreas Dilger. + * Copyright (c) 1995-1996 Guy Eric Schalnat, Group 42, Inc. + +The software is supplied "as is", without warranty of any kind, +express or implied, including, without limitation, the warranties +of merchantability, fitness for a particular purpose, title, and +non-infringement. In no event shall the Copyright owners, or +anyone distributing the software, be liable for any damages or +other liability, whether in contract, tort or otherwise, arising +from, out of, or in connection with the software, or the use or +other dealings in the software, even if advised of the possibility +of such damage. + +Permission is hereby granted to use, copy, modify, and distribute +this software, or portions hereof, for any purpose, without fee, +subject to the following restrictions: + + 1. The origin of this software must not be misrepresented; you + must not claim that you wrote the original software. If you + use this software in a product, an acknowledgment in the product + documentation would be appreciated, but is not required. + + 2. Altered source versions must be plainly marked as such, and must + not be misrepresented as being the original software. + + 3. This Copyright notice may not be removed or altered from any + source or altered source distribution. + + +PNG Reference Library License version 1 (for libpng 0.5 through 1.6.35) +----------------------------------------------------------------------- + +libpng versions 1.0.7, July 1, 2000, through 1.6.35, July 15, 2018 are +Copyright (c) 2000-2002, 2004, 2006-2018 Glenn Randers-Pehrson, are +derived from libpng-1.0.6, and are distributed according to the same +disclaimer and license as libpng-1.0.6 with the following individuals +added to the list of Contributing Authors: + + Simon-Pierre Cadieux + Eric S. Raymond + Mans Rullgard + Cosmin Truta + Gilles Vollant + James Yu + Mandar Sahastrabuddhe + Google Inc. + Vadim Barkov + +and with the following additions to the disclaimer: + + There is no warranty against interference with your enjoyment of + the library or against infringement. There is no warranty that our + efforts or the library will fulfill any of your particular purposes + or needs. This library is provided with all faults, and the entire + risk of satisfactory quality, performance, accuracy, and effort is + with the user. + +Some files in the "contrib" directory and some configure-generated +files that are distributed with libpng have other copyright owners, and +are released under other open source licenses. + +libpng versions 0.97, January 1998, through 1.0.6, March 20, 2000, are +Copyright (c) 1998-2000 Glenn Randers-Pehrson, are derived from +libpng-0.96, and are distributed according to the same disclaimer and +license as libpng-0.96, with the following individuals added to the +list of Contributing Authors: + + Tom Lane + Glenn Randers-Pehrson + Willem van Schaik + +libpng versions 0.89, June 1996, through 0.96, May 1997, are +Copyright (c) 1996-1997 Andreas Dilger, are derived from libpng-0.88, +and are distributed according to the same disclaimer and license as +libpng-0.88, with the following individuals added to the list of +Contributing Authors: + + John Bowler + Kevin Bracey + Sam Bushell + Magnus Holmgren + Greg Roelofs + Tom Tanner + +Some files in the "scripts" directory have other copyright owners, +but are released under this license. + +libpng versions 0.5, May 1995, through 0.88, January 1996, are +Copyright (c) 1995-1996 Guy Eric Schalnat, Group 42, Inc. + +For the purposes of this copyright and license, "Contributing Authors" +is defined as the following set of individuals: + + Andreas Dilger + Dave Martindale + Guy Eric Schalnat + Paul Schmidt + Tim Wegner + +The PNG Reference Library is supplied "AS IS". The Contributing +Authors and Group 42, Inc. disclaim all warranties, expressed or +implied, including, without limitation, the warranties of +merchantability and of fitness for any purpose. The Contributing +Authors and Group 42, Inc. assume no liability for direct, indirect, +incidental, special, exemplary, or consequential damages, which may +result from the use of the PNG Reference Library, even if advised of +the possibility of such damage. + +Permission is hereby granted to use, copy, modify, and distribute this +source code, or portions hereof, for any purpose, without fee, subject +to the following restrictions: + + 1. The origin of this source code must not be misrepresented. + + 2. Altered versions must be plainly marked as such and must not + be misrepresented as being the original source. + + 3. This Copyright notice may not be removed or altered from any + source or altered source distribution. + +The Contributing Authors and Group 42, Inc. specifically permit, +without fee, and encourage the use of this source code as a component +to supporting the PNG file format in commercial products. If you use +this source code in a product, acknowledgment is not required but would +be appreciated. + + +---- + +LIBTIFF + +Copyright (c) 1988-1997 Sam Leffler +Copyright (c) 1991-1997 Silicon Graphics, Inc. + +Permission to use, copy, modify, distribute, and sell this software and +its documentation for any purpose is hereby granted without fee, provided +that (i) the above copyright notices and this permission notice appear in +all copies of the software and related documentation, and (ii) the names of +Sam Leffler and Silicon Graphics may not be used in any advertising or +publicity relating to the software without the specific, prior written +permission of Sam Leffler and Silicon Graphics. + +THE SOFTWARE IS PROVIDED "AS-IS" AND WITHOUT WARRANTY OF ANY KIND, +EXPRESS, IMPLIED OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY +WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. + +IN NO EVENT SHALL SAM LEFFLER OR SILICON GRAPHICS BE LIABLE FOR +ANY SPECIAL, INCIDENTAL, INDIRECT OR CONSEQUENTIAL DAMAGES OF ANY KIND, +OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, +WHETHER OR NOT ADVISED OF THE POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF +LIABILITY, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE +OF THIS SOFTWARE. + + +---- + +LIBWEBP + +Copyright (c) 2010, Google Inc. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + + * Neither the name of Google nor the names of its contributors may + be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +---- + +LIBYUV + +Copyright 2011 The LibYuv Project Authors. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + + * Neither the name of Google nor the names of its contributors may + be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +---- + +OPENJPEG + +* + * The copyright in this software is being made available under the 2-clauses + * BSD License, included below. This software may be subject to other third + * party and contributor rights, including patent rights, and no such rights + * are granted under this license. + * + * Copyright (c) 2002-2014, Universite catholique de Louvain (UCL), Belgium + * Copyright (c) 2002-2014, Professor Benoit Macq + * Copyright (c) 2003-2014, Antonin Descampe + * Copyright (c) 2003-2009, Francois-Olivier Devaux + * Copyright (c) 2005, Herve Drolon, FreeImage Team + * Copyright (c) 2002-2003, Yannick Verschueren + * Copyright (c) 2001-2003, David Janssens + * Copyright (c) 2011-2012, Centre National d'Etudes Spatiales (CNES), France + * Copyright (c) 2012, CS Systemes d'Information, France + * + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS `AS IS' + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ + + +---- + +RAQM + +The MIT License (MIT) + +Copyright © 2015 Information Technology Authority (ITA) +Copyright © 2016 Khaled Hosny + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +---- + +XAU + +Copyright 1988, 1993, 1994, 1998 The Open Group + +Permission to use, copy, modify, distribute, and sell this software and its +documentation for any purpose is hereby granted without fee, provided that +the above copyright notice appear in all copies and that both that +copyright notice and this permission notice appear in supporting +documentation. + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +OPEN GROUP BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN +AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the name of The Open Group shall not be +used in advertising or otherwise to promote the sale, use or other dealings +in this Software without prior written authorization from The Open Group. + + +---- + +XCB + +Copyright (C) 2001-2006 Bart Massey, Jamey Sharp, and Josh Triplett. +All Rights Reserved. + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated +documentation files (the "Software"), to deal in the +Software without restriction, including without limitation +the rights to use, copy, modify, merge, publish, distribute, +sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall +be included in all copies or substantial portions of the +Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY +KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE +WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR +PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS +BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the names of the authors +or their institutions shall not be used in advertising or +otherwise to promote the sale, use or other dealings in this +Software without prior written authorization from the +authors. + + +---- + +XDMCP + +Copyright 1989, 1998 The Open Group + +Permission to use, copy, modify, distribute, and sell this software and its +documentation for any purpose is hereby granted without fee, provided that +the above copyright notice appear in all copies and that both that +copyright notice and this permission notice appear in supporting +documentation. + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +OPEN GROUP BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN +AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the name of The Open Group shall not be +used in advertising or otherwise to promote the sale, use or other dealings +in this Software without prior written authorization from The Open Group. + +Author: Keith Packard, MIT X Consortium + + +---- + +ZLIB + + (C) 1995-2017 Jean-loup Gailly and Mark Adler + + This software is provided 'as-is', without any express or implied + warranty. In no event will the authors be held liable for any damages + arising from the use of this software. + + Permission is granted to anyone to use this software for any purpose, + including commercial applications, and to alter it and redistribute it + freely, subject to the following restrictions: + + 1. The origin of this software must not be misrepresented; you must not + claim that you wrote the original software. If you use this software + in a product, an acknowledgment in the product documentation would be + appreciated but is not required. + 2. Altered source versions must be plainly marked as such, and must not be + misrepresented as being the original software. + 3. This notice may not be removed or altered from any source distribution. + + Jean-loup Gailly Mark Adler + jloup@gzip.org madler@alumni.caltech.edu + +If you use the zlib library in a product, we would appreciate *not* receiving +lengthy legal documents to sign. The sources are provided for free but without +warranty of any kind. The library has been entirely written by Jean-loup +Gailly and Mark Adler; it does not include third-party code. + +If you redistribute modified sources, we would appreciate that you include in +the file ChangeLog history information documenting your changes. Please read +the FAQ for more information on the distribution of modified source versions. + + +platformdirs +MIT +https://github.com/tox-dev/platformdirs +MIT License + +Copyright (c) 2010-202x The platformdirs developers + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +pluggy +MIT License +UNKNOWN +The MIT License (MIT) + +Copyright (c) 2015 holger krekel (rather uses bitbucket/hpk42) + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +plyfile +GNU General Public License v3 or later (GPLv3+) +https://github.com/dranjan/python-plyfile + GNU GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + Copyright (C) + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, your program's commands +might be different; for a GUI interface, you would use an "about box". + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU GPL, see +. + + The GNU General Public License does not permit incorporating your program +into proprietary programs. If your program is a subroutine library, you +may consider it more useful to permit linking proprietary applications with +the library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. But first, please read +. + + +polars +MIT License +https://www.pola.rs/ +Copyright (c) 2025 Ritchie Vink +Some portions Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +polars-runtime-32 +MIT License +https://www.pola.rs/ +Copyright (c) 2025 Ritchie Vink +Some portions Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +polyscope +MIT License +https://github.com/nmwsharp/polyscope +MIT License + +Copyright (c) 2017-2020 Nicholas Sharp and the Polyscope contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +portalocker +BSD-3-Clause +https://github.com/wolph/portalocker/ +Copyright 2022 Rick van Hattem + +Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +proglog +MIT +https://github.com/Edinburgh-Genome-Foundry/proglog +MIT License + +Copyright (c) 2017 Edinburgh Genome Foundry, University of Edinburgh + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +prometheus_client +Apache-2.0 AND BSD-2-Clause +https://github.com/prometheus/client_python + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +prompt_toolkit +BSD License +https://github.com/prompt-toolkit/python-prompt-toolkit +Copyright (c) 2014, Jonathan Slenders +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, this + list of conditions and the following disclaimer in the documentation and/or + other materials provided with the distribution. + +* Neither the name of the {organization} nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +propcache +Apache Software License +https://github.com/aio-libs/propcache + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +proto-plus +Apache Software License +https://github.com/googleapis/proto-plus-python + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +protobuf +3-Clause BSD License +https://developers.google.com/protocol-buffers/ +Copyright 2008 Google Inc. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Code generated by the Protocol Buffer compiler is owned by the owner +of the input file used when generating it. This code is not +standalone and requires a support library to be linked with it. This +support library is itself covered by the above license. + + +psutil +BSD-3-Clause +https://github.com/giampaolo/psutil +BSD 3-Clause License + +Copyright (c) 2009, Jay Loden, Dave Daeschler, Giampaolo Rodola +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + * Neither the name of the psutil authors nor the names of its contributors + may be used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +psycopg2-binary +GNU Library or Lesser General Public License (LGPL) +https://psycopg.org/ +psycopg2 and the LGPL +--------------------- + +psycopg2 is free software: you can redistribute it and/or modify it +under the terms of the GNU Lesser General Public License as published +by the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +psycopg2 is distributed in the hope that it will be useful, but WITHOUT +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public +License for more details. + +In addition, as a special exception, the copyright holders give +permission to link this program with the OpenSSL library (or with +modified versions of OpenSSL that use the same license as OpenSSL), +and distribute linked combinations including the two. + +You must obey the GNU Lesser General Public License in all respects for +all of the code used other than OpenSSL. If you modify file(s) with this +exception, you may extend this exception to your version of the file(s), +but you are not obligated to do so. If you do not wish to do so, delete +this exception statement from your version. If you delete this exception +statement from all source files in the program, then also delete it here. + +You should have received a copy of the GNU Lesser General Public License +along with psycopg2 (see the doc/ directory.) +If not, see . + + +Alternative licenses +-------------------- + +The following BSD-like license applies (at your option) to the files following +the pattern ``psycopg/adapter*.{h,c}`` and ``psycopg/microprotocol*.{h,c}``: + + Permission is granted to anyone to use this software for any purpose, + including commercial applications, and to alter it and redistribute it + freely, subject to the following restrictions: + + 1. The origin of this software must not be misrepresented; you must not + claim that you wrote the original software. If you use this + software in a product, an acknowledgment in the product documentation + would be appreciated but is not required. + + 2. Altered source versions must be plainly marked as such, and must not + be misrepresented as being the original software. + + 3. This notice may not be removed or altered from any source distribution. + + +ptyprocess +ISC License (ISCL) +https://github.com/pexpect/ptyprocess +Ptyprocess is under the ISC license, as code derived from Pexpect. + http://opensource.org/licenses/ISC + +Copyright (c) 2013-2014, Pexpect development team +Copyright (c) 2012, Noah Spurrier + +PERMISSION TO USE, COPY, MODIFY, AND/OR DISTRIBUTE THIS SOFTWARE FOR ANY PURPOSE +WITH OR WITHOUT FEE IS HEREBY GRANTED, PROVIDED THAT THE ABOVE COPYRIGHT NOTICE +AND THIS PERMISSION NOTICE APPEAR IN ALL COPIES. THE SOFTWARE IS PROVIDED +"AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE +INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT +SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL +DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, +WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING +OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + + + +pure_eval +MIT License +http://github.com/alexmojaki/pure_eval +MIT License + +Copyright (c) 2019 Alex Hall + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +py-spy +MIT License +https://github.com/benfred/py-spy +The MIT License (MIT) + +Copyright (c) 2018-2019 Ben Frederickson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +py3nvml +BSD License +https://github.com/fbcotter/py3nvml.git +COPYRIGHT +--------- +Copyright (c) 2011-2015, NVIDIA Corporation. All rights reserved. + +LICENSE +------- +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +- Redistributions of source code must retain the above copyright notice, this +list of conditions and the following disclaimer. + +- Redistributions in binary form must reproduce the above copyright notice, +this list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +- Neither the name of the NVIDIA Corporation nor the names of its contributors +may be used to endorse or promote products derived from this software without +specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + + + +pyarrow +Apache Software License +https://arrow.apache.org/ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +-------------------------------------------------------------------------------- + +src/arrow/util (some portions): Apache 2.0, and 3-clause BSD + +Some portions of this module are derived from code in the Chromium project, +copyright (c) Google inc and (c) The Chromium Authors and licensed under the +Apache 2.0 License or the under the 3-clause BSD license: + + Copyright (c) 2013 The Chromium Authors. All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are + met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following disclaimer + in the documentation and/or other materials provided with the + distribution. + * Neither the name of Google Inc. nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +-------------------------------------------------------------------------------- + +This project includes code from Daniel Lemire's FrameOfReference project. + +https://github.com/lemire/FrameOfReference/blob/6ccaf9e97160f9a3b299e23a8ef739e711ef0c71/src/bpacking.cpp +https://github.com/lemire/FrameOfReference/blob/146948b6058a976bc7767262ad3a2ce201486b93/scripts/turbopacking64.py + +Copyright: 2013 Daniel Lemire +Home page: http://lemire.me/en/ +Project page: https://github.com/lemire/FrameOfReference +License: Apache License Version 2.0 http://www.apache.org/licenses/LICENSE-2.0 + +-------------------------------------------------------------------------------- + +This project includes code from the TensorFlow project + +Copyright 2015 The TensorFlow Authors. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +-------------------------------------------------------------------------------- + +This project includes code from the NumPy project. + +https://github.com/numpy/numpy/blob/e1f191c46f2eebd6cb892a4bfe14d9dd43a06c4e/numpy/core/src/multiarray/multiarraymodule.c#L2910 + +https://github.com/numpy/numpy/blob/68fd82271b9ea5a9e50d4e761061dfcca851382a/numpy/core/src/multiarray/datetime.c + +Copyright (c) 2005-2017, NumPy Developers. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided + with the distribution. + + * Neither the name of the NumPy Developers nor the names of any + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +-------------------------------------------------------------------------------- + +This project includes code from the Boost project + +Boost Software License - Version 1.0 - August 17th, 2003 + +Permission is hereby granted, free of charge, to any person or organization +obtaining a copy of the software and accompanying documentation covered by +this license (the "Software") to use, reproduce, display, distribute, +execute, and transmit the Software, and to prepare derivative works of the +Software, and to permit third-parties to whom the Software is furnished to +do so, all subject to the following: + +The copyright notices in the Software and this entire statement, including +the above license grant, this restriction and the following disclaimer, +must be included in all copies of the Software, in whole or in part, and +all derivative works of the Software, unless such copies or derivative +works are solely in the form of machine-executable object code generated by +a source language processor. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT +SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE +FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, +ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. + +-------------------------------------------------------------------------------- + +This project includes code from the FlatBuffers project + +Copyright 2014 Google Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +-------------------------------------------------------------------------------- + +This project includes code from the tslib project + +Copyright 2015 Microsoft Corporation. All rights reserved. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +-------------------------------------------------------------------------------- + +This project includes code from the jemalloc project + +https://github.com/jemalloc/jemalloc + +Copyright (C) 2002-2017 Jason Evans . +All rights reserved. +Copyright (C) 2007-2012 Mozilla Foundation. All rights reserved. +Copyright (C) 2009-2017 Facebook, Inc. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: +1. Redistributions of source code must retain the above copyright notice(s), + this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright notice(s), + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS +OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO +EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE +OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF +ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +-------------------------------------------------------------------------------- + +This project includes code from the Go project, BSD 3-clause license + PATENTS +weak patent termination clause +(https://github.com/golang/go/blob/master/PATENTS). + +Copyright (c) 2009 The Go Authors. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +-------------------------------------------------------------------------------- + +This project includes code from the hs2client + +https://github.com/cloudera/hs2client + +Copyright 2016 Cloudera Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +-------------------------------------------------------------------------------- + +The script ci/scripts/util_wait_for_it.sh has the following license + +Copyright (c) 2016 Giles Hall + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +-------------------------------------------------------------------------------- + +The script r/configure has the following license (MIT) + +Copyright (c) 2017, Jeroen Ooms and Jim Hester + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +-------------------------------------------------------------------------------- + +cpp/src/arrow/util/logging.cc, cpp/src/arrow/util/logging.h and +cpp/src/arrow/util/logging-test.cc are adapted from +Ray Project (https://github.com/ray-project/ray) (Apache 2.0). + +Copyright (c) 2016 Ray Project (https://github.com/ray-project/ray) + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +-------------------------------------------------------------------------------- +The files cpp/src/arrow/vendored/datetime/date.h, cpp/src/arrow/vendored/datetime/tz.h, +cpp/src/arrow/vendored/datetime/tz_private.h, cpp/src/arrow/vendored/datetime/ios.h, +cpp/src/arrow/vendored/datetime/ios.mm, +cpp/src/arrow/vendored/datetime/tz.cpp are adapted from +Howard Hinnant's date library (https://github.com/HowardHinnant/date) +It is licensed under MIT license. + +The MIT License (MIT) +Copyright (c) 2015, 2016, 2017 Howard Hinnant +Copyright (c) 2016 Adrian Colomitchi +Copyright (c) 2017 Florian Dang +Copyright (c) 2017 Paul Thompson +Copyright (c) 2018 Tomasz Kamiński + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +-------------------------------------------------------------------------------- + +The file cpp/src/arrow/util/utf8.h includes code adapted from the page + https://bjoern.hoehrmann.de/utf-8/decoder/dfa/ +with the following license (MIT) + +Copyright (c) 2008-2009 Bjoern Hoehrmann + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +-------------------------------------------------------------------------------- + +The files in cpp/src/arrow/vendored/xxhash/ have the following license +(BSD 2-Clause License) + +xxHash Library +Copyright (c) 2012-2014, Yann Collet +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, this + list of conditions and the following disclaimer in the documentation and/or + other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +You can contact the author at : +- xxHash homepage: http://www.xxhash.com +- xxHash source repository : https://github.com/Cyan4973/xxHash + +-------------------------------------------------------------------------------- + +The files in cpp/src/arrow/vendored/double-conversion/ have the following license +(BSD 3-Clause License) + +Copyright 2006-2011, the V8 project authors. All rights reserved. +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided + with the distribution. + * Neither the name of Google Inc. nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +-------------------------------------------------------------------------------- + +The files in cpp/src/arrow/vendored/uriparser/ have the following license +(BSD 3-Clause License) + +uriparser - RFC 3986 URI parsing library + +Copyright (C) 2007, Weijia Song +Copyright (C) 2007, Sebastian Pipping +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + + * Redistributions of source code must retain the above + copyright notice, this list of conditions and the following + disclaimer. + + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials + provided with the distribution. + + * Neither the name of the nor the names of its + contributors may be used to endorse or promote products + derived from this software without specific prior written + permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS +FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE +COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED +OF THE POSSIBILITY OF SUCH DAMAGE. + +-------------------------------------------------------------------------------- + +The files under dev/tasks/conda-recipes have the following license + +BSD 3-clause license +Copyright (c) 2015-2018, conda-forge +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its contributors + may be used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR +TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF +THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +-------------------------------------------------------------------------------- + +The files in cpp/src/arrow/vendored/utfcpp/ have the following license + +Copyright 2006-2018 Nemanja Trifunovic + +Permission is hereby granted, free of charge, to any person or organization +obtaining a copy of the software and accompanying documentation covered by +this license (the "Software") to use, reproduce, display, distribute, +execute, and transmit the Software, and to prepare derivative works of the +Software, and to permit third-parties to whom the Software is furnished to +do so, all subject to the following: + +The copyright notices in the Software and this entire statement, including +the above license grant, this restriction and the following disclaimer, +must be included in all copies of the Software, in whole or in part, and +all derivative works of the Software, unless such copies or derivative +works are solely in the form of machine-executable object code generated by +a source language processor. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT +SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE +FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, +ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. + +-------------------------------------------------------------------------------- + +This project includes code from Apache Kudu. + + * cpp/cmake_modules/CompilerInfo.cmake is based on Kudu's cmake_modules/CompilerInfo.cmake + +Copyright: 2016 The Apache Software Foundation. +Home page: https://kudu.apache.org/ +License: http://www.apache.org/licenses/LICENSE-2.0 + +-------------------------------------------------------------------------------- + +This project includes code from Apache Impala (incubating), formerly +Impala. The Impala code and rights were donated to the ASF as part of the +Incubator process after the initial code imports into Apache Parquet. + +Copyright: 2012 Cloudera, Inc. +Copyright: 2016 The Apache Software Foundation. +Home page: http://impala.apache.org/ +License: http://www.apache.org/licenses/LICENSE-2.0 + +-------------------------------------------------------------------------------- + +This project includes code from Apache Aurora. + +* dev/release/{release,changelog,release-candidate} are based on the scripts from + Apache Aurora + +Copyright: 2016 The Apache Software Foundation. +Home page: https://aurora.apache.org/ +License: http://www.apache.org/licenses/LICENSE-2.0 + +-------------------------------------------------------------------------------- + +This project includes code from Snappy. + +* cpp/cmake_modules/{SnappyCMakeLists.txt,SnappyConfig.h} are based on code + from Google's Snappy project. + +Copyright: 2009 Google Inc. All rights reserved. +Homepage: https://github.com/google/snappy +License: 3-clause BSD + +-------------------------------------------------------------------------------- + +This project includes code from the manylinux project. + +* python/manylinux1/scripts/{build_python.sh,python-tag-abi-tag.py, + requirements.txt} are based on code from the manylinux project. + +Copyright: 2016 manylinux +Homepage: https://github.com/pypa/manylinux +License: The MIT License (MIT) + +-------------------------------------------------------------------------------- + +This project includes code from the cymove project: + +* python/pyarrow/includes/common.pxd includes code from the cymove project + +The MIT License (MIT) +Copyright (c) 2019 Omer Ozarslan + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, +DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR +OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE +OR OTHER DEALINGS IN THE SOFTWARE. + +-------------------------------------------------------------------------------- + +The projects includes code from the Ursabot project under the dev/archery +directory. + +License: BSD 2-Clause + +Copyright 2019 RStudio, Inc. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +-------------------------------------------------------------------------------- + +This project include code from mingw-w64. + +* cpp/src/arrow/util/cpu-info.cc has a polyfill for mingw-w64 < 5 + +Copyright (c) 2009 - 2013 by the mingw-w64 project +Homepage: https://mingw-w64.org +License: Zope Public License (ZPL) Version 2.1. + +--------------------------------------------------------------------------------- + +This project include code from Google's Asylo project. + +* cpp/src/arrow/result.h is based on status_or.h + +Copyright (c) Copyright 2017 Asylo authors +Homepage: https://asylo.dev/ +License: Apache 2.0 + +-------------------------------------------------------------------------------- + +This project includes code from Google's protobuf project + +* cpp/src/arrow/result.h ARROW_ASSIGN_OR_RAISE is based off ASSIGN_OR_RETURN +* cpp/src/arrow/util/bit_stream_utils.h contains code from wire_format_lite.h + +Copyright 2008 Google Inc. All rights reserved. +Homepage: https://developers.google.com/protocol-buffers/ +License: + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Code generated by the Protocol Buffer compiler is owned by the owner +of the input file used when generating it. This code is not +standalone and requires a support library to be linked with it. This +support library is itself covered by the above license. + +-------------------------------------------------------------------------------- + +3rdparty dependency LLVM is statically linked in certain binary distributions. +Additionally some sections of source code have been derived from sources in LLVM +and have been clearly labeled as such. LLVM has the following license: + +============================================================================== +The LLVM Project is under the Apache License v2.0 with LLVM Exceptions: +============================================================================== + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +---- LLVM Exceptions to the Apache 2.0 License ---- + +As an exception, if, as a result of your compiling your source code, portions +of this Software are embedded into an Object form of such source code, you +may redistribute such embedded portions in such Object form without complying +with the conditions of Sections 4(a), 4(b) and 4(d) of the License. + +In addition, if you combine or link compiled forms of this Software with +software that is licensed under the GPLv2 ("Combined Software") and if a +court of competent jurisdiction determines that the patent provision (Section +3), the indemnity provision (Section 9) or other Section of the License +conflicts with the conditions of the GPLv2, you may retroactively and +prospectively choose to deem waived or otherwise exclude such Section(s) of +the License, but only in their entirety and only with respect to the Combined +Software. + +============================================================================== +Software from third parties included in the LLVM Project: +============================================================================== +The LLVM Project contains third party software which is under different license +terms. All such code will be identified clearly using at least one of two +mechanisms: +1) It will be in a separate directory tree with its own `LICENSE.txt` or + `LICENSE` file at the top containing the specific license and restrictions + which apply to that software, or +2) It will contain specific license and restriction terms at the top of every + file. + +-------------------------------------------------------------------------------- + +3rdparty dependency gRPC is statically linked in certain binary +distributions, like the python wheels. gRPC has the following license: + +Copyright 2014 gRPC authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +-------------------------------------------------------------------------------- + +3rdparty dependency Apache Thrift is statically linked in certain binary +distributions, like the python wheels. Apache Thrift has the following license: + +Apache Thrift +Copyright (C) 2006 - 2019, The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +-------------------------------------------------------------------------------- + +3rdparty dependency Apache ORC is statically linked in certain binary +distributions, like the python wheels. Apache ORC has the following license: + +Apache ORC +Copyright 2013-2019 The Apache Software Foundation + +This product includes software developed by The Apache Software +Foundation (http://www.apache.org/). + +This product includes software developed by Hewlett-Packard: +(c) Copyright [2014-2015] Hewlett-Packard Development Company, L.P + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +-------------------------------------------------------------------------------- + +3rdparty dependency zstd is statically linked in certain binary +distributions, like the python wheels. ZSTD has the following license: + +BSD License + +For Zstandard software + +Copyright (c) 2016-present, Facebook, Inc. All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + * Neither the name Facebook nor the names of its contributors may be used to + endorse or promote products derived from this software without specific + prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +-------------------------------------------------------------------------------- + +3rdparty dependency lz4 is statically linked in certain binary +distributions, like the python wheels. lz4 has the following license: + +LZ4 Library +Copyright (c) 2011-2016, Yann Collet +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, this + list of conditions and the following disclaimer in the documentation and/or + other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +-------------------------------------------------------------------------------- + +3rdparty dependency Brotli is statically linked in certain binary +distributions, like the python wheels. Brotli has the following license: + +Copyright (c) 2009, 2010, 2013-2016 by the Brotli Authors. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +-------------------------------------------------------------------------------- + +3rdparty dependency rapidjson is statically linked in certain binary +distributions, like the python wheels. rapidjson and its dependencies have the +following licenses: + +Tencent is pleased to support the open source community by making RapidJSON +available. + +Copyright (C) 2015 THL A29 Limited, a Tencent company, and Milo Yip. +All rights reserved. + +If you have downloaded a copy of the RapidJSON binary from Tencent, please note +that the RapidJSON binary is licensed under the MIT License. +If you have downloaded a copy of the RapidJSON source code from Tencent, please +note that RapidJSON source code is licensed under the MIT License, except for +the third-party components listed below which are subject to different license +terms. Your integration of RapidJSON into your own projects may require +compliance with the MIT License, as well as the other licenses applicable to +the third-party components included within RapidJSON. To avoid the problematic +JSON license in your own projects, it's sufficient to exclude the +bin/jsonchecker/ directory, as it's the only code under the JSON license. +A copy of the MIT License is included in this file. + +Other dependencies and licenses: + + Open Source Software Licensed Under the BSD License: + -------------------------------------------------------------------- + + The msinttypes r29 + Copyright (c) 2006-2013 Alexander Chemeris + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + * Neither the name of copyright holder nor the names of its contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND ANY + EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + DISCLAIMED. IN NO EVENT SHALL THE REGENTS AND CONTRIBUTORS BE LIABLE FOR + ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH + DAMAGE. + + Terms of the MIT License: + -------------------------------------------------------------------- + + Permission is hereby granted, free of charge, to any person obtaining a + copy of this software and associated documentation files (the "Software"), + to deal in the Software without restriction, including without limitation + the rights to use, copy, modify, merge, publish, distribute, sublicense, + and/or sell copies of the Software, and to permit persons to whom the + Software is furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included + in all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER + DEALINGS IN THE SOFTWARE. + +-------------------------------------------------------------------------------- + +3rdparty dependency snappy is statically linked in certain binary +distributions, like the python wheels. snappy has the following license: + +Copyright 2011, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + * Neither the name of Google Inc. nor the names of its contributors may be + used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +=== + +Some of the benchmark data in testdata/ is licensed differently: + + - fireworks.jpeg is Copyright 2013 Steinar H. Gunderson, and + is licensed under the Creative Commons Attribution 3.0 license + (CC-BY-3.0). See https://creativecommons.org/licenses/by/3.0/ + for more information. + + - kppkn.gtb is taken from the Gaviota chess tablebase set, and + is licensed under the MIT License. See + https://sites.google.com/site/gaviotachessengine/Home/endgame-tablebases-1 + for more information. + + - paper-100k.pdf is an excerpt (bytes 92160 to 194560) from the paper + “Combinatorial Modeling of Chromatin Features Quantitatively Predicts DNA + Replication Timing in _Drosophila_” by Federico Comoglio and Renato Paro, + which is licensed under the CC-BY license. See + http://www.ploscompbiol.org/static/license for more ifnormation. + + - alice29.txt, asyoulik.txt, plrabn12.txt and lcet10.txt are from Project + Gutenberg. The first three have expired copyrights and are in the public + domain; the latter does not have expired copyright, but is still in the + public domain according to the license information + (http://www.gutenberg.org/ebooks/53). + +-------------------------------------------------------------------------------- + +3rdparty dependency gflags is statically linked in certain binary +distributions, like the python wheels. gflags has the following license: + +Copyright (c) 2006, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +-------------------------------------------------------------------------------- + +3rdparty dependency glog is statically linked in certain binary +distributions, like the python wheels. glog has the following license: + +Copyright (c) 2008, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +A function gettimeofday in utilities.cc is based on + +http://www.google.com/codesearch/p?hl=en#dR3YEbitojA/COPYING&q=GetSystemTimeAsFileTime%20license:bsd + +The license of this code is: + +Copyright (c) 2003-2008, Jouni Malinen and contributors +All Rights Reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +3. Neither the name(s) of the above-listed copyright holder(s) nor the + names of its contributors may be used to endorse or promote products + derived from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +-------------------------------------------------------------------------------- + +3rdparty dependency re2 is statically linked in certain binary +distributions, like the python wheels. re2 has the following license: + +Copyright (c) 2009 The RE2 Authors. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided + with the distribution. + * Neither the name of Google Inc. nor the names of its contributors + may be used to endorse or promote products derived from this + software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +-------------------------------------------------------------------------------- + +3rdparty dependency c-ares is statically linked in certain binary +distributions, like the python wheels. c-ares has the following license: + +# c-ares license + +Copyright (c) 2007 - 2018, Daniel Stenberg with many contributors, see AUTHORS +file. + +Copyright 1998 by the Massachusetts Institute of Technology. + +Permission to use, copy, modify, and distribute this software and its +documentation for any purpose and without fee is hereby granted, provided that +the above copyright notice appear in all copies and that both that copyright +notice and this permission notice appear in supporting documentation, and that +the name of M.I.T. not be used in advertising or publicity pertaining to +distribution of the software without specific, written prior permission. +M.I.T. makes no representations about the suitability of this software for any +purpose. It is provided "as is" without express or implied warranty. + +-------------------------------------------------------------------------------- + +3rdparty dependency zlib is redistributed as a dynamically linked shared +library in certain binary distributions, like the python wheels. In the future +this will likely change to static linkage. zlib has the following license: + +zlib.h -- interface of the 'zlib' general purpose compression library + version 1.2.11, January 15th, 2017 + + Copyright (C) 1995-2017 Jean-loup Gailly and Mark Adler + + This software is provided 'as-is', without any express or implied + warranty. In no event will the authors be held liable for any damages + arising from the use of this software. + + Permission is granted to anyone to use this software for any purpose, + including commercial applications, and to alter it and redistribute it + freely, subject to the following restrictions: + + 1. The origin of this software must not be misrepresented; you must not + claim that you wrote the original software. If you use this software + in a product, an acknowledgment in the product documentation would be + appreciated but is not required. + 2. Altered source versions must be plainly marked as such, and must not be + misrepresented as being the original software. + 3. This notice may not be removed or altered from any source distribution. + + Jean-loup Gailly Mark Adler + jloup@gzip.org madler@alumni.caltech.edu + +-------------------------------------------------------------------------------- + +3rdparty dependency openssl is redistributed as a dynamically linked shared +library in certain binary distributions, like the python wheels. openssl +preceding version 3 has the following license: + + LICENSE ISSUES + ============== + + The OpenSSL toolkit stays under a double license, i.e. both the conditions of + the OpenSSL License and the original SSLeay license apply to the toolkit. + See below for the actual license texts. + + OpenSSL License + --------------- + +/* ==================================================================== + * Copyright (c) 1998-2019 The OpenSSL Project. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * 3. All advertising materials mentioning features or use of this + * software must display the following acknowledgment: + * "This product includes software developed by the OpenSSL Project + * for use in the OpenSSL Toolkit. (http://www.openssl.org/)" + * + * 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to + * endorse or promote products derived from this software without + * prior written permission. For written permission, please contact + * openssl-core@openssl.org. + * + * 5. Products derived from this software may not be called "OpenSSL" + * nor may "OpenSSL" appear in their names without prior written + * permission of the OpenSSL Project. + * + * 6. Redistributions of any form whatsoever must retain the following + * acknowledgment: + * "This product includes software developed by the OpenSSL Project + * for use in the OpenSSL Toolkit (http://www.openssl.org/)" + * + * THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY + * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR + * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED + * OF THE POSSIBILITY OF SUCH DAMAGE. + * ==================================================================== + * + * This product includes cryptographic software written by Eric Young + * (eay@cryptsoft.com). This product includes software written by Tim + * Hudson (tjh@cryptsoft.com). + * + */ + + Original SSLeay License + ----------------------- + +/* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com) + * All rights reserved. + * + * This package is an SSL implementation written + * by Eric Young (eay@cryptsoft.com). + * The implementation was written so as to conform with Netscapes SSL. + * + * This library is free for commercial and non-commercial use as long as + * the following conditions are aheared to. The following conditions + * apply to all code found in this distribution, be it the RC4, RSA, + * lhash, DES, etc., code; not just the SSL code. The SSL documentation + * included with this distribution is covered by the same copyright terms + * except that the holder is Tim Hudson (tjh@cryptsoft.com). + * + * Copyright remains Eric Young's, and as such any Copyright notices in + * the code are not to be removed. + * If this package is used in a product, Eric Young should be given attribution + * as the author of the parts of the library used. + * This can be in the form of a textual message at program startup or + * in documentation (online or textual) provided with the package. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. All advertising materials mentioning features or use of this software + * must display the following acknowledgement: + * "This product includes cryptographic software written by + * Eric Young (eay@cryptsoft.com)" + * The word 'cryptographic' can be left out if the rouines from the library + * being used are not cryptographic related :-). + * 4. If you include any Windows specific code (or a derivative thereof) from + * the apps directory (application code) you must include an acknowledgement: + * "This product includes software written by Tim Hudson (tjh@cryptsoft.com)" + * + * THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * The licence and distribution terms for any publically available version or + * derivative of this code cannot be changed. i.e. this code cannot simply be + * copied and put under another distribution licence + * [including the GNU Public Licence.] + */ + +-------------------------------------------------------------------------------- + +This project includes code from the rtools-backports project. + +* ci/scripts/PKGBUILD and ci/scripts/r_windows_build.sh are based on code + from the rtools-backports project. + +Copyright: Copyright (c) 2013 - 2019, Алексей and Jeroen Ooms. +All rights reserved. +Homepage: https://github.com/r-windows/rtools-backports +License: 3-clause BSD + +-------------------------------------------------------------------------------- + +Some code from pandas has been adapted for the pyarrow codebase. pandas is +available under the 3-clause BSD license, which follows: + +pandas license +============== + +Copyright (c) 2011-2012, Lambda Foundry, Inc. and PyData Development Team +All rights reserved. + +Copyright (c) 2008-2011 AQR Capital Management, LLC +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided + with the distribution. + + * Neither the name of the copyright holder nor the names of any + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +-------------------------------------------------------------------------------- + +Some bits from DyND, in particular aspects of the build system, have been +adapted from libdynd and dynd-python under the terms of the BSD 2-clause +license + +The BSD 2-Clause License + + Copyright (C) 2011-12, Dynamic NDArray Developers + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are + met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided + with the distribution. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Dynamic NDArray Developers list: + + * Mark Wiebe + * Continuum Analytics + +-------------------------------------------------------------------------------- + +Some source code from Ibis (https://github.com/cloudera/ibis) has been adapted +for PyArrow. Ibis is released under the Apache License, Version 2.0. + +-------------------------------------------------------------------------------- + +dev/tasks/homebrew-formulae/apache-arrow.rb has the following license: + +BSD 2-Clause License + +Copyright (c) 2009-present, Homebrew contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +---------------------------------------------------------------------- + +cpp/src/arrow/vendored/base64.cpp has the following license + +ZLIB License + +Copyright (C) 2004-2017 René Nyffenegger + +This source code is provided 'as-is', without any express or implied +warranty. In no event will the author be held liable for any damages arising +from the use of this software. + +Permission is granted to anyone to use this software for any purpose, including +commercial applications, and to alter it and redistribute it freely, subject to +the following restrictions: + +1. The origin of this source code must not be misrepresented; you must not + claim that you wrote the original source code. If you use this source code + in a product, an acknowledgment in the product documentation would be + appreciated but is not required. + +2. Altered source versions must be plainly marked as such, and must not be + misrepresented as being the original source code. + +3. This notice may not be removed or altered from any source distribution. + +René Nyffenegger rene.nyffenegger@adp-gmbh.ch + +-------------------------------------------------------------------------------- + +This project includes code from Folly. + + * cpp/src/arrow/vendored/ProducerConsumerQueue.h + +is based on Folly's + + * folly/Portability.h + * folly/lang/Align.h + * folly/ProducerConsumerQueue.h + +Copyright: Copyright (c) Facebook, Inc. and its affiliates. +Home page: https://github.com/facebook/folly +License: http://www.apache.org/licenses/LICENSE-2.0 + +-------------------------------------------------------------------------------- + +The file cpp/src/arrow/vendored/musl/strptime.c has the following license + +Copyright © 2005-2020 Rich Felker, et al. + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, +TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +-------------------------------------------------------------------------------- + +The file cpp/cmake_modules/BuildUtils.cmake contains code from + +https://gist.github.com/cristianadam/ef920342939a89fae3e8a85ca9459b49 + +which is made available under the MIT license + +Copyright (c) 2019 Cristian Adam + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +-------------------------------------------------------------------------------- + +The files in cpp/src/arrow/vendored/portable-snippets/ contain code from + +https://github.com/nemequ/portable-snippets + +and have the following copyright notice: + +Each source file contains a preamble explaining the license situation +for that file, which takes priority over this file. With the +exception of some code pulled in from other repositories (such as +µnit, an MIT-licensed project which is used for testing), the code is +public domain, released using the CC0 1.0 Universal dedication (*). + +(*) https://creativecommons.org/publicdomain/zero/1.0/legalcode + +-------------------------------------------------------------------------------- + +The files in cpp/src/arrow/vendored/fast_float/ contain code from + +https://github.com/lemire/fast_float + +which is made available under the Apache License 2.0. + +-------------------------------------------------------------------------------- + +The file python/pyarrow/vendored/docscrape.py contains code from + +https://github.com/numpy/numpydoc/ + +which is made available under the BSD 2-clause license. + +-------------------------------------------------------------------------------- + +The file python/pyarrow/vendored/version.py contains code from + +https://github.com/pypa/packaging/ + +which is made available under both the Apache license v2.0 and the +BSD 2-clause license. + +-------------------------------------------------------------------------------- + +The files in cpp/src/arrow/vendored/pcg contain code from + +https://github.com/imneme/pcg-cpp + +and have the following copyright notice: + +Copyright 2014-2019 Melissa O'Neill , + and the PCG Project contributors. + +SPDX-License-Identifier: (Apache-2.0 OR MIT) + +Licensed under the Apache License, Version 2.0 (provided in +LICENSE-APACHE.txt and at http://www.apache.org/licenses/LICENSE-2.0) +or under the MIT license (provided in LICENSE-MIT.txt and at +http://opensource.org/licenses/MIT), at your option. This file may not +be copied, modified, or distributed except according to those terms. + +Distributed on an "AS IS" BASIS, WITHOUT WARRANTY OF ANY KIND, either +express or implied. See your chosen license for details. + +-------------------------------------------------------------------------------- +r/R/dplyr-count-tally.R (some portions) + +Some portions of this file are derived from code from + +https://github.com/tidyverse/dplyr/ + +which is made available under the MIT license + +Copyright (c) 2013-2019 RStudio and others. + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the “Software”), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +-------------------------------------------------------------------------------- + +The file src/arrow/util/io_util.cc contains code from the CPython project +which is made available under the Python Software Foundation License Version 2. + +-------------------------------------------------------------------------------- + +3rdparty dependency opentelemetry-cpp is statically linked in certain binary +distributions. opentelemetry-cpp is made available under the Apache License 2.0. + +Copyright The OpenTelemetry Authors +SPDX-License-Identifier: OpenMDW-1.1 + +-------------------------------------------------------------------------------- + +ci/conan/ is based on code from Conan Package and Dependency Manager. + +Copyright (c) 2019 Conan.io + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +-------------------------------------------------------------------------------- + +3rdparty dependency UCX is redistributed as a dynamically linked shared +library in certain binary distributions. UCX has the following license: + +Copyright (c) 2014-2015 UT-Battelle, LLC. All rights reserved. +Copyright (C) 2014-2020 Mellanox Technologies Ltd. All rights reserved. +Copyright (C) 2014-2015 The University of Houston System. All rights reserved. +Copyright (C) 2015 The University of Tennessee and The University + of Tennessee Research Foundation. All rights reserved. +Copyright (C) 2016-2020 ARM Ltd. All rights reserved. +Copyright (c) 2016 Los Alamos National Security, LLC. All rights reserved. +Copyright (C) 2016-2020 Advanced Micro Devices, Inc. All rights reserved. +Copyright (C) 2019 UChicago Argonne, LLC. All rights reserved. +Copyright (c) 2018-2020 NVIDIA CORPORATION. All rights reserved. +Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved. +Copyright (C) 2016-2020 Stony Brook University. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +1. Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. +3. Neither the name of the copyright holder nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +-------------------------------------------------------------------------------- + +The file dev/tasks/r/github.packages.yml contains code from + +https://github.com/ursa-labs/arrow-r-nightly + +which is made available under the Apache License 2.0. + +-------------------------------------------------------------------------------- +.github/actions/sync-nightlies/action.yml (some portions) + +Some portions of this file are derived from code from + +https://github.com/JoshPiper/rsync-docker + +which is made available under the MIT license + +Copyright (c) 2020 Joshua Piper + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +-------------------------------------------------------------------------------- +.github/actions/sync-nightlies/action.yml (some portions) + +Some portions of this file are derived from code from + +https://github.com/burnett01/rsync-deployments + +which is made available under the MIT license + +Copyright (c) 2019-2022 Contention +Copyright (c) 2019-2022 Burnett01 + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +-------------------------------------------------------------------------------- +java/vector/src/main/java/org/apache/arrow/vector/util/IntObjectHashMap.java +java/vector/src/main/java/org/apache/arrow/vector/util/IntObjectMap.java + +These files are derived from code from Netty, which is made available under the +Apache License 2.0. + +-------------------------------------------------------------------------------- +cpp/src/arrow/util/math_internal.cc (some portions) + +Some portions of this file are derived from + +https://github.com/ankane/dist-rust/ + +which is made available under the MIT license + +The MIT License (MIT) + +Copyright (c) 2021-2023 Contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +-------------------------------------------------------------------------------- +The files cpp/src/arrow/vendored/whereami/whereami.h, +cpp/src/arrow/vendored/whereami/whereami.cc are adapted from +Grégory Pakosz's whereami library (https://github.com/gpakosz/whereami) +It is dual licensed under both the WTFPLv2 and MIT licenses. + +The WTFPLv2 License + DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE + Version 2, December 2004 + + Copyright (C) 2004 Sam Hocevar + + Everyone is permitted to copy and distribute verbatim or modified + copies of this license document, and changing it is allowed as long + as the name is changed. + + DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. You just DO WHAT THE FUCK YOU WANT TO. + 1. Bla bla bla + 2. Montesqieu et camembert, vive la France, zut alors! + +The MIT License (MIT) +Copyright Gregory Pakosz + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +pyasn1 +BSD-2-Clause +https://github.com/pyasn1/pyasn1 +Copyright (c) 2005-2020, Ilya Etingof +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + + +pyasn1_modules +BSD License +https://github.com/pyasn1/pyasn1-modules +Copyright (c) 2005-2020, Ilya Etingof +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + + +pycocotools +FreeBSD +https://github.com/ppwwyyxx/cocoapi +UNKNOWN + +pycparser +BSD-3-Clause +https://github.com/eliben/pycparser +pycparser -- A C parser in Python + +Copyright (c) 2008-2022, Eli Bendersky +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. +* Neither the name of the copyright holder nor the names of its contributors may + be used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE +GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +pycryptodomex +BSD License; Public Domain +https://www.pycryptodome.org +The source code in PyCryptodome is partially in the public domain +and partially released under the BSD 2-Clause license. + +In either case, there are minimal if no restrictions on the redistribution, +modification and usage of the software. + +Public domain +============= + +All code originating from PyCrypto is free and unencumbered software +released into the public domain. + +Anyone is free to copy, modify, publish, use, compile, sell, or +distribute this software, either in source code form or as a compiled +binary, for any purpose, commercial or non-commercial, and by any +means. + +In jurisdictions that recognize copyright laws, the author or authors +of this software dedicate any and all copyright interest in the +software to the public domain. We make this dedication for the benefit +of the public at large and to the detriment of our heirs and +successors. We intend this dedication to be an overt act of +relinquishment in perpetuity of all present and future rights to this +software under copyright law. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +OTHER DEALINGS IN THE SOFTWARE. + +For more information, please refer to + +BSD license +=========== + +All direct contributions to PyCryptodome are released under the following +license. The copyright of each piece belongs to the respective author. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +pydantic +MIT +https://github.com/pydantic/pydantic +The MIT License (MIT) + +Copyright (c) 2017 to present Pydantic Services Inc. and individual contributors. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +pydantic_core +MIT +https://github.com/pydantic/pydantic-core +The MIT License (MIT) + +Copyright (c) 2022 Samuel Colvin + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +pydub +MIT License +http://pydub.com +Copyright (c) 2011 James Robert, http://jiaaro.com + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +pygltflib +MIT License +https://gitlab.com/dodgyville/pygltflib +Copyright (c) 2018 Luke Miller + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +pyinstrument +BSD License +https://github.com/joerick/pyinstrument +Copyright (c) 2014-2020, Joe Rickerby and contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this +list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, +this list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its contributors +may be used to endorse or promote products derived from this software without +specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +pynput +GNU Lesser General Public License v3 (LGPLv3) +https://github.com/moses-palmer/pynput + GNU LESSER GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007-2024 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + + This version of the GNU Lesser General Public License incorporates +the terms and conditions of version 3 of the GNU General Public +License, supplemented by the additional permissions listed below. + + 0. Additional Definitions. + + As used herein, "this License" refers to version 3 of the GNU Lesser +General Public License, and the "GNU GPL" refers to version 3 of the GNU +General Public License. + + "The Library" refers to a covered work governed by this License, +other than an Application or a Combined Work as defined below. + + An "Application" is any work that makes use of an interface provided +by the Library, but which is not otherwise based on the Library. +Defining a subclass of a class defined by the Library is deemed a mode +of using an interface provided by the Library. + + A "Combined Work" is a work produced by combining or linking an +Application with the Library. The particular version of the Library +with which the Combined Work was made is also called the "Linked +Version". + + The "Minimal Corresponding Source" for a Combined Work means the +Corresponding Source for the Combined Work, excluding any source code +for portions of the Combined Work that, considered in isolation, are +based on the Application, and not on the Linked Version. + + The "Corresponding Application Code" for a Combined Work means the +object code and/or source code for the Application, including any data +and utility programs needed for reproducing the Combined Work from the +Application, but excluding the System Libraries of the Combined Work. + + 1. Exception to Section 3 of the GNU GPL. + + You may convey a covered work under sections 3 and 4 of this License +without being bound by section 3 of the GNU GPL. + + 2. Conveying Modified Versions. + + If you modify a copy of the Library, and, in your modifications, a +facility refers to a function or data to be supplied by an Application +that uses the facility (other than as an argument passed when the +facility is invoked), then you may convey a copy of the modified +version: + + a) under this License, provided that you make a good faith effort to + ensure that, in the event an Application does not supply the + function or data, the facility still operates, and performs + whatever part of its purpose remains meaningful, or + + b) under the GNU GPL, with none of the additional permissions of + this License applicable to that copy. + + 3. Object Code Incorporating Material from Library Header Files. + + The object code form of an Application may incorporate material from +a header file that is part of the Library. You may convey such object +code under terms of your choice, provided that, if the incorporated +material is not limited to numerical parameters, data structure +layouts and accessors, or small macros, inline functions and templates +(ten or fewer lines in length), you do both of the following: + + a) Give prominent notice with each copy of the object code that the + Library is used in it and that the Library and its use are + covered by this License. + + b) Accompany the object code with a copy of the GNU GPL and this license + document. + + 4. Combined Works. + + You may convey a Combined Work under terms of your choice that, +taken together, effectively do not restrict modification of the +portions of the Library contained in the Combined Work and reverse +engineering for debugging such modifications, if you also do each of +the following: + + a) Give prominent notice with each copy of the Combined Work that + the Library is used in it and that the Library and its use are + covered by this License. + + b) Accompany the Combined Work with a copy of the GNU GPL and this license + document. + + c) For a Combined Work that displays copyright notices during + execution, include the copyright notice for the Library among + these notices, as well as a reference directing the user to the + copies of the GNU GPL and this license document. + + d) Do one of the following: + + 0) Convey the Minimal Corresponding Source under the terms of this + License, and the Corresponding Application Code in a form + suitable for, and under terms that permit, the user to + recombine or relink the Application with a modified version of + the Linked Version to produce a modified Combined Work, in the + manner specified by section 6 of the GNU GPL for conveying + Corresponding Source. + + 1) Use a suitable shared library mechanism for linking with the + Library. A suitable mechanism is one that (a) uses at run time + a copy of the Library already present on the user's computer + system, and (b) will operate properly with a modified version + of the Library that is interface-compatible with the Linked + Version. + + e) Provide Installation Information, but only if you would otherwise + be required to provide such information under section 6 of the + GNU GPL, and only to the extent that such information is + necessary to install and execute a modified version of the + Combined Work produced by recombining or relinking the + Application with a modified version of the Linked Version. (If + you use option 4d0, the Installation Information must accompany + the Minimal Corresponding Source and Corresponding Application + Code. If you use option 4d1, you must provide the Installation + Information in the manner specified by section 6 of the GNU GPL + for conveying Corresponding Source.) + + 5. Combined Libraries. + + You may place library facilities that are a work based on the +Library side by side in a single library together with other library +facilities that are not Applications and are not covered by this +License, and convey such a combined library under terms of your +choice, if you do both of the following: + + a) Accompany the combined library with a copy of the same work based + on the Library, uncombined with any other library facilities, + conveyed under the terms of this License. + + b) Give prominent notice with the combined library that part of it + is a work based on the Library, and explaining where to find the + accompanying uncombined form of the same work. + + 6. Revised Versions of the GNU Lesser General Public License. + + The Free Software Foundation may publish revised and/or new versions +of the GNU Lesser General Public License from time to time. Such new +versions will be similar in spirit to the present version, but may +differ in detail to address new problems or concerns. + + Each version is given a distinguishing version number. If the +Library as you received it specifies that a certain numbered version +of the GNU Lesser General Public License "or any later version" +applies to it, you have the option of following the terms and +conditions either of that published version or of any later version +published by the Free Software Foundation. If the Library as you +received it does not specify a version number of the GNU Lesser +General Public License, you may choose any version of the GNU Lesser +General Public License ever published by the Free Software Foundation. + + If the Library as you received it specifies that a proxy can decide +whether future versions of the GNU Lesser General Public License shall +apply, that proxy's public statement of acceptance of any version is +permanent authorization for you to choose that version for the +Library. + + +pyparsing +MIT +https://github.com/pyparsing/pyparsing/ +Copyright (c) 2003-2025 Paul McGuire + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, +TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +pyproject_hooks +MIT License +https://github.com/pypa/pyproject-hooks +The MIT License (MIT) + +Copyright (c) 2017 Thomas Kluyver + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +pyrefly +UNKNOWN +https://pyrefly.org +UNKNOWN + +pyserial +BSD License +https://github.com/pyserial/pyserial +UNKNOWN + +pytest +MIT +https://docs.pytest.org/en/latest/ +The MIT License (MIT) + +Copyright (c) 2004 Holger Krekel and others + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +pytest-cov +MIT +https://pytest-cov.readthedocs.io/en/latest/changelog.html +The MIT License + +Copyright (c) 2010 Meme Dough + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +pytest-custom-exit-code +MIT License +https://github.com/yashtodi94/pytest-custom_exit_code + +The MIT License (MIT) + +Copyright (c) 2019 Yash Todi + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +pytest-datadir +MIT License +http://github.com/gabrielcnr/pytest-datadir +This is the MIT license: http://www.opensource.org/licenses/mit-license.php + +Copyright (c) 2015-2022 the pytest-datadir authors and contributors . + +Permission is hereby granted, free of charge, to any person obtaining a copy of this +software and associated documentation files (the "Software"), to deal in the Software +without restriction, including without limitation the rights to use, copy, modify, merge, +publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons +to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or +substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, +INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR +PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE +FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR +OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. + + +pytest-env +MIT License +https://github.com/pytest-dev/pytest-env +MIT License + +Copyright (c) 2010-202x The pytest-env developers + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +pytest-instafail +BSD License +https://github.com/pytest-dev/pytest-instafail +Copyright (c) 2013-2016, Janne Vanhala + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* The names of the contributors may not be used to endorse or promote products + derived from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY DIRECT, +INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, +BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE +OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF +ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +pytest-regressions +MIT License +https://github.com/ESSS/pytest-regressions + +The MIT License (MIT) + +Copyright (c) 2018 ESSS + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +pytest-xdist +MIT +https://github.com/pytest-dev/pytest-xdist +MIT License + +Copyright (c) 2010 Holger Krekel and contributors. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +python-dateutil +Apache Software License; BSD License +https://github.com/dateutil/dateutil +Copyright 2017- Paul Ganssle +Copyright 2017- dateutil contributors (see AUTHORS file) + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +The above license applies to all contributions after 2017-12-01, as well as +all contributions that have been re-licensed (see AUTHORS file for the list of +contributors who have re-licensed their code). +-------------------------------------------------------------------------------- +dateutil - Extensions to the standard Python datetime module. + +Copyright (c) 2003-2011 - Gustavo Niemeyer +Copyright (c) 2012-2014 - Tomi Pieviläinen +Copyright (c) 2014-2016 - Yaron de Leeuw +Copyright (c) 2015- - Paul Ganssle +Copyright (c) 2015- - dateutil contributors (see AUTHORS file) + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + * Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +The above BSD License Applies to all code, even that also covered by Apache 2.0. + +python-discovery +MIT License +https://github.com/tox-dev/python-discovery +Permission is hereby granted, free of charge, to any person obtaining a +copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be included +in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS +OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, +TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +python-dotenv +BSD-3-Clause +https://github.com/theskumar/python-dotenv +Copyright (c) 2014, Saurabh Kumar (python-dotenv), 2013, Ted Tieken (django-dotenv-rw), 2013, Jacob Kaplan-Moss (django-dotenv) + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + +- Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + +- Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +- Neither the name of django-dotenv nor the names of its contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +python-json-logger +BSD License +https://nhairs.github.io/python-json-logger +Copyright (c) 2011, Zakaria Zajac and the python-json-logger Contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. +* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +python-memcached +Python Software Foundation License +https://github.com/linsomniac/python-memcached +UNKNOWN + +python-multipart +Apache-2.0 +https://github.com/Kludex/python-multipart +Copyright 2012, Andrew Dunham + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + + + +python-xlib +GNU Lesser General Public License v2 or later (LGPLv2+) +https://github.com/python-xlib/python-xlib + GNU LESSER GENERAL PUBLIC LICENSE + Version 2.1, February 1999 + + Copyright (C) 1991, 1999 Free Software Foundation, Inc. + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + +[This is the first released version of the Lesser GPL. It also counts + as the successor of the GNU Library Public License, version 2, hence + the version number 2.1.] + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +Licenses are intended to guarantee your freedom to share and change +free software--to make sure the software is free for all its users. + + This license, the Lesser General Public License, applies to some +specially designated software packages--typically libraries--of the +Free Software Foundation and other authors who decide to use it. You +can use it too, but we suggest you first think carefully about whether +this license or the ordinary General Public License is the better +strategy to use in any particular case, based on the explanations below. + + When we speak of free software, we are referring to freedom of use, +not price. Our General Public Licenses are designed to make sure that +you have the freedom to distribute copies of free software (and charge +for this service if you wish); that you receive source code or can get +it if you want it; that you can change the software and use pieces of +it in new free programs; and that you are informed that you can do +these things. + + To protect your rights, we need to make restrictions that forbid +distributors to deny you these rights or to ask you to surrender these +rights. These restrictions translate to certain responsibilities for +you if you distribute copies of the library or if you modify it. + + For example, if you distribute copies of the library, whether gratis +or for a fee, you must give the recipients all the rights that we gave +you. You must make sure that they, too, receive or can get the source +code. If you link other code with the library, you must provide +complete object files to the recipients, so that they can relink them +with the library after making changes to the library and recompiling +it. And you must show them these terms so they know their rights. + + We protect your rights with a two-step method: (1) we copyright the +library, and (2) we offer you this license, which gives you legal +permission to copy, distribute and/or modify the library. + + To protect each distributor, we want to make it very clear that +there is no warranty for the free library. Also, if the library is +modified by someone else and passed on, the recipients should know +that what they have is not the original version, so that the original +author's reputation will not be affected by problems that might be +introduced by others. + + Finally, software patents pose a constant threat to the existence of +any free program. We wish to make sure that a company cannot +effectively restrict the users of a free program by obtaining a +restrictive license from a patent holder. Therefore, we insist that +any patent license obtained for a version of the library must be +consistent with the full freedom of use specified in this license. + + Most GNU software, including some libraries, is covered by the +ordinary GNU General Public License. This license, the GNU Lesser +General Public License, applies to certain designated libraries, and +is quite different from the ordinary General Public License. We use +this license for certain libraries in order to permit linking those +libraries into non-free programs. + + When a program is linked with a library, whether statically or using +a shared library, the combination of the two is legally speaking a +combined work, a derivative of the original library. The ordinary +General Public License therefore permits such linking only if the +entire combination fits its criteria of freedom. The Lesser General +Public License permits more lax criteria for linking other code with +the library. + + We call this license the "Lesser" General Public License because it +does Less to protect the user's freedom than the ordinary General +Public License. It also provides other free software developers Less +of an advantage over competing non-free programs. These disadvantages +are the reason we use the ordinary General Public License for many +libraries. However, the Lesser license provides advantages in certain +special circumstances. + + For example, on rare occasions, there may be a special need to +encourage the widest possible use of a certain library, so that it becomes +a de-facto standard. To achieve this, non-free programs must be +allowed to use the library. A more frequent case is that a free +library does the same job as widely used non-free libraries. In this +case, there is little to gain by limiting the free library to free +software only, so we use the Lesser General Public License. + + In other cases, permission to use a particular library in non-free +programs enables a greater number of people to use a large body of +free software. For example, permission to use the GNU C Library in +non-free programs enables many more people to use the whole GNU +operating system, as well as its variant, the GNU/Linux operating +system. + + Although the Lesser General Public License is Less protective of the +users' freedom, it does ensure that the user of a program that is +linked with the Library has the freedom and the wherewithal to run +that program using a modified version of the Library. + + The precise terms and conditions for copying, distribution and +modification follow. Pay close attention to the difference between a +"work based on the library" and a "work that uses the library". The +former contains code derived from the library, whereas the latter must +be combined with the library in order to run. + + GNU LESSER GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License Agreement applies to any software library or other +program which contains a notice placed by the copyright holder or +other authorized party saying it may be distributed under the terms of +this Lesser General Public License (also called "this License"). +Each licensee is addressed as "you". + + A "library" means a collection of software functions and/or data +prepared so as to be conveniently linked with application programs +(which use some of those functions and data) to form executables. + + The "Library", below, refers to any such software library or work +which has been distributed under these terms. A "work based on the +Library" means either the Library or any derivative work under +copyright law: that is to say, a work containing the Library or a +portion of it, either verbatim or with modifications and/or translated +straightforwardly into another language. (Hereinafter, translation is +included without limitation in the term "modification".) + + "Source code" for a work means the preferred form of the work for +making modifications to it. For a library, complete source code means +all the source code for all modules it contains, plus any associated +interface definition files, plus the scripts used to control compilation +and installation of the library. + + Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running a program using the Library is not restricted, and output from +such a program is covered only if its contents constitute a work based +on the Library (independent of the use of the Library in a tool for +writing it). Whether that is true depends on what the Library does +and what the program that uses the Library does. + + 1. You may copy and distribute verbatim copies of the Library's +complete source code as you receive it, in any medium, provided that +you conspicuously and appropriately publish on each copy an +appropriate copyright notice and disclaimer of warranty; keep intact +all the notices that refer to this License and to the absence of any +warranty; and distribute a copy of this License along with the +Library. + + You may charge a fee for the physical act of transferring a copy, +and you may at your option offer warranty protection in exchange for a +fee. + + 2. You may modify your copy or copies of the Library or any portion +of it, thus forming a work based on the Library, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) The modified work must itself be a software library. + + b) You must cause the files modified to carry prominent notices + stating that you changed the files and the date of any change. + + c) You must cause the whole of the work to be licensed at no + charge to all third parties under the terms of this License. + + d) If a facility in the modified Library refers to a function or a + table of data to be supplied by an application program that uses + the facility, other than as an argument passed when the facility + is invoked, then you must make a good faith effort to ensure that, + in the event an application does not supply such function or + table, the facility still operates, and performs whatever part of + its purpose remains meaningful. + + (For example, a function in a library to compute square roots has + a purpose that is entirely well-defined independent of the + application. Therefore, Subsection 2d requires that any + application-supplied function or table used by this function must + be optional: if the application does not supply it, the square + root function must still compute square roots.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Library, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Library, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote +it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Library. + +In addition, mere aggregation of another work not based on the Library +with the Library (or with a work based on the Library) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may opt to apply the terms of the ordinary GNU General Public +License instead of this License to a given copy of the Library. To do +this, you must alter all the notices that refer to this License, so +that they refer to the ordinary GNU General Public License, version 2, +instead of to this License. (If a newer version than version 2 of the +ordinary GNU General Public License has appeared, then you can specify +that version instead if you wish.) Do not make any other change in +these notices. + + Once this change is made in a given copy, it is irreversible for +that copy, so the ordinary GNU General Public License applies to all +subsequent copies and derivative works made from that copy. + + This option is useful when you wish to copy part of the code of +the Library into a program that is not a library. + + 4. You may copy and distribute the Library (or a portion or +derivative of it, under Section 2) in object code or executable form +under the terms of Sections 1 and 2 above provided that you accompany +it with the complete corresponding machine-readable source code, which +must be distributed under the terms of Sections 1 and 2 above on a +medium customarily used for software interchange. + + If distribution of object code is made by offering access to copy +from a designated place, then offering equivalent access to copy the +source code from the same place satisfies the requirement to +distribute the source code, even though third parties are not +compelled to copy the source along with the object code. + + 5. A program that contains no derivative of any portion of the +Library, but is designed to work with the Library by being compiled or +linked with it, is called a "work that uses the Library". Such a +work, in isolation, is not a derivative work of the Library, and +therefore falls outside the scope of this License. + + However, linking a "work that uses the Library" with the Library +creates an executable that is a derivative of the Library (because it +contains portions of the Library), rather than a "work that uses the +library". The executable is therefore covered by this License. +Section 6 states terms for distribution of such executables. + + When a "work that uses the Library" uses material from a header file +that is part of the Library, the object code for the work may be a +derivative work of the Library even though the source code is not. +Whether this is true is especially significant if the work can be +linked without the Library, or if the work is itself a library. The +threshold for this to be true is not precisely defined by law. + + If such an object file uses only numerical parameters, data +structure layouts and accessors, and small macros and small inline +functions (ten lines or less in length), then the use of the object +file is unrestricted, regardless of whether it is legally a derivative +work. (Executables containing this object code plus portions of the +Library will still fall under Section 6.) + + Otherwise, if the work is a derivative of the Library, you may +distribute the object code for the work under the terms of Section 6. +Any executables containing that work also fall under Section 6, +whether or not they are linked directly with the Library itself. + + 6. As an exception to the Sections above, you may also combine or +link a "work that uses the Library" with the Library to produce a +work containing portions of the Library, and distribute that work +under terms of your choice, provided that the terms permit +modification of the work for the customer's own use and reverse +engineering for debugging such modifications. + + You must give prominent notice with each copy of the work that the +Library is used in it and that the Library and its use are covered by +this License. You must supply a copy of this License. If the work +during execution displays copyright notices, you must include the +copyright notice for the Library among them, as well as a reference +directing the user to the copy of this License. Also, you must do one +of these things: + + a) Accompany the work with the complete corresponding + machine-readable source code for the Library including whatever + changes were used in the work (which must be distributed under + Sections 1 and 2 above); and, if the work is an executable linked + with the Library, with the complete machine-readable "work that + uses the Library", as object code and/or source code, so that the + user can modify the Library and then relink to produce a modified + executable containing the modified Library. (It is understood + that the user who changes the contents of definitions files in the + Library will not necessarily be able to recompile the application + to use the modified definitions.) + + b) Use a suitable shared library mechanism for linking with the + Library. A suitable mechanism is one that (1) uses at run time a + copy of the library already present on the user's computer system, + rather than copying library functions into the executable, and (2) + will operate properly with a modified version of the library, if + the user installs one, as long as the modified version is + interface-compatible with the version that the work was made with. + + c) Accompany the work with a written offer, valid for at + least three years, to give the same user the materials + specified in Subsection 6a, above, for a charge no more + than the cost of performing this distribution. + + d) If distribution of the work is made by offering access to copy + from a designated place, offer equivalent access to copy the above + specified materials from the same place. + + e) Verify that the user has already received a copy of these + materials or that you have already sent this user a copy. + + For an executable, the required form of the "work that uses the +Library" must include any data and utility programs needed for +reproducing the executable from it. However, as a special exception, +the materials to be distributed need not include anything that is +normally distributed (in either source or binary form) with the major +components (compiler, kernel, and so on) of the operating system on +which the executable runs, unless that component itself accompanies +the executable. + + It may happen that this requirement contradicts the license +restrictions of other proprietary libraries that do not normally +accompany the operating system. Such a contradiction means you cannot +use both them and the Library together in an executable that you +distribute. + + 7. You may place library facilities that are a work based on the +Library side-by-side in a single library together with other library +facilities not covered by this License, and distribute such a combined +library, provided that the separate distribution of the work based on +the Library and of the other library facilities is otherwise +permitted, and provided that you do these two things: + + a) Accompany the combined library with a copy of the same work + based on the Library, uncombined with any other library + facilities. This must be distributed under the terms of the + Sections above. + + b) Give prominent notice with the combined library of the fact + that part of it is a work based on the Library, and explaining + where to find the accompanying uncombined form of the same work. + + 8. You may not copy, modify, sublicense, link with, or distribute +the Library except as expressly provided under this License. Any +attempt otherwise to copy, modify, sublicense, link with, or +distribute the Library is void, and will automatically terminate your +rights under this License. However, parties who have received copies, +or rights, from you under this License will not have their licenses +terminated so long as such parties remain in full compliance. + + 9. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Library or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Library (or any work based on the +Library), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Library or works based on it. + + 10. Each time you redistribute the Library (or any work based on the +Library), the recipient automatically receives a license from the +original licensor to copy, distribute, link with or modify the Library +subject to these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties with +this License. + + 11. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Library at all. For example, if a patent +license would not permit royalty-free redistribution of the Library by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Library. + +If any portion of this section is held invalid or unenforceable under any +particular circumstance, the balance of the section is intended to apply, +and the section as a whole is intended to apply in other circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 12. If the distribution and/or use of the Library is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Library under this License may add +an explicit geographical distribution limitation excluding those countries, +so that distribution is permitted only in or among countries not thus +excluded. In such case, this License incorporates the limitation as if +written in the body of this License. + + 13. The Free Software Foundation may publish revised and/or new +versions of the Lesser General Public License from time to time. +Such new versions will be similar in spirit to the present version, +but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Library +specifies a version number of this License which applies to it and +"any later version", you have the option of following the terms and +conditions either of that version or of any later version published by +the Free Software Foundation. If the Library does not specify a +license version number, you may choose any version ever published by +the Free Software Foundation. + + 14. If you wish to incorporate parts of the Library into other free +programs whose distribution conditions are incompatible with these, +write to the author to ask for permission. For software which is +copyrighted by the Free Software Foundation, write to the Free +Software Foundation; we sometimes make exceptions for this. Our +decision will be guided by the two goals of preserving the free status +of all derivatives of our free software and of promoting the sharing +and reuse of software generally. + + NO WARRANTY + + 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO +WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. +EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR +OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY +KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE +LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME +THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN +WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY +AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU +FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR +CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE +LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING +RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A +FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF +SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH +DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Libraries + + If you develop a new library, and you want it to be of the greatest +possible use to the public, we recommend making it free software that +everyone can redistribute and change. You can do so by permitting +redistribution under these terms (or, alternatively, under the terms of the +ordinary General Public License). + + To apply these terms, attach the following notices to the library. It is +safest to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least the +"copyright" line and a pointer to where the full notice is found. + + {description} + Copyright (C) {year} {fullname} + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 + USA + +Also add information on how to contact you by electronic and paper mail. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the library, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the + library `Frob' (a library for tweaking knobs) written by James Random + Hacker. + + {signature of Ty Coon}, 1 April 1990 + Ty Coon, President of Vice + +That's all there is to it! + + +pytorch-lightning +Apache Software License +https://github.com/Lightning-AI/lightning + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2018-2021 William Falcon + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +pytorch-ranger +MIT License +https://github.com/mpariente/Ranger-Deep-Learning-Optimizer + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +pytz +MIT License +http://pythonhosted.org/pytz +Copyright (c) 2003-2019 Stuart Bishop + +Permission is hereby granted, free of charge, to any person obtaining a +copy of this software and associated documentation files (the "Software"), +to deal in the Software without restriction, including without limitation +the rights to use, copy, modify, merge, publish, distribute, sublicense, +and/or sell copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. + + +pyzmq +BSD License +https://pyzmq.readthedocs.org +BSD 3-Clause License + +Copyright (c) 2009-2012, Brian Granger, Min Ragan-Kelley + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +qwen-vl-utils +Apache Software License +https://github.com/QwenLM/Qwen2-VL/tree/main/qwen-vl-utils +UNKNOWN + +ray +Apache 2.0 +https://github.com/ray-project/ray +A. HISTORY OF THE SOFTWARE +========================== + +Python was created in the early 1990s by Guido van Rossum at Stichting +Mathematisch Centrum (CWI, see https://www.cwi.nl) in the Netherlands +as a successor of a language called ABC. Guido remains Python's +principal author, although it includes many contributions from others. + +In 1995, Guido continued his work on Python at the Corporation for +National Research Initiatives (CNRI, see https://www.cnri.reston.va.us) +in Reston, Virginia where he released several versions of the +software. + +In May 2000, Guido and the Python core development team moved to +BeOpen.com to form the BeOpen PythonLabs team. In October of the same +year, the PythonLabs team moved to Digital Creations, which became +Zope Corporation. In 2001, the Python Software Foundation (PSF, see +https://www.python.org/psf/) was formed, a non-profit organization +created specifically to own Python-related Intellectual Property. +Zope Corporation was a sponsoring member of the PSF. + +All Python releases are Open Source (see https://opensource.org for +the Open Source Definition). Historically, most, but not all, Python +releases have also been GPL-compatible; the table below summarizes +the various releases. + + Release Derived Year Owner GPL- + from compatible? (1) + + 0.9.0 thru 1.2 1991-1995 CWI yes + 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes + 1.6 1.5.2 2000 CNRI no + 2.0 1.6 2000 BeOpen.com no + 1.6.1 1.6 2001 CNRI yes (2) + 2.1 2.0+1.6.1 2001 PSF no + 2.0.1 2.0+1.6.1 2001 PSF yes + 2.1.1 2.1+2.0.1 2001 PSF yes + 2.1.2 2.1.1 2002 PSF yes + 2.1.3 2.1.2 2002 PSF yes + 2.2 and above 2.1.1 2001-now PSF yes + +Footnotes: + +(1) GPL-compatible doesn't mean that we're distributing Python under + the GPL. All Python licenses, unlike the GPL, let you distribute + a modified version without making your changes open source. The + GPL-compatible licenses make it possible to combine Python with + other software that is released under the GPL; the others don't. + +(2) According to Richard Stallman, 1.6.1 is not GPL-compatible, + because its license has a choice of law clause. According to + CNRI, however, Stallman's lawyer has told CNRI's lawyer that 1.6.1 + is "not incompatible" with the GPL. + +Thanks to the many outside volunteers who have worked under Guido's +direction to make these releases possible. + + +B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON +=============================================================== + +Python software and documentation are licensed under the +Python Software Foundation License Version 2. + +Starting with Python 3.8.6, examples, recipes, and other code in +the documentation are dual licensed under the PSF License Version 2 +and the Zero-Clause BSD license. + +Some software incorporated into Python is under different licenses. +The licenses are listed with code falling under that license. + + +PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 +-------------------------------------------- + +1. This LICENSE AGREEMENT is between the Python Software Foundation +("PSF"), and the Individual or Organization ("Licensee") accessing and +otherwise using this software ("Python") in source or binary form and +its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, PSF hereby +grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, +analyze, test, perform and/or display publicly, prepare derivative works, +distribute, and otherwise use Python alone or in any derivative version, +provided, however, that PSF's License Agreement and PSF's notice of copyright, +i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, +2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023 Python Software Foundation; +All Rights Reserved" are retained in Python alone or in any derivative version +prepared by Licensee. + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python. + +4. PSF is making Python available to Licensee on an "AS IS" +basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. Nothing in this License Agreement shall be deemed to create any +relationship of agency, partnership, or joint venture between PSF and +Licensee. This License Agreement does not grant permission to use PSF +trademarks or trade name in a trademark sense to endorse or promote +products or services of Licensee, or any third party. + +8. By copying, installing or otherwise using Python, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0 +------------------------------------------- + +BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1 + +1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an +office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the +Individual or Organization ("Licensee") accessing and otherwise using +this software in source or binary form and its associated +documentation ("the Software"). + +2. Subject to the terms and conditions of this BeOpen Python License +Agreement, BeOpen hereby grants Licensee a non-exclusive, +royalty-free, world-wide license to reproduce, analyze, test, perform +and/or display publicly, prepare derivative works, distribute, and +otherwise use the Software alone or in any derivative version, +provided, however, that the BeOpen Python License is retained in the +Software, alone or in any derivative version prepared by Licensee. + +3. BeOpen is making the Software available to Licensee on an "AS IS" +basis. BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE +SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS +AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY +DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +5. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +6. This License Agreement shall be governed by and interpreted in all +respects by the law of the State of California, excluding conflict of +law provisions. Nothing in this License Agreement shall be deemed to +create any relationship of agency, partnership, or joint venture +between BeOpen and Licensee. This License Agreement does not grant +permission to use BeOpen trademarks or trade names in a trademark +sense to endorse or promote products or services of Licensee, or any +third party. As an exception, the "BeOpen Python" logos available at +http://www.pythonlabs.com/logos.html may be used according to the +permissions granted on that web page. + +7. By copying, installing or otherwise using the software, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1 +--------------------------------------- + +1. This LICENSE AGREEMENT is between the Corporation for National +Research Initiatives, having an office at 1895 Preston White Drive, +Reston, VA 20191 ("CNRI"), and the Individual or Organization +("Licensee") accessing and otherwise using Python 1.6.1 software in +source or binary form and its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, CNRI +hereby grants Licensee a nonexclusive, royalty-free, world-wide +license to reproduce, analyze, test, perform and/or display publicly, +prepare derivative works, distribute, and otherwise use Python 1.6.1 +alone or in any derivative version, provided, however, that CNRI's +License Agreement and CNRI's notice of copyright, i.e., "Copyright (c) +1995-2001 Corporation for National Research Initiatives; All Rights +Reserved" are retained in Python 1.6.1 alone or in any derivative +version prepared by Licensee. Alternately, in lieu of CNRI's License +Agreement, Licensee may substitute the following text (omitting the +quotes): "Python 1.6.1 is made available subject to the terms and +conditions in CNRI's License Agreement. This Agreement together with +Python 1.6.1 may be located on the internet using the following +unique, persistent identifier (known as a handle): 1895.22/1013. This +Agreement may also be obtained from a proxy server on the internet +using the following URL: http://hdl.handle.net/1895.22/1013". + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python 1.6.1 or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python 1.6.1. + +4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS" +basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1 WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. This License Agreement shall be governed by the federal +intellectual property law of the United States, including without +limitation the federal copyright law, and, to the extent such +U.S. federal law does not apply, by the law of the Commonwealth of +Virginia, excluding Virginia's conflict of law provisions. +Notwithstanding the foregoing, with regard to derivative works based +on Python 1.6.1 that incorporate non-separable material that was +previously distributed under the GNU General Public License (GPL), the +law of the Commonwealth of Virginia shall govern this License +Agreement only as to issues arising under or with respect to +Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this +License Agreement shall be deemed to create any relationship of +agency, partnership, or joint venture between CNRI and Licensee. This +License Agreement does not grant permission to use CNRI trademarks or +trade name in a trademark sense to endorse or promote products or +services of Licensee, or any third party. + +8. By clicking on the "ACCEPT" button where indicated, or by copying, +installing or otherwise using Python 1.6.1, Licensee agrees to be +bound by the terms and conditions of this License Agreement. + + ACCEPT + + +CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2 +-------------------------------------------------- + +Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, +The Netherlands. All rights reserved. + +Permission to use, copy, modify, and distribute this software and its +documentation for any purpose and without fee is hereby granted, +provided that the above copyright notice appear in all copies and that +both that copyright notice and this permission notice appear in +supporting documentation, and that the name of Stichting Mathematisch +Centrum or CWI not be used in advertising or publicity pertaining to +distribution of the software without specific, written prior +permission. + +STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO +THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND +FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE +FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT +OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + +ZERO-CLAUSE BSD LICENSE FOR CODE IN THE PYTHON DOCUMENTATION +---------------------------------------------------------------------- + +Permission to use, copy, modify, and/or distribute this software for any +purpose with or without fee is hereby granted. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH +REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY +AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, +INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM +LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR +OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR +PERFORMANCE OF THIS SOFTWARE. + + +referencing +MIT +https://github.com/python-jsonschema/referencing +Copyright (c) 2022 Julian Berman + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +regex +Apache-2.0 AND CNRI-Python +https://github.com/mrabarnett/mrab-regex +This work was derived from the 're' module of CPython 2.6 and CPython 3.1, +copyright (c) 1998-2001 by Secret Labs AB and licensed under CNRI's Python 1.6 +license. + +All additions and alterations are licensed under the Apache 2.0 License. + + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2020 Matthew Barnett + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +requests +Apache Software License +https://requests.readthedocs.io + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + +retinaface-py +MIT License +https://github.com/andresprados/Pytorch_Retinaface +MIT License + +Copyright (c) 2019 + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +rfc3339-validator +MIT License +https://github.com/naimetti/rfc3339-validator +MIT License + +Copyright (c) 2019, Nicolas Aimetti + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + + +rfc3986-validator +MIT License +https://github.com/naimetti/rfc3986-validator +MIT License + +Copyright (c) 2019, Nicolas Aimetti + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + + +rfc3987-syntax +MIT +https://github.com/willynilly/rfc3987-syntax +MIT License + +Copyright (c) 2025 Will Riley + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +rich +MIT License +https://github.com/Textualize/rich +Copyright (c) 2020 Will McGugan + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +robotmq +MIT License +https://github.com/yihuai-gao/robot-message-queue +MIT License + +Copyright (c) 2024 Yihuai Gao + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +rpds-py +MIT +https://github.com/crate-py/rpds +Copyright (c) 2023 Julian Berman + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +rsa +Apache Software License +https://stuvel.eu/rsa +Copyright 2011 Sybren A. Stüvel + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + + +ruff +MIT License +https://docs.astral.sh/ruff +MIT License + +Copyright (c) 2022 Charles Marsh + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +end of terms and conditions + +The externally maintained libraries from which parts of the Software is derived +are: + +- flake8-comprehensions, licensed as follows: + """ + MIT License + + Copyright (c) 2017 Adam Johnson + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-no-pep420, licensed as follows: + """ + MIT License + + Copyright (c) 2020 Adam Johnson + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-tidy-imports, licensed as follows: + """ + MIT License + + Copyright (c) 2017 Adam Johnson + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-return, licensed as follows: + """ + MIT License + + Copyright (c) 2019 Afonasev Evgeniy + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-2020, licensed as follows: + """ + Copyright (c) 2019 Anthony Sottile + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. + """ + +- pyupgrade, licensed as follows: + """ + Copyright (c) 2017 Anthony Sottile + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. + """ + +- flake8-blind-except, licensed as follows: + """ + The MIT License (MIT) + + Copyright (c) 2014 Elijah Andrews + + Permission is hereby granted, free of charge, to any person obtaining a copy of + this software and associated documentation files (the "Software"), to deal in + the Software without restriction, including without limitation the rights to + use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of + the Software, and to permit persons to whom the Software is furnished to do so, + subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER + IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + """ + +- flake8-gettext, licensed as follows: + """ + BSD Zero Clause License + + Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted. + + THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + """ + +- flake8-implicit-str-concat, licensed as follows: + """ + The MIT License (MIT) + + Copyright (c) 2019 Dylan Turner + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. + """ + +- flake8-debugger, licensed as follows: + """ + MIT License + + Copyright (c) 2016 Joseph Kahn + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-pyi, licensed as follows: + """ + The MIT License (MIT) + + Copyright (c) 2016 Łukasz Langa + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-print, licensed as follows: + """ + MIT License + + Copyright (c) 2016 Joseph Kahn + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-import-conventions, licensed as follows: + """ + MIT License + + Copyright (c) 2021 João Palmeiro + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-simplify, licensed as follows: + """ + MIT License + + Copyright (c) 2020 Martin Thoma + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-slots, licensed as follows: + """ + Copyright (c) 2021 Dominic Davis-Foster + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, + DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR + OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE + OR OTHER DEALINGS IN THE SOFTWARE. + """ + +- flake8-todos, licensed as follows: + """ + Copyright (c) 2019 EclecticIQ. All rights reserved. + Copyright (c) 2020 Gram . All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are met: + + 1. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + + 3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from this + software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE + FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + """ + +- flake8-unused-arguments, licensed as follows: + """ + MIT License + + Copyright (c) 2019 Nathan Hoad + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- pygrep-hooks, licensed as follows: + """ + Copyright (c) 2018 Anthony Sottile + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. + """ + +- autoflake, licensed as follows: + """ + Copyright (C) 2012-2018 Steven Myint + + Permission is hereby granted, free of charge, to any person obtaining a copy of + this software and associated documentation files (the "Software"), to deal in + the Software without restriction, including without limitation the rights to + use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies + of the Software, and to permit persons to whom the Software is furnished to do + so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- autotyping, licensed as follows: + """ + MIT License + + Copyright (c) 2023 Jelle Zijlstra + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- Flake8, licensed as follows: + """ + == Flake8 License (MIT) == + + Copyright (C) 2011-2013 Tarek Ziade + Copyright (C) 2012-2016 Ian Cordasco + + Permission is hereby granted, free of charge, to any person obtaining a copy of + this software and associated documentation files (the "Software"), to deal in + the Software without restriction, including without limitation the rights to + use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies + of the Software, and to permit persons to whom the Software is furnished to do + so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-bugbear, licensed as follows: + """ + The MIT License (MIT) + + Copyright (c) 2016 Łukasz Langa + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-commas, licensed as follows: + """ + The MIT License (MIT) + + Copyright (c) 2017 Thomas Grainger. + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. + + + Portions of this flake8-commas Software may utilize the following + copyrighted material, the use of which is hereby acknowledged. + + Original flake8-commas: https://github.com/trevorcreech/flake8-commas/commit/e8563b71b1d5442e102c8734c11cb5202284293d + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. + """ + +- flynt, licensed as follows: + """ + MIT License + + Copyright (c) 2019-2022 Ilya Kamenshchikov + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- isort, licensed as follows: + """ + The MIT License (MIT) + + Copyright (c) 2013 Timothy Edmund Crosley + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. + """ + +- pep8-naming, licensed as follows: + """ + Copyright © 2013 Florent Xicluna + + Licensed under the terms of the Expat License + + Permission is hereby granted, free of charge, to any person + obtaining a copy of this software and associated documentation files + (the "Software"), to deal in the Software without restriction, + including without limitation the rights to use, copy, modify, merge, + publish, distribute, sublicense, and/or sell copies of the Software, + and to permit persons to whom the Software is furnished to do so, + subject to the following conditions: + + The above copyright notice and this permission notice shall be + included in all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- pycodestyle, licensed as follows: + """ + Copyright © 2006-2009 Johann C. Rocholl + Copyright © 2009-2014 Florent Xicluna + Copyright © 2014-2020 Ian Lee + + Licensed under the terms of the Expat License + + Permission is hereby granted, free of charge, to any person + obtaining a copy of this software and associated documentation files + (the "Software"), to deal in the Software without restriction, + including without limitation the rights to use, copy, modify, merge, + publish, distribute, sublicense, and/or sell copies of the Software, + and to permit persons to whom the Software is furnished to do so, + subject to the following conditions: + + The above copyright notice and this permission notice shall be + included in all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- pydocstyle, licensed as follows: + """ + Copyright (c) 2012 GreenSteam, + + Copyright (c) 2014-2020 Amir Rachum, + + Copyright (c) 2020 Sambhav Kothari, + + Permission is hereby granted, free of charge, to any person obtaining a copy of + this software and associated documentation files (the "Software"), to deal in + the Software without restriction, including without limitation the rights to + use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies + of the Software, and to permit persons to whom the Software is furnished to do + so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- Pyflakes, licensed as follows: + """ + Copyright 2005-2011 Divmod, Inc. + Copyright 2013-2014 Florent Xicluna + + Permission is hereby granted, free of charge, to any person obtaining + a copy of this software and associated documentation files (the + "Software"), to deal in the Software without restriction, including + without limitation the rights to use, copy, modify, merge, publish, + distribute, sublicense, and/or sell copies of the Software, and to + permit persons to whom the Software is furnished to do so, subject to + the following conditions: + + The above copyright notice and this permission notice shall be + included in all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE + LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION + OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + """ + +- flake8-use-pathlib, licensed as follows: + """ + MIT License + + Copyright (c) 2021 Rodolphe Pelloux-Prayer + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- RustPython, licensed as follows: + """ + MIT License + + Copyright (c) 2020 RustPython Team + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-annotations, licensed as follows: + """ + MIT License + + Copyright (c) 2019 - Present S. Co1 + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-async, licensed as follows: + """ + MIT License + + Copyright (c) 2022 Cooper Lees + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-type-checking, licensed as follows: + """ + Copyright (c) 2021, Sondre Lillebø Gundersen + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + * Neither the name of pytest-{{ cookiecutter.plugin_name }} nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE + FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + """ + +- flake8-bandit, licensed as follows: + """ + Copyright (c) 2017 Tyler Wince + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. + """ + +- flake8-eradicate, licensed as follows: + """ + MIT License + + Copyright (c) 2018 Nikita Sobolev + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-quotes, licensed as follows: + """ + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. + """ + +- flake8-logging-format, licensed as follows: + """ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + """ + +- flake8-raise, licensed as follows: + """ + MIT License + + Copyright (c) 2020 Jon Dufresne + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-self, licensed as follows: + """ + MIT License + + Copyright (c) 2023 Korijn van Golen + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-django, licensed under the GPL license. + +- perflint, licensed as follows: + """ + MIT License + + Copyright (c) 2022 Anthony Shaw + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-logging, licensed as follows: + """ + MIT License + + Copyright (c) 2023 Adam Johnson + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- flake8-trio, licensed as follows: + """ + MIT License + + Copyright (c) 2022 Zac Hatfield-Dodds + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- Pyright, licensed as follows: + """ + MIT License + + Pyright - A static type checker for the Python language + Copyright (c) Microsoft Corporation. All rights reserved. + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE + """ + +- rust-analyzer/text-size, licensed under the MIT license: + """ + Permission is hereby granted, free of charge, to any + person obtaining a copy of this software and associated + documentation files (the "Software"), to deal in the + Software without restriction, including without + limitation the rights to use, copy, modify, merge, + publish, distribute, sublicense, and/or sell copies of + the Software, and to permit persons to whom the Software + is furnished to do so, subject to the following + conditions: + + The above copyright notice and this permission notice + shall be included in all copies or substantial portions + of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF + ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED + TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A + PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT + SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY + CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION + OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR + IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER + DEALINGS IN THE SOFTWARE. + """ + +- rome/tools, licensed under the MIT license: + """ + MIT License + + Copyright (c) Rome Tools, Inc. and its affiliates. + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + +- pydoclint, licensed as follows: + """ + MIT License + + Copyright (c) 2023 jsh9 + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + """ + + +s3fs +BSD License +http://github.com/fsspec/s3fs/ +Copyright (c) 2016, Continuum Analytics, Inc. and contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + +Redistributions of source code must retain the above copyright notice, +this list of conditions and the following disclaimer. + +Redistributions in binary form must reproduce the above copyright notice, +this list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +Neither the name of Continuum Analytics nor the names of any contributors +may be used to endorse or promote products derived from this software +without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF +THE POSSIBILITY OF SUCH DAMAGE. + + +s3transfer +Apache Software License +https://github.com/boto/s3transfer + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +safehttpx +MIT License +https://github.com/gradio-app/safehttpx + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +safetensors +Apache Software License +https://github.com/huggingface/safetensors + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +scikit-image +BSD License +https://scikit-image.org +Files: * +Copyright: 2009-2022 the scikit-image team +License: BSD-3-Clause + +Files: doc/source/themes/scikit-image/layout.html +Copyright: 2007-2010 the Sphinx team +License: BSD-3-Clause + +Files: skimage/feature/_canny.py + skimage/filters/edges.py + skimage/filters/_rank_order.py + skimage/morphology/_skeletonize.py + skimage/morphology/tests/test_watershed.py + skimage/morphology/watershed.py + skimage/segmentation/heap_general.pxi + skimage/segmentation/heap_watershed.pxi + skimage/segmentation/_watershed.py + skimage/segmentation/_watershed_cy.pyx +Copyright: 2003-2009 Massachusetts Institute of Technology + 2009-2011 Broad Institute + 2003 Lee Kamentsky + 2003-2005 Peter J. Verveer +License: BSD-3-Clause + +Files: skimage/filters/thresholding.py + skimage/graph/_mcp.pyx + skimage/graph/heap.pyx +Copyright: 2009-2015 Board of Regents of the University of + Wisconsin-Madison, Broad Institute of MIT and Harvard, + and Max Planck Institute of Molecular Cell Biology and + Genetics + 2009 Zachary Pincus + 2009 Almar Klein +License: BSD-2-Clause + +File: skimage/morphology/grayreconstruct.py + skimage/morphology/tests/test_reconstruction.py +Copyright: 2003-2009 Massachusetts Institute of Technology + 2009-2011 Broad Institute + 2003 Lee Kamentsky +License: BSD-3-Clause + +File: skimage/morphology/_grayreconstruct.pyx +Copyright: 2003-2009 Massachusetts Institute of Technology + 2009-2011 Broad Institute + 2003 Lee Kamentsky + 2022 Gregory Lee (added a 64-bit integer variant for large images) +License: BSD-3-Clause + +File: skimage/segmentation/_expand_labels.py +Copyright: 2020 Broad Institute + 2020 CellProfiler team +License: BSD-3-Clause + +File: skimage/exposure/_adapthist.py +Copyright: 1994 Karel Zuiderveld +License: BSD-3-Clause + +Function: skimage/morphology/_skeletonize_various_cy.pyx:_skeletonize_loop +Copyright: 2003-2009 Massachusetts Institute of Technology + 2009-2011 Broad Institute + 2003 Lee Kamentsky +License: BSD-3-Clause + +Function: skimage/_shared/version_requirements.py:_check_version +Copyright: 2013 The IPython Development Team +License: BSD-3-Clause + +Function: skimage/_shared/version_requirements.py:is_installed +Copyright: 2009-2011 Pierre Raybaut +License: MIT + +File: skimage/feature/_fisher_vector.py +Copyright: 2014 2014 Dan Oneata +License: MIT + +File: skimage/_vendored/numpy_lookfor.py +Copyright: 2005-2023, NumPy Developers +License: BSD-3-Clause + +File: skimage/transform/_thin_plate_splines.py +Copyright: 2007 Zachary Pincus +License: BSD-3-Clause + +License: BSD-2-Clause + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. +. +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE HOLDERS OR +CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +License: BSD-3-Clause + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. +3. Neither the name of the University nor the names of its contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. +. +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE HOLDERS OR +CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +License: MIT + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +scipy +BSD License +https://scipy.org/ +Copyright (c) 2001-2002 Enthought, Inc. 2003, SciPy Developers. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided + with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +---- + +This binary distribution of SciPy can also bundle the following software +(depending on the build): + + +Name: OpenBLAS +Files: scipy.libs/libscipy_openblas*.so +Description: bundled as a dynamically linked library +Availability: https://github.com/OpenMathLib/OpenBLAS/ +License: BSD-3-Clause + Copyright (c) 2011-2014, The OpenBLAS Project + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are + met: + + 1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + 3. Neither the name of the OpenBLAS project nor the names of + its contributors may be used to endorse or promote products + derived from this software without specific prior written + permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE + USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +Name: LAPACK +Files: scipy.libs/libscipy_openblas*.so +Description: bundled in OpenBLAS +Availability: https://github.com/OpenMathLib/OpenBLAS/ +License: BSD-3-Clause-Open-MPI + Copyright (c) 1992-2013 The University of Tennessee and The University + of Tennessee Research Foundation. All rights + reserved. + Copyright (c) 2000-2013 The University of California Berkeley. All + rights reserved. + Copyright (c) 2006-2013 The University of Colorado Denver. All rights + reserved. + + $COPYRIGHT$ + + Additional copyrights may follow + + $HEADER$ + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are + met: + + - Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + - Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer listed + in this license in the documentation and/or other materials + provided with the distribution. + + - Neither the name of the copyright holders nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + + The copyright holders provide no reassurances that the source code + provided does not infringe any patent, copyright, or any other + intellectual property rights of third parties. The copyright holders + disclaim any liability to any recipient for claims brought against + recipient by any third party for infringement of that parties + intellectual property rights. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +Name: GCC runtime library +Files: scipy.libs/libgfortran*.so +Description: dynamically linked to files compiled with gcc +Availability: https://gcc.gnu.org/git/?p=gcc.git;a=tree;f=libgfortran +License: GPL-3.0-or-later WITH GCC-exception-3.1 + Copyright (C) 2002-2017 Free Software Foundation, Inc. + + Libgfortran is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 3, or (at your option) + any later version. + + Libgfortran is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + Under Section 7 of GPL version 3, you are granted additional + permissions described in the GCC Runtime Library Exception, version + 3.1, as published by the Free Software Foundation. + + You should have received a copy of the GNU General Public License and + a copy of the GCC Runtime Library Exception along with this program; + see the files COPYING3 and COPYING.RUNTIME respectively. If not, see + . + +---- + +Full text of license texts referred to above follows (that they are +listed below does not necessarily imply the conditions apply to the +present binary release): + +---- + +GCC RUNTIME LIBRARY EXCEPTION + +Version 3.1, 31 March 2009 + +Copyright (C) 2009 Free Software Foundation, Inc. + +Everyone is permitted to copy and distribute verbatim copies of this +license document, but changing it is not allowed. + +This GCC Runtime Library Exception ("Exception") is an additional +permission under section 7 of the GNU General Public License, version +3 ("GPLv3"). It applies to a given file (the "Runtime Library") that +bears a notice placed by the copyright holder of the file stating that +the file is governed by GPLv3 along with this Exception. + +When you use GCC to compile a program, GCC may combine portions of +certain GCC header files and runtime libraries with the compiled +program. The purpose of this Exception is to allow compilation of +non-GPL (including proprietary) programs to use, in this way, the +header files and runtime libraries covered by this Exception. + +0. Definitions. + +A file is an "Independent Module" if it either requires the Runtime +Library for execution after a Compilation Process, or makes use of an +interface provided by the Runtime Library, but is not otherwise based +on the Runtime Library. + +"GCC" means a version of the GNU Compiler Collection, with or without +modifications, governed by version 3 (or a specified later version) of +the GNU General Public License (GPL) with the option of using any +subsequent versions published by the FSF. + +"GPL-compatible Software" is software whose conditions of propagation, +modification and use would permit combination with GCC in accord with +the license of GCC. + +"Target Code" refers to output from any compiler for a real or virtual +target processor architecture, in executable form or suitable for +input to an assembler, loader, linker and/or execution +phase. Notwithstanding that, Target Code does not include data in any +format that is used as a compiler intermediate representation, or used +for producing a compiler intermediate representation. + +The "Compilation Process" transforms code entirely represented in +non-intermediate languages designed for human-written code, and/or in +Java Virtual Machine byte code, into Target Code. Thus, for example, +use of source code generators and preprocessors need not be considered +part of the Compilation Process, since the Compilation Process can be +understood as starting with the output of the generators or +preprocessors. + +A Compilation Process is "Eligible" if it is done using GCC, alone or +with other GPL-compatible software, or if it is done without using any +work based on GCC. For example, using non-GPL-compatible Software to +optimize any GCC intermediate representations would not qualify as an +Eligible Compilation Process. + +1. Grant of Additional Permission. + +You have permission to propagate a work of Target Code formed by +combining the Runtime Library with Independent Modules, even if such +propagation would otherwise violate the terms of GPLv3, provided that +all Target Code was generated by Eligible Compilation Processes. You +may then convey such a combination under terms of your choice, +consistent with the licensing of the Independent Modules. + +2. No Weakening of GCC Copyleft. + +The availability of this Exception does not imply any general +presumption that third-party software is unaffected by the copyleft +requirements of the license of GCC. + +---- + + GNU GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + Copyright (C) + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, your program's commands +might be different; for a GUI interface, you would use an "about box". + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU GPL, see +. + + The GNU General Public License does not permit incorporating your program +into proprietary programs. If your program is a subroutine library, you +may consider it more useful to permit linking proprietary applications with +the library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. But first, please read +. + + +Name: libquadmath +Files: scipy.libs/libquadmath*.so +Description: dynamically linked to files compiled with gcc +Availability: https://gcc.gnu.org/git/?p=gcc.git;a=tree;f=libquadmath +License: LGPL-2.1-or-later + + GCC Quad-Precision Math Library + Copyright (C) 2010-2019 Free Software Foundation, Inc. + Written by Francois-Xavier Coudert + + This file is part of the libquadmath library. + Libquadmath is free software; you can redistribute it and/or + modify it under the terms of the GNU Library General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + Libquadmath is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html + + +semantic-version +BSD License +https://github.com/rbarrois/python-semanticversion +Copyright (c) The python-semanticversion project +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +semver +BSD License +https://python-semver.readthedocs.io/en/latest/changelog.html +Copyright (c) 2013, Konstantine Rybnikov +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + + Redistributions in binary form must reproduce the above copyright notice, this + list of conditions and the following disclaimer in the documentation and/or + other materials provided with the distribution. + + Neither the name of the python-semver org nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +sentencepiece +UNKNOWN +https://github.com/google/sentencepiece +UNKNOWN + +sentry-sdk +BSD License +https://github.com/getsentry/sentry-python +MIT License + +Copyright (c) 2018 Functional Software, Inc. dba Sentry + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +shellingham +ISC License (ISCL) +https://github.com/sarugaku/shellingham +Copyright (c) 2018, Tzu-ping Chung + +Permission to use, copy, modify, and distribute this software for any +purpose with or without fee is hereby granted, provided that the above +copyright notice and this permission notice appear in all copies. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF +OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + + +six +MIT License +https://github.com/benjaminp/six +Copyright (c) 2010-2024 Benjamin Peterson + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +slangtorch +MIT License +https://github.com/shader-slang/slang +SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +LLVM Exceptions to the Apache 2.0 License + +As an exception, if, as a result of your compiling your source code, portions +of this Software are embedded into an Object form of such source code, you +may redistribute such embedded portions in such Object form without complying +with the conditions of Sections 4(a), 4(b) and 4(d) of the License. + +In addition, if you combine or link compiled forms of this Software with +software that is licensed under the GPLv2 ("Combined Software") and if a +court of competent jurisdiction determines that the patent provision (Section +3), the indemnity provision (Section 9) or other Section of the License +conflicts with the conditions of the GPLv2, you may retroactively and +prospectively choose to deem waived or otherwise exclude such Section(s) of +the License, but only in their entirety and only with respect to the Combined +Software. + + +smart_open +MIT License +https://github.com/piskvorky/smart_open +The MIT License (MIT) + +Copyright (c) 2015 Radim Řehůřek + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + + +smmap +BSD License +https://github.com/gitpython-developers/smmap +Copyright (C) 2010, 2011 Sebastian Thiel and contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +* Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +* Neither the name of the async project nor the names of +its contributors may be used to endorse or promote products derived +from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + + +sniffio +Apache Software License; MIT License +https://github.com/python-trio/sniffio +This software is made available under the terms of *either* of the +licenses found in LICENSE.APACHE2 or LICENSE.MIT. Contributions to are +made under the terms of *both* these licenses. + + +soundfile +BSD License +https://github.com/bastibe/python-soundfile + GNU LESSER GENERAL PUBLIC LICENSE + Version 2.1, February 1999 + + Copyright (C) 1991, 1999 Free Software Foundation, Inc. + 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + +[This is the first released version of the Lesser GPL. It also counts + as the successor of the GNU Library Public License, version 2, hence + the version number 2.1.] + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +Licenses are intended to guarantee your freedom to share and change +free software--to make sure the software is free for all its users. + + This license, the Lesser General Public License, applies to some +specially designated software packages--typically libraries--of the +Free Software Foundation and other authors who decide to use it. You +can use it too, but we suggest you first think carefully about whether +this license or the ordinary General Public License is the better +strategy to use in any particular case, based on the explanations below. + + When we speak of free software, we are referring to freedom of use, +not price. Our General Public Licenses are designed to make sure that +you have the freedom to distribute copies of free software (and charge +for this service if you wish); that you receive source code or can get +it if you want it; that you can change the software and use pieces of +it in new free programs; and that you are informed that you can do +these things. + + To protect your rights, we need to make restrictions that forbid +distributors to deny you these rights or to ask you to surrender these +rights. These restrictions translate to certain responsibilities for +you if you distribute copies of the library or if you modify it. + + For example, if you distribute copies of the library, whether gratis +or for a fee, you must give the recipients all the rights that we gave +you. You must make sure that they, too, receive or can get the source +code. If you link other code with the library, you must provide +complete object files to the recipients, so that they can relink them +with the library after making changes to the library and recompiling +it. And you must show them these terms so they know their rights. + + We protect your rights with a two-step method: (1) we copyright the +library, and (2) we offer you this license, which gives you legal +permission to copy, distribute and/or modify the library. + + To protect each distributor, we want to make it very clear that +there is no warranty for the free library. Also, if the library is +modified by someone else and passed on, the recipients should know +that what they have is not the original version, so that the original +author's reputation will not be affected by problems that might be +introduced by others. + + Finally, software patents pose a constant threat to the existence of +any free program. We wish to make sure that a company cannot +effectively restrict the users of a free program by obtaining a +restrictive license from a patent holder. Therefore, we insist that +any patent license obtained for a version of the library must be +consistent with the full freedom of use specified in this license. + + Most GNU software, including some libraries, is covered by the +ordinary GNU General Public License. This license, the GNU Lesser +General Public License, applies to certain designated libraries, and +is quite different from the ordinary General Public License. We use +this license for certain libraries in order to permit linking those +libraries into non-free programs. + + When a program is linked with a library, whether statically or using +a shared library, the combination of the two is legally speaking a +combined work, a derivative of the original library. The ordinary +General Public License therefore permits such linking only if the +entire combination fits its criteria of freedom. The Lesser General +Public License permits more lax criteria for linking other code with +the library. + + We call this license the "Lesser" General Public License because it +does Less to protect the user's freedom than the ordinary General +Public License. It also provides other free software developers Less +of an advantage over competing non-free programs. These disadvantages +are the reason we use the ordinary General Public License for many +libraries. However, the Lesser license provides advantages in certain +special circumstances. + + For example, on rare occasions, there may be a special need to +encourage the widest possible use of a certain library, so that it becomes +a de-facto standard. To achieve this, non-free programs must be +allowed to use the library. A more frequent case is that a free +library does the same job as widely used non-free libraries. In this +case, there is little to gain by limiting the free library to free +software only, so we use the Lesser General Public License. + + In other cases, permission to use a particular library in non-free +programs enables a greater number of people to use a large body of +free software. For example, permission to use the GNU C Library in +non-free programs enables many more people to use the whole GNU +operating system, as well as its variant, the GNU/Linux operating +system. + + Although the Lesser General Public License is Less protective of the +users' freedom, it does ensure that the user of a program that is +linked with the Library has the freedom and the wherewithal to run +that program using a modified version of the Library. + + The precise terms and conditions for copying, distribution and +modification follow. Pay close attention to the difference between a +"work based on the library" and a "work that uses the library". The +former contains code derived from the library, whereas the latter must +be combined with the library in order to run. + + GNU LESSER GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License Agreement applies to any software library or other +program which contains a notice placed by the copyright holder or +other authorized party saying it may be distributed under the terms of +this Lesser General Public License (also called "this License"). +Each licensee is addressed as "you". + + A "library" means a collection of software functions and/or data +prepared so as to be conveniently linked with application programs +(which use some of those functions and data) to form executables. + + The "Library", below, refers to any such software library or work +which has been distributed under these terms. A "work based on the +Library" means either the Library or any derivative work under +copyright law: that is to say, a work containing the Library or a +portion of it, either verbatim or with modifications and/or translated +straightforwardly into another language. (Hereinafter, translation is +included without limitation in the term "modification".) + + "Source code" for a work means the preferred form of the work for +making modifications to it. For a library, complete source code means +all the source code for all modules it contains, plus any associated +interface definition files, plus the scripts used to control compilation +and installation of the library. + + Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running a program using the Library is not restricted, and output from +such a program is covered only if its contents constitute a work based +on the Library (independent of the use of the Library in a tool for +writing it). Whether that is true depends on what the Library does +and what the program that uses the Library does. + + 1. You may copy and distribute verbatim copies of the Library's +complete source code as you receive it, in any medium, provided that +you conspicuously and appropriately publish on each copy an +appropriate copyright notice and disclaimer of warranty; keep intact +all the notices that refer to this License and to the absence of any +warranty; and distribute a copy of this License along with the +Library. + + You may charge a fee for the physical act of transferring a copy, +and you may at your option offer warranty protection in exchange for a +fee. + + 2. You may modify your copy or copies of the Library or any portion +of it, thus forming a work based on the Library, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) The modified work must itself be a software library. + + b) You must cause the files modified to carry prominent notices + stating that you changed the files and the date of any change. + + c) You must cause the whole of the work to be licensed at no + charge to all third parties under the terms of this License. + + d) If a facility in the modified Library refers to a function or a + table of data to be supplied by an application program that uses + the facility, other than as an argument passed when the facility + is invoked, then you must make a good faith effort to ensure that, + in the event an application does not supply such function or + table, the facility still operates, and performs whatever part of + its purpose remains meaningful. + + (For example, a function in a library to compute square roots has + a purpose that is entirely well-defined independent of the + application. Therefore, Subsection 2d requires that any + application-supplied function or table used by this function must + be optional: if the application does not supply it, the square + root function must still compute square roots.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Library, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Library, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote +it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Library. + +In addition, mere aggregation of another work not based on the Library +with the Library (or with a work based on the Library) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may opt to apply the terms of the ordinary GNU General Public +License instead of this License to a given copy of the Library. To do +this, you must alter all the notices that refer to this License, so +that they refer to the ordinary GNU General Public License, version 2, +instead of to this License. (If a newer version than version 2 of the +ordinary GNU General Public License has appeared, then you can specify +that version instead if you wish.) Do not make any other change in +these notices. + + Once this change is made in a given copy, it is irreversible for +that copy, so the ordinary GNU General Public License applies to all +subsequent copies and derivative works made from that copy. + + This option is useful when you wish to copy part of the code of +the Library into a program that is not a library. + + 4. You may copy and distribute the Library (or a portion or +derivative of it, under Section 2) in object code or executable form +under the terms of Sections 1 and 2 above provided that you accompany +it with the complete corresponding machine-readable source code, which +must be distributed under the terms of Sections 1 and 2 above on a +medium customarily used for software interchange. + + If distribution of object code is made by offering access to copy +from a designated place, then offering equivalent access to copy the +source code from the same place satisfies the requirement to +distribute the source code, even though third parties are not +compelled to copy the source along with the object code. + + 5. A program that contains no derivative of any portion of the +Library, but is designed to work with the Library by being compiled or +linked with it, is called a "work that uses the Library". Such a +work, in isolation, is not a derivative work of the Library, and +therefore falls outside the scope of this License. + + However, linking a "work that uses the Library" with the Library +creates an executable that is a derivative of the Library (because it +contains portions of the Library), rather than a "work that uses the +library". The executable is therefore covered by this License. +Section 6 states terms for distribution of such executables. + + When a "work that uses the Library" uses material from a header file +that is part of the Library, the object code for the work may be a +derivative work of the Library even though the source code is not. +Whether this is true is especially significant if the work can be +linked without the Library, or if the work is itself a library. The +threshold for this to be true is not precisely defined by law. + + If such an object file uses only numerical parameters, data +structure layouts and accessors, and small macros and small inline +functions (ten lines or less in length), then the use of the object +file is unrestricted, regardless of whether it is legally a derivative +work. (Executables containing this object code plus portions of the +Library will still fall under Section 6.) + + Otherwise, if the work is a derivative of the Library, you may +distribute the object code for the work under the terms of Section 6. +Any executables containing that work also fall under Section 6, +whether or not they are linked directly with the Library itself. + + 6. As an exception to the Sections above, you may also combine or +link a "work that uses the Library" with the Library to produce a +work containing portions of the Library, and distribute that work +under terms of your choice, provided that the terms permit +modification of the work for the customer's own use and reverse +engineering for debugging such modifications. + + You must give prominent notice with each copy of the work that the +Library is used in it and that the Library and its use are covered by +this License. You must supply a copy of this License. If the work +during execution displays copyright notices, you must include the +copyright notice for the Library among them, as well as a reference +directing the user to the copy of this License. Also, you must do one +of these things: + + a) Accompany the work with the complete corresponding + machine-readable source code for the Library including whatever + changes were used in the work (which must be distributed under + Sections 1 and 2 above); and, if the work is an executable linked + with the Library, with the complete machine-readable "work that + uses the Library", as object code and/or source code, so that the + user can modify the Library and then relink to produce a modified + executable containing the modified Library. (It is understood + that the user who changes the contents of definitions files in the + Library will not necessarily be able to recompile the application + to use the modified definitions.) + + b) Use a suitable shared library mechanism for linking with the + Library. A suitable mechanism is one that (1) uses at run time a + copy of the library already present on the user's computer system, + rather than copying library functions into the executable, and (2) + will operate properly with a modified version of the library, if + the user installs one, as long as the modified version is + interface-compatible with the version that the work was made with. + + c) Accompany the work with a written offer, valid for at + least three years, to give the same user the materials + specified in Subsection 6a, above, for a charge no more + than the cost of performing this distribution. + + d) If distribution of the work is made by offering access to copy + from a designated place, offer equivalent access to copy the above + specified materials from the same place. + + e) Verify that the user has already received a copy of these + materials or that you have already sent this user a copy. + + For an executable, the required form of the "work that uses the +Library" must include any data and utility programs needed for +reproducing the executable from it. However, as a special exception, +the materials to be distributed need not include anything that is +normally distributed (in either source or binary form) with the major +components (compiler, kernel, and so on) of the operating system on +which the executable runs, unless that component itself accompanies +the executable. + + It may happen that this requirement contradicts the license +restrictions of other proprietary libraries that do not normally +accompany the operating system. Such a contradiction means you cannot +use both them and the Library together in an executable that you +distribute. + + 7. You may place library facilities that are a work based on the +Library side-by-side in a single library together with other library +facilities not covered by this License, and distribute such a combined +library, provided that the separate distribution of the work based on +the Library and of the other library facilities is otherwise +permitted, and provided that you do these two things: + + a) Accompany the combined library with a copy of the same work + based on the Library, uncombined with any other library + facilities. This must be distributed under the terms of the + Sections above. + + b) Give prominent notice with the combined library of the fact + that part of it is a work based on the Library, and explaining + where to find the accompanying uncombined form of the same work. + + 8. You may not copy, modify, sublicense, link with, or distribute +the Library except as expressly provided under this License. Any +attempt otherwise to copy, modify, sublicense, link with, or +distribute the Library is void, and will automatically terminate your +rights under this License. However, parties who have received copies, +or rights, from you under this License will not have their licenses +terminated so long as such parties remain in full compliance. + + 9. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Library or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Library (or any work based on the +Library), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Library or works based on it. + + 10. Each time you redistribute the Library (or any work based on the +Library), the recipient automatically receives a license from the +original licensor to copy, distribute, link with or modify the Library +subject to these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties with +this License. + + 11. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Library at all. For example, if a patent +license would not permit royalty-free redistribution of the Library by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Library. + +If any portion of this section is held invalid or unenforceable under any +particular circumstance, the balance of the section is intended to apply, +and the section as a whole is intended to apply in other circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 12. If the distribution and/or use of the Library is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Library under this License may add +an explicit geographical distribution limitation excluding those countries, +so that distribution is permitted only in or among countries not thus +excluded. In such case, this License incorporates the limitation as if +written in the body of this License. + + 13. The Free Software Foundation may publish revised and/or new +versions of the Lesser General Public License from time to time. +Such new versions will be similar in spirit to the present version, +but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Library +specifies a version number of this License which applies to it and +"any later version", you have the option of following the terms and +conditions either of that version or of any later version published by +the Free Software Foundation. If the Library does not specify a +license version number, you may choose any version ever published by +the Free Software Foundation. + + 14. If you wish to incorporate parts of the Library into other free +programs whose distribution conditions are incompatible with these, +write to the author to ask for permission. For software which is +copyrighted by the Free Software Foundation, write to the Free +Software Foundation; we sometimes make exceptions for this. Our +decision will be guided by the two goals of preserving the free status +of all derivatives of our free software and of promoting the sharing +and reuse of software generally. + + NO WARRANTY + + 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO +WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. +EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR +OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY +KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE +LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME +THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN +WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY +AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU +FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR +CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE +LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING +RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A +FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF +SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH +DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Libraries + + If you develop a new library, and you want it to be of the greatest +possible use to the public, we recommend making it free software that +everyone can redistribute and change. You can do so by permitting +redistribution under these terms (or, alternatively, under the terms of the +ordinary General Public License). + + To apply these terms, attach the following notices to the library. It is +safest to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least the +"copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +Also add information on how to contact you by electronic and paper mail. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the library, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the + library `Frob' (a library for tweaking knobs) written by James Random Hacker. + + , 1 April 1990 + Ty Coon, President of Vice + +That's all there is to it! + + + +soupsieve +MIT +https://github.com/facelessuser/soupsieve +MIT License + +Copyright (c) 2018 - 2026 Isaac Muse + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +stack-data +MIT License +http://github.com/alexmojaki/stack_data +MIT License + +Copyright (c) 2019 Alex Hall + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +starlette +BSD-3-Clause +https://github.com/Kludex/starlette +Copyright © 2018, [Encode OSS Ltd](https://www.encode.io/). +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +sympy +BSD License +https://sympy.org +Copyright (c) 2006-2023 SymPy Development Team + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + a. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + b. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + c. Neither the name of SymPy nor the names of its contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. + + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH +DAMAGE. + +-------------------------------------------------------------------------------- + +Patches that were taken from the Diofant project (https://github.com/diofant/diofant) +are licensed as: + +Copyright (c) 2006-2018 SymPy Development Team, + 2013-2023 Sergey B Kirpichev + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + a. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + b. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + c. Neither the name of Diofant or SymPy nor the names of its contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. + + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH +DAMAGE. + +-------------------------------------------------------------------------------- + +Submodules taken from the multipledispatch project (https://github.com/mrocklin/multipledispatch) +are licensed as: + +Copyright (c) 2014 Matthew Rocklin + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + a. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + b. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + c. Neither the name of multipledispatch nor the names of its contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. + + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH +DAMAGE. + +-------------------------------------------------------------------------------- + +The files under the directory sympy/parsing/autolev/tests/pydy-example-repo +are directly copied from PyDy project and are licensed as: + +Copyright (c) 2009-2023, PyDy Authors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. +* Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. +* Neither the name of this project nor the names of its contributors may be + used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL PYDY AUTHORS BE LIABLE FOR ANY DIRECT, +INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, +BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE +OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF +ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +-------------------------------------------------------------------------------- + +The files under the directory sympy/parsing/latex +are directly copied from latex2sympy project and are licensed as: + +Copyright 2016, latex2sympy + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +tabulate +MIT +https://github.com/astanin/python-tabulate +Copyright (c) 2011-2020 Sergey Astanin and contributors + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +tensorboard +Apache Software License +https://github.com/tensorflow/tensorboard +# TensorBoard License + +TensorBoard is licensed Apache 2.0 and distributed with +vendored content licensed Apache 2.0, MIT, and BSD-3. + +## Table of Contents + +- tensorboard/pip_package/LICENSE.tensorflow +- external/npm/node_modules/d3/LICENSE +- external/com_google_fonts_roboto/LICENSE +- external/org_mozilla_bleach/LICENSE +- external/org_pythonhosted_webencodings/LICENSE +- third_party/bh_tsne.LICENSE + +## Licenses + + + +### tensorboard/pip_package/LICENSE.tensorflow + +Copyright 2017 The TensorFlow Authors. All rights reserved. + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2017, The TensorFlow Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +### external/npm/node_modules/d3/LICENSE + +Copyright 2010-2017 Mike Bostock +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the author nor the names of contributors may be used to + endorse or promote products derived from this software without specific prior + written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + + +### external/com_google_fonts_roboto/LICENSE + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +### external/org_mozilla_bleach/LICENSE + +Copyright (c) 2014-2017, Mozilla Foundation + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + + + +### external/org_pythonhosted_webencodings/LICENSE + +Copyright (c) 2012 by Simon Sapin. + +Some rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided + with the distribution. + + * The names of the contributors may not be used to endorse or + promote products derived from this software without specific + prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + + +### third_party/bh_tsne.LICENSE + +The MIT License (MIT) + +Copyright (c) 2015 Andrej Karpathy + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +tensorboard-data-server +Apache Software License +https://github.com/tensorflow/tensorboard/tree/master/tensorboard/data/server +UNKNOWN + +tensorstore +Apache-2.0 +https://github.com/google/tensorstore +Files: **/* + +Copyright 2018 The TensorStore Authors. All rights reserved. + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2017, The TensorStore Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +-------------------------------------------------------------------------------- + +Files: internal/utf8.cc + +Copyright (c) 2008-2009 Bjoern Hoehrmann + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +-------------------------------------------------------------------------------- + +Files: third_party/snappy/bundled.BUILD.bazel + +Copyright 2011 Google Inc. All Rights Reserved. +Author: sesse@google.com (Steinar H. Gunderson) + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +-------------------------------------------------------------------------------- + +Files: tools/cmake/FindAVIF.cmake + +Copyright (C) 2021 Igalia S.L. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS +BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF +THE POSSIBILITY OF SUCH DAMAGE. + + + +termcolor +MIT +https://github.com/termcolor/termcolor +Copyright (c) 2008-2011 Volvox Development Team + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +terminado +BSD License +https://github.com/jupyter/terminado +BSD 2-Clause License + +- Copyright (c) 2014-, Jupyter development team +- Copyright (c) 2014, Ramalingam Saravanan + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +tifffile +BSD-3-Clause +https://www.cgohlke.com +BSD-3-Clause license + +Copyright (c) 2008-2026, Christoph Gohlke +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + + +tiktoken +MIT License + +Copyright (c) 2022 OpenAI, Shantanu Jain + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +https://github.com/openai/tiktoken +MIT License + +Copyright (c) 2022 OpenAI, Shantanu Jain + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +timm +Apache Software License +https://github.com/huggingface/pytorch-image-models + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2019 Ross Wightman + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +tinycss2 +BSD License +https://www.courtbouillon.org/tinycss2 +BSD 3-Clause License + +Copyright (c) 2013-2020, Simon Sapin and contributors. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +tokenizers +Apache Software License +https://github.com/huggingface/tokenizers +UNKNOWN + +tomli +MIT +https://github.com/hukkin/tomli +MIT License + +Copyright (c) 2021 Taneli Hukkinen + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +tomli_w +MIT License +https://github.com/hukkin/tomli-w +MIT License + +Copyright (c) 2021 Taneli Hukkinen + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +tomlkit +MIT License +https://github.com/sdispater/tomlkit +Copyright (c) 2018 Sébastien Eustace + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +torch +BSD-3-Clause +https://pytorch.org +From PyTorch: + +Copyright (c) 2016- Facebook, Inc (Adam Paszke) +Copyright (c) 2014- Facebook, Inc (Soumith Chintala) +Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert) +Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu) +Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu) +Copyright (c) 2011-2013 NYU (Clement Farabet) +Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston) +Copyright (c) 2006 Idiap Research Institute (Samy Bengio) +Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz) + +From Caffe2: + +Copyright (c) 2016-present, Facebook Inc. All rights reserved. + +All contributions by Facebook: +Copyright (c) 2016 Facebook Inc. + +All contributions by Google: +Copyright (c) 2015 Google Inc. +All rights reserved. + +All contributions by Yangqing Jia: +Copyright (c) 2015 Yangqing Jia +All rights reserved. + +All contributions by Kakao Brain: +Copyright 2019-2020 Kakao Brain + +All contributions by Cruise LLC: +Copyright (c) 2022 Cruise LLC. +All rights reserved. + +All contributions by Tri Dao: +Copyright (c) 2024 Tri Dao. +All rights reserved. + +All contributions by Arm: +Copyright (c) 2021, 2023-2025 Arm Limited and/or its affiliates + +All contributions from Caffe: +Copyright(c) 2013, 2014, 2015, the respective contributors +All rights reserved. + +All other contributions: +Copyright(c) 2015, 2016 the respective contributors +All rights reserved. + +Caffe2 uses a copyright model similar to Caffe: each contributor holds +copyright over their contributions to Caffe2. The project versioning records +all such contribution and copyright details. If a contributor wants to further +mark their specific copyright on a particular contribution, they should +indicate their copyright solely in the commit message of the change when it is +committed. + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +3. Neither the names of Facebook, Deepmind Technologies, NYU, NEC Laboratories America + and IDIAP Research Institute nor the names of its contributors may be + used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + + +The PyTorch repository and source distributions bundle several libraries that are +compatibly licensed. We list these here. + +Name: DCGM +License: Apache-2.0 +Files: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM + For details, see the files concatenated below: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM/LICENSE + +Name: FP16 +License: MIT +Files: /pytorch/third_party/FP16 + For details, see the files concatenated below: /pytorch/third_party/FP16/LICENSE + +Name: FXdiv +License: MIT +Files: /pytorch/third_party/FXdiv + For details, see the files concatenated below: /pytorch/third_party/FXdiv/LICENSE + +Name: NNPACK +License: BSD-2-Clause +Files: /pytorch/third_party/NNPACK + For details, see the files concatenated below: /pytorch/third_party/NNPACK/LICENSE + +Name: NVTX +License: Apache-2.0 with exception +Files: /pytorch/third_party/NVTX + For details, see the files concatenated below: /pytorch/third_party/NVTX/LICENSE.txt + +Name: VulkanMemoryAllocator +License: MIT +Files: /pytorch/third_party/VulkanMemoryAllocator + For details, see the files concatenated below: /pytorch/third_party/VulkanMemoryAllocator/LICENSE.txt + +Name: XNNPACK +License: BSD-3-Clause +Files: /pytorch/third_party/XNNPACK + For details, see the files concatenated below: /pytorch/third_party/XNNPACK/LICENSE + +Name: aiter +License: MIT +Files: /pytorch/third_party/aiter + For details, see the files concatenated below: /pytorch/third_party/aiter/LICENSE + +Name: benchmark +License: Apache-2.0 +Files: /pytorch/third_party/benchmark, + /pytorch/third_party/opentelemetry-cpp/third_party/benchmark, + /pytorch/third_party/protobuf/third_party/benchmark + For details, see the files concatenated below: /pytorch/third_party/benchmark/LICENSE, + /pytorch/third_party/opentelemetry-cpp/third_party/benchmark/LICENSE, + /pytorch/third_party/protobuf/third_party/benchmark/LICENSE + +Name: boost-vcpkg-helpers +License: MIT +Files: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/boost-vcpkg-helpers + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/boost-vcpkg-helpers/LICENSE.txt + +Name: cJSON +License: MIT +Files: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb/examples/rest/cJSON, + /pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb/examples/rest/cJSON + For details, see the files concatenated below: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb/examples/rest/cJSON/LICENSE, + /pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb/examples/rest/cJSON/LICENSE + +Name: catch2 +License: BSL-1.0 +Files: /pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp/3rd_party/include/opentracing/catch2 + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp/3rd_party/include/opentracing/catch2/LICENSE.txt + +Name: clog +License: BSD-2-Clause +Files: /pytorch/third_party/cpuinfo/deps/clog, + /pytorch/third_party/fbgemm/external/cpuinfo/deps/clog + For details, see the files concatenated below: /pytorch/third_party/cpuinfo/deps/clog/LICENSE, + /pytorch/third_party/fbgemm/external/cpuinfo/deps/clog/LICENSE + +Name: colorama +License: BSD-3-Clause +Files: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM/testing/python3/libs_3rdparty/colorama + For details, see the files concatenated below: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM/testing/python3/libs_3rdparty/colorama/LICENSE.txt + +Name: composable_kernel +License: MIT +Files: /pytorch/third_party/aiter/3rdparty/composable_kernel, + /pytorch/third_party/composable_kernel, + /pytorch/third_party/fbgemm/external/composable_kernel, + /pytorch/third_party/flash-attention/csrc/composable_kernel + For details, see the files concatenated below: /pytorch/third_party/aiter/3rdparty/composable_kernel/LICENSE, + /pytorch/third_party/composable_kernel/LICENSE, + /pytorch/third_party/fbgemm/external/composable_kernel/LICENSE, + /pytorch/third_party/flash-attention/csrc/composable_kernel/LICENSE + +Name: cpp-httplib +License: MIT +Files: /pytorch/third_party/cpp-httplib + For details, see the files concatenated below: /pytorch/third_party/cpp-httplib/LICENSE + +Name: cpplint +License: BSD-3-Clause +Files: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json/third_party/cpplint + For details, see the files concatenated below: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json/third_party/cpplint/LICENSE + +Name: cpr +License: MIT +Files: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr + For details, see the files concatenated below: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr/LICENSE + +Name: cpuinfo +License: BSD-2-Clause +Files: /pytorch/third_party/cpuinfo, + /pytorch/third_party/fbgemm/external/cpuinfo + For details, see the files concatenated below: /pytorch/third_party/cpuinfo/LICENSE, + /pytorch/third_party/fbgemm/external/cpuinfo/LICENSE + +Name: cudnn_frontend +License: MIT +Files: /pytorch/third_party/cudnn_frontend + For details, see the files concatenated below: /pytorch/third_party/cudnn_frontend/LICENSE.txt + +Name: cutlass +License: BSD-3-Clause +Files: /pytorch/third_party/cutlass, + /pytorch/third_party/fbgemm/external/cutlass, + /pytorch/third_party/flash-attention/csrc/cutlass + For details, see the files concatenated below: /pytorch/third_party/cutlass/LICENSE.txt, + /pytorch/third_party/fbgemm/external/cutlass/LICENSE.txt, + /pytorch/third_party/flash-attention/csrc/cutlass/LICENSE.txt + +Name: dart +License: Apache-2.0 +Files: /pytorch/third_party/flatbuffers/dart + For details, see the files concatenated below: /pytorch/third_party/flatbuffers/dart/LICENSE + +Name: docs +License: Apache-2.0 with exception +Files: /pytorch/third_party/NVTX/docs + For details, see the files concatenated below: /pytorch/third_party/NVTX/docs/LICENSE.txt + +Name: doctest +License: MIT +Files: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json/test/thirdparty/doctest + For details, see the files concatenated below: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json/test/thirdparty/doctest/LICENSE.txt + +Name: duktape-1.5.2 +License: MIT +Files: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb/src/third_party/duktape-1.5.2, + /pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb/src/third_party/duktape-1.5.2 + For details, see the files concatenated below: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb/src/third_party/duktape-1.5.2/LICENSE.txt, + /pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb/src/third_party/duktape-1.5.2/LICENSE.txt + +Name: duktape-1.8.0 +License: MIT +Files: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb/src/third_party/duktape-1.8.0, + /pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb/src/third_party/duktape-1.8.0 + For details, see the files concatenated below: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb/src/third_party/duktape-1.8.0/LICENSE.txt, + /pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb/src/third_party/duktape-1.8.0/LICENSE.txt + +Name: dynolog +License: MIT +Files: /pytorch/third_party/kineto/libkineto/third_party/dynolog + For details, see the files concatenated below: /pytorch/third_party/kineto/libkineto/third_party/dynolog/LICENSE + +Name: etw +License: MIT +Files: /pytorch/third_party/opentelemetry-cpp/exporters/etw/include/opentelemetry/exporters/etw + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/exporters/etw/include/opentelemetry/exporters/etw/LICENSE + +Name: expected +License: MIT +Files: /pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp/3rd_party/include/opentracing/expected + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp/3rd_party/include/opentracing/expected/LICENSE + +Name: fbgemm +License: BSD-3-Clause +Files: /pytorch/third_party/fbgemm + For details, see the files concatenated below: /pytorch/third_party/fbgemm/LICENSE + +Name: ffnvcodec +License: MIT with exception +Files: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/ffnvcodec + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/ffnvcodec/LICENSE.txt + +Name: flash-attention +License: BSD-3-Clause +Files: /pytorch/third_party/flash-attention + For details, see the files concatenated below: /pytorch/third_party/flash-attention/LICENSE + +Name: flatbuffers +License: Apache-2.0 +Files: /pytorch/third_party/flatbuffers + For details, see the files concatenated below: /pytorch/third_party/flatbuffers/LICENSE + +Name: fmt +License: MIT with exception +Files: /pytorch/third_party/fmt, + /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt, + /pytorch/third_party/kineto/libkineto/third_party/fmt + For details, see the files concatenated below: /pytorch/third_party/fmt/LICENSE, + /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt/LICENSE.rst, + /pytorch/third_party/kineto/libkineto/third_party/fmt/LICENSE + +Name: gemmlowp +License: Apache-2.0 +Files: /pytorch/third_party/gemmlowp/gemmlowp + For details, see the files concatenated below: /pytorch/third_party/gemmlowp/gemmlowp/LICENSE + +Name: generator +License: Apache-2.0 +Files: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest/googlemock/scripts/generator, + /pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest/googlemock/scripts/generator, + /pytorch/third_party/protobuf/third_party/googletest/googlemock/scripts/generator, + /pytorch/third_party/tensorpipe/third_party/googletest/googlemock/scripts/generator + For details, see the files concatenated below: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest/googlemock/scripts/generator/LICENSE, + /pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest/googlemock/scripts/generator/LICENSE, + /pytorch/third_party/protobuf/third_party/googletest/googlemock/scripts/generator/LICENSE, + /pytorch/third_party/tensorpipe/third_party/googletest/googlemock/scripts/generator/LICENSE + +Name: gettimeofday +License: Apache-2.0 +Files: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/gettimeofday + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/gettimeofday/LICENSE + +Name: gloo +License: BSD-3-Clause +Files: /pytorch/third_party/gloo + For details, see the files concatenated below: /pytorch/third_party/gloo/LICENSE + +Name: googlemock +License: BSD-3-Clause +Files: /pytorch/third_party/protobuf/third_party/googletest/googlemock, + /pytorch/third_party/tensorpipe/third_party/googletest/googlemock + For details, see the files concatenated below: /pytorch/third_party/protobuf/third_party/googletest/googlemock/LICENSE, + /pytorch/third_party/tensorpipe/third_party/googletest/googlemock/LICENSE + +Name: googletest +License: BSD-3-Clause +Files: /pytorch/third_party/fbgemm/external/googletest, + /pytorch/third_party/googletest, + /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest, + /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest, + /pytorch/third_party/kineto/libkineto/third_party/googletest, + /pytorch/third_party/opentelemetry-cpp/third_party/googletest, + /pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest, + /pytorch/third_party/protobuf/third_party/googletest, + /pytorch/third_party/protobuf/third_party/googletest/googletest, + /pytorch/third_party/tensorpipe/third_party/googletest, + /pytorch/third_party/tensorpipe/third_party/googletest/googletest + For details, see the files concatenated below: /pytorch/third_party/fbgemm/external/googletest/LICENSE, + /pytorch/third_party/googletest/LICENSE, + /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest/LICENSE, + /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest/LICENSE, + /pytorch/third_party/kineto/libkineto/third_party/googletest/LICENSE, + /pytorch/third_party/opentelemetry-cpp/third_party/googletest/LICENSE, + /pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest/LICENSE, + /pytorch/third_party/protobuf/third_party/googletest/LICENSE, + /pytorch/third_party/protobuf/third_party/googletest/googletest/LICENSE, + /pytorch/third_party/tensorpipe/third_party/googletest/LICENSE, + /pytorch/third_party/tensorpipe/third_party/googletest/googletest/LICENSE + +Name: gtest +License: BSD-3-Clause +Files: /pytorch/third_party/ideep/mkl-dnn/tests/gtests/gtest + For details, see the files concatenated below: /pytorch/third_party/ideep/mkl-dnn/tests/gtests/gtest/LICENSE + +Name: hipify_torch +License: MIT +Files: /pytorch/third_party/fbgemm/external/hipify_torch + For details, see the files concatenated below: /pytorch/third_party/fbgemm/external/hipify_torch/LICENSE.txt + +Name: hstu +License: BSD-3-Clause +Files: /pytorch/third_party/fbgemm/fbgemm_gpu/experimental/hstu + For details, see the files concatenated below: /pytorch/third_party/fbgemm/fbgemm_gpu/experimental/hstu/LICENSE + +Name: hungarian +License: Permissive (free to use) +Files: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/hungarian + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/hungarian/LICENSE.txt + +Name: ideep +License: MIT +Files: /pytorch/third_party/ideep + For details, see the files concatenated below: /pytorch/third_party/ideep/LICENSE + +Name: irrlicht +License: MIT +Files: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/irrlicht + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/irrlicht/LICENSE.txt + +Name: kineto +License: BSD-3-Clause +Files: /pytorch/third_party/kineto + For details, see the files concatenated below: /pytorch/third_party/kineto/LICENSE + +Name: libnop +License: Apache-2.0 +Files: /pytorch/third_party/tensorpipe/third_party/libnop + For details, see the files concatenated below: /pytorch/third_party/tensorpipe/third_party/libnop/LICENSE + +Name: libstemmer +License: BSD-3-Clause +Files: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/libstemmer + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/libstemmer/LICENSE + +Name: libuv +License: MIT +Files: /pytorch/third_party/tensorpipe/third_party/libuv + For details, see the files concatenated below: /pytorch/third_party/tensorpipe/third_party/libuv/LICENSE + +Name: mimalloc +License: MIT +Files: /pytorch/third_party/mimalloc + For details, see the files concatenated below: /pytorch/third_party/mimalloc/LICENSE + +Name: miniz-3.0.2 +License: MIT +Files: /pytorch/third_party/miniz-3.0.2 + For details, see the files concatenated below: /pytorch/third_party/miniz-3.0.2/LICENSE + +Name: mkl-dnn +License: Apache-2.0 +Files: /pytorch/third_party/ideep/mkl-dnn + For details, see the files concatenated below: /pytorch/third_party/ideep/mkl-dnn/LICENSE + +Name: ms-gsl +License: MIT +Files: /pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl/LICENSE + +Name: mx +License: MIT +Files: /pytorch/third_party/fbgemm/fbgemm_gpu/src/quantize_ops/mx, + /pytorch/third_party/fbgemm/fbgemm_gpu/test/quantize/mx + For details, see the files concatenated below: /pytorch/third_party/fbgemm/fbgemm_gpu/src/quantize_ops/mx/LICENSE, + /pytorch/third_party/fbgemm/fbgemm_gpu/test/quantize/mx/LICENSE + +Name: onnx +License: Apache-2.0 +Files: /pytorch/third_party/onnx + For details, see the files concatenated below: /pytorch/third_party/onnx/LICENSE + +Name: opentelemetry-cpp +License: Apache-2.0 +Files: /pytorch/third_party/opentelemetry-cpp + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/LICENSE + +Name: opentelemetry-proto +License: Apache-2.0 +Files: /pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto/LICENSE + +Name: opentracing-cpp +License: Apache-2.0 +Files: /pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp/LICENSE + +Name: pdcurses +License: Public Domain for core +Files: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/pdcurses + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/pdcurses/LICENSE + +Name: pfs +License: Apache-2.0 +Files: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs + For details, see the files concatenated below: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs/LICENSE + +Name: physac +License: MIT +Files: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/physac + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/physac/LICENSE + +Name: pqp +License: Apache-2.0 +Files: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/pqp + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/pqp/LICENSE + +Name: prometheus-cpp +License: MIT +Files: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp, + /pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp + For details, see the files concatenated below: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/LICENSE, + /pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/LICENSE + +Name: protobuf +License: BSD-3-Clause +Files: /pytorch/third_party/protobuf + For details, see the files concatenated below: /pytorch/third_party/protobuf/LICENSE + +Name: psimd +License: MIT +Files: /pytorch/third_party/psimd + For details, see the files concatenated below: /pytorch/third_party/psimd/LICENSE + +Name: pthreadpool +License: BSD-2-Clause +Files: /pytorch/third_party/pthreadpool + For details, see the files concatenated below: /pytorch/third_party/pthreadpool/LICENSE + +Name: pybind11 +License: BSD-3-Clause +Files: /pytorch/third_party/onnx/third_party/pybind11, + /pytorch/third_party/pybind11, + /pytorch/third_party/tensorpipe/third_party/pybind11 + For details, see the files concatenated below: /pytorch/third_party/onnx/third_party/pybind11/LICENSE, + /pytorch/third_party/pybind11/LICENSE, + /pytorch/third_party/tensorpipe/third_party/pybind11/LICENSE + +Name: python +License: Apache-2.0 with exception +Files: /pytorch/third_party/NVTX/python + For details, see the files concatenated below: /pytorch/third_party/NVTX/python/LICENSE.txt + +Name: python +License: BSD-3-Clause +Files: /pytorch/third_party/cutlass/python + For details, see the files concatenated below: /pytorch/third_party/cutlass/python/LICENSE.txt + +Name: python +License: BSD-3-Clause +Files: /pytorch/third_party/fbgemm/external/cutlass/python + For details, see the files concatenated below: /pytorch/third_party/fbgemm/external/cutlass/python/LICENSE.txt + +Name: python +License: BSD-3-Clause +Files: /pytorch/third_party/flash-attention/csrc/cutlass/python + For details, see the files concatenated below: /pytorch/third_party/flash-attention/csrc/cutlass/python/LICENSE.txt + +Name: python-peachpy +License: BSD-2-Clause +Files: /pytorch/third_party/python-peachpy + For details, see the files concatenated below: /pytorch/third_party/python-peachpy/LICENSE.rst + +Name: sigslot +License: Public Domain +Files: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/sigslot + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/sigslot/LICENSE + +Name: sleef +License: BSL-1.0 +Files: /pytorch/third_party/sleef + For details, see the files concatenated below: /pytorch/third_party/sleef/LICENSE.txt + +Name: swift +License: Apache-2.0 +Files: /pytorch/third_party/flatbuffers/swift + For details, see the files concatenated below: /pytorch/third_party/flatbuffers/swift/LICENSE + +Name: tb_plugin +License: BSD-3-Clause +Files: /pytorch/third_party/kineto/tb_plugin + For details, see the files concatenated below: /pytorch/third_party/kineto/tb_plugin/LICENSE + +Name: tensorflow-common +License: MIT +Files: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/tensorflow-common + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/tensorflow-common/LICENSE.txt + +Name: tensorpipe +License: BSD-3-Clause +Files: /pytorch/third_party/tensorpipe + For details, see the files concatenated below: /pytorch/third_party/tensorpipe/LICENSE.txt + +Name: test +License: MIT with exception +Files: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr/test + For details, see the files concatenated below: /pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr/test/LICENSE + +Name: variant +License: BSD-3-Clause +Files: /pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp/3rd_party/include/opentracing/variant + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp/3rd_party/include/opentracing/variant/LICENSE + +Name: vcpkg +License: MIT +Files: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/LICENSE.txt + +Name: vulkan +License: Apache-2.0 with exception +Files: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/vulkan + For details, see the files concatenated below: /pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/vulkan/LICENSE.txt + +/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM/LICENSE +---------------------------------------------------------------------------------- +Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + + +/pytorch/third_party/FP16/LICENSE +--------------------------------- +The MIT License (MIT) + +Copyright (c) 2017 Facebook Inc. +Copyright (c) 2017 Georgia Institute of Technology +Copyright 2019 Google LLC + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +/pytorch/third_party/FXdiv/LICENSE +---------------------------------- +The MIT License (MIT) + +Copyright (c) 2017 Facebook Inc. +Copyright (c) 2016-2017 Marat Dukhan + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +/pytorch/third_party/NNPACK/LICENSE +----------------------------------- +Copyright (c) 2017 Facebook Inc. +Copyright (c) 2015-2017, Georgia Institute of Technology +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/NVTX/LICENSE.txt +------------------------------------- +============================================================================== +NVTX is under the Apache License v2.0 with LLVM Exceptions: +============================================================================== + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +---- LLVM Exceptions to the Apache 2.0 License ---- + +As an exception, if, as a result of your compiling your source code, portions +of this Software are embedded into an Object form of such source code, you +may redistribute such embedded portions in such Object form without complying +with the conditions of Sections 4(a), 4(b) and 4(d) of the License. + +In addition, if you combine or link compiled forms of this Software with +software that is licensed under the GPLv2 ("Combined Software") and if a +court of competent jurisdiction determines that the patent provision (Section +3), the indemnity provision (Section 9) or other Section of the License +conflicts with the conditions of the GPLv2, you may retroactively and +prospectively choose to deem waived or otherwise exclude such Section(s) of +the License, but only in their entirety and only with respect to the Combined +Software. + + + +/pytorch/third_party/VulkanMemoryAllocator/LICENSE.txt +------------------------------------------------------ +Copyright (c) 2017-2025 Advanced Micro Devices, Inc. All rights reserved. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +/pytorch/third_party/XNNPACK/LICENSE +------------------------------------ +BSD License + +For XNNPACK software + +Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. +Copyright 2019 Google LLC + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + * Neither the name Facebook nor the names of its contributors may be used to + endorse or promote products derived from this software without specific + prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/aiter/LICENSE +---------------------------------- +Copyright © Advanced Micro Devices, Inc. All rights reserved. + +MIT License + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +/pytorch/third_party/benchmark/LICENSE +-------------------------------------- + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +/pytorch/third_party/opentelemetry-cpp/third_party/benchmark/LICENSE +-------------------------------------------------------------------- + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +/pytorch/third_party/protobuf/third_party/benchmark/LICENSE +----------------------------------------------------------- + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +/pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/boost-vcpkg-helpers/LICENSE.txt +---------------------------------------------------------------------------------------- +Copyright (c) Microsoft Corporation + +All rights reserved. + +MIT License + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb/examples/rest/cJSON/LICENSE +---------------------------------------------------------------------------------------------------------------------------------- +Copyright (c) 2009-2017 Dave Gamble and cJSON contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + + +/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb/examples/rest/cJSON/LICENSE +--------------------------------------------------------------------------------------------------------------- +Copyright (c) 2009-2017 Dave Gamble and cJSON contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + + +/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp/3rd_party/include/opentracing/catch2/LICENSE.txt +------------------------------------------------------------------------------------------------------------------- +Boost Software License - Version 1.0 - August 17th, 2003 + +Permission is hereby granted, free of charge, to any person or organization +obtaining a copy of the software and accompanying documentation covered by +this license (the "Software") to use, reproduce, display, distribute, +execute, and transmit the Software, and to prepare derivative works of the +Software, and to permit third-parties to whom the Software is furnished to +do so, all subject to the following: + +The copyright notices in the Software and this entire statement, including +the above license grant, this restriction and the following disclaimer, +must be included in all copies of the Software, in whole or in part, and +all derivative works of the Software, unless such copies or derivative +works are solely in the form of machine-executable object code generated by +a source language processor. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT +SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE +FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, +ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. + + +/pytorch/third_party/cpuinfo/deps/clog/LICENSE +---------------------------------------------- +Copyright (C) 2018 Marat Dukhan +Copyright (c) 2017-2018 Facebook Inc. +Copyright (c) 2017 Georgia Institute of Technology + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/fbgemm/external/cpuinfo/deps/clog/LICENSE +-------------------------------------------------------------- +Copyright (C) 2018 Marat Dukhan +Copyright (c) 2017-2018 Facebook Inc. +Copyright (c) 2017 Georgia Institute of Technology + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM/testing/python3/libs_3rdparty/colorama/LICENSE.txt +----------------------------------------------------------------------------------------------------------------------------- +Copyright (c) 2010 Jonathan Hartley + +Released under the New BSD license (reproduced below), or alternatively you may +use this software under any OSI approved open source license such as those at +http://opensource.org/licenses/alphabetical + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name(s) of the copyright holders, nor those of its contributors + may be used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + + +/pytorch/third_party/aiter/3rdparty/composable_kernel/LICENSE +------------------------------------------------------------- +Copyright (c) 2018- , Advanced Micro Devices, Inc. (Chao Liu, Jing Zhang) +Copyright (c) 2019- , Advanced Micro Devices, Inc. (Letao Qin, Qianfeng Zhang, Liang Huang, Shaojie Wang) +Copyright (c) 2022- , Advanced Micro Devices, Inc. (Anthony Chang, Chunyu Lai, Illia Silin, Adam Osewski, Poyen Chen, Jehandad Khan) +Copyright (c) 2019-2021, Advanced Micro Devices, Inc. (Hanwen Chang) +Copyright (c) 2019-2020, Advanced Micro Devices, Inc. (Tejash Shah) +Copyright (c) 2020 , Advanced Micro Devices, Inc. (Xiaoyan Zhou) +Copyright (c) 2021-2022, Advanced Micro Devices, Inc. (Jianfeng Yan) + +SPDX-License-Identifier: MIT +Copyright (c) 2018-2025, Advanced Micro Devices, Inc. All rights reserved. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +/pytorch/third_party/composable_kernel/LICENSE +---------------------------------------------- +Copyright (c) 2018- , Advanced Micro Devices, Inc. (Chao Liu, Jing Zhang) +Copyright (c) 2019- , Advanced Micro Devices, Inc. (Letao Qin, Qianfeng Zhang, Liang Huang, Shaojie Wang) +Copyright (c) 2022- , Advanced Micro Devices, Inc. (Anthony Chang, Chunyu Lai, Illia Silin, Adam Osewski, Poyen Chen, Jehandad Khan) +Copyright (c) 2019-2021, Advanced Micro Devices, Inc. (Hanwen Chang) +Copyright (c) 2019-2020, Advanced Micro Devices, Inc. (Tejash Shah) +Copyright (c) 2020 , Advanced Micro Devices, Inc. (Xiaoyan Zhou) +Copyright (c) 2021-2022, Advanced Micro Devices, Inc. (Jianfeng Yan) + +SPDX-License-Identifier: MIT +Copyright (c) 2018-2025, Advanced Micro Devices, Inc. All rights reserved. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +/pytorch/third_party/fbgemm/external/composable_kernel/LICENSE +-------------------------------------------------------------- +Copyright (c) 2018- , Advanced Micro Devices, Inc. (Chao Liu, Jing Zhang) +Copyright (c) 2019- , Advanced Micro Devices, Inc. (Letao Qin, Qianfeng Zhang, Liang Huang, Shaojie Wang) +Copyright (c) 2022- , Advanced Micro Devices, Inc. (Anthony Chang, Chunyu Lai, Illia Silin, Adam Osewski, Poyen Chen, Jehandad Khan) +Copyright (c) 2019-2021, Advanced Micro Devices, Inc. (Hanwen Chang) +Copyright (c) 2019-2020, Advanced Micro Devices, Inc. (Tejash Shah) +Copyright (c) 2020 , Advanced Micro Devices, Inc. (Xiaoyan Zhou) +Copyright (c) 2021-2022, Advanced Micro Devices, Inc. (Jianfeng Yan) + +SPDX-License-Identifier: MIT +Copyright (c) 2018-2025, Advanced Micro Devices, Inc. All rights reserved. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +/pytorch/third_party/flash-attention/csrc/composable_kernel/LICENSE +------------------------------------------------------------------- +Copyright (c) 2018- , Advanced Micro Devices, Inc. (Chao Liu, Jing Zhang) +Copyright (c) 2019- , Advanced Micro Devices, Inc. (Letao Qin, Qianfeng Zhang, Liang Huang, Shaojie Wang) +Copyright (c) 2022- , Advanced Micro Devices, Inc. (Anthony Chang, Chunyu Lai, Illia Silin, Adam Osewski, Poyen Chen, Jehandad Khan) +Copyright (c) 2019-2021, Advanced Micro Devices, Inc. (Hanwen Chang) +Copyright (c) 2019-2020, Advanced Micro Devices, Inc. (Tejash Shah) +Copyright (c) 2020 , Advanced Micro Devices, Inc. (Xiaoyan Zhou) +Copyright (c) 2021-2022, Advanced Micro Devices, Inc. (Jianfeng Yan) + +SPDX-License-Identifier: MIT +Copyright (c) 2018-2024, Advanced Micro Devices, Inc. All rights reserved. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +/pytorch/third_party/cpp-httplib/LICENSE +---------------------------------------- +The MIT License (MIT) + +Copyright (c) 2017 yhirose + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + + +/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json/third_party/cpplint/LICENSE +------------------------------------------------------------------------------------------------------ +cpplint.py and its corresponding unit tests are Copyright (C) 2009 Google Inc. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr/LICENSE +--------------------------------------------------------------------------------- +This license applies to everything except the contents of the "test" +directory and its subdirectories. + +MIT License + +Copyright (c) 2017-2021 Huu Nguyen +Copyright (c) 2022 libcpr and many other contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +/pytorch/third_party/cpuinfo/LICENSE +------------------------------------ +Copyright (c) 2019 Google LLC +Copyright (c) 2017-2018 Facebook Inc. +Copyright (C) 2012-2017 Georgia Institute of Technology +Copyright (C) 2010-2012 Marat Dukhan + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/fbgemm/external/cpuinfo/LICENSE +---------------------------------------------------- +Copyright (c) 2019 Google LLC +Copyright (c) 2017-2018 Facebook Inc. +Copyright (C) 2012-2017 Georgia Institute of Technology +Copyright (C) 2010-2012 Marat Dukhan + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/cudnn_frontend/LICENSE.txt +----------------------------------------------- +/* + * Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER + * DEALINGS IN THE SOFTWARE. + */ + + +/pytorch/third_party/cutlass/LICENSE.txt +---------------------------------------- +Copyright (c) 2017 - 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: BSD-3-Clause + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this +list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, +this list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Certain files within this repository are subject to separate licensing terms: + +- The files located in the `python/CuTeDSL` directory are licensed under the + NVIDIA End User License Agreement (EULA). Please refer to + https://docs.nvidia.com/cutlass/media/docs/pythonDSL/license.html + for the full terms. + + +/pytorch/third_party/fbgemm/external/cutlass/LICENSE.txt +-------------------------------------------------------- +Copyright (c) 2017 - 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: BSD-3-Clause + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this +list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, +this list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Certain files within this repository are subject to separate licensing terms: + +- The files located in the `python/CuTeDSL` directory are licensed under the + NVIDIA End User License Agreement (EULA). Please refer to + https://docs.nvidia.com/cutlass/media/docs/pythonDSL/license.html + for the full terms. + + +/pytorch/third_party/flash-attention/csrc/cutlass/LICENSE.txt +------------------------------------------------------------- +Copyright (c) 2017 - 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: BSD-3-Clause + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this +list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, +this list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/flatbuffers/dart/LICENSE +--------------------------------------------- + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2014 Google Inc. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +/pytorch/third_party/NVTX/docs/LICENSE.txt +------------------------------------------ +============================================================================== +NVTX is under the Apache License v2.0 with LLVM Exceptions: +============================================================================== + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +---- LLVM Exceptions to the Apache 2.0 License ---- + +As an exception, if, as a result of your compiling your source code, portions +of this Software are embedded into an Object form of such source code, you +may redistribute such embedded portions in such Object form without complying +with the conditions of Sections 4(a), 4(b) and 4(d) of the License. + +In addition, if you combine or link compiled forms of this Software with +software that is licensed under the GPLv2 ("Combined Software") and if a +court of competent jurisdiction determines that the patent provision (Section +3), the indemnity provision (Section 9) or other Section of the License +conflicts with the conditions of the GPLv2, you may retroactively and +prospectively choose to deem waived or otherwise exclude such Section(s) of +the License, but only in their entirety and only with respect to the Combined +Software. + + + +/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json/test/thirdparty/doctest/LICENSE.txt +-------------------------------------------------------------------------------------------------------------- +The MIT License (MIT) + +Copyright (c) 2016-2021 Viktor Kirilov + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb/src/third_party/duktape-1.5.2/LICENSE.txt +------------------------------------------------------------------------------------------------------------------------------------------------ +=============== +Duktape license +=============== + +(http://opensource.org/licenses/MIT) + +Copyright (c) 2013-2016 by Duktape authors (see AUTHORS.rst) + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb/src/third_party/duktape-1.5.2/LICENSE.txt +----------------------------------------------------------------------------------------------------------------------------- +=============== +Duktape license +=============== + +(http://opensource.org/licenses/MIT) + +Copyright (c) 2013-2016 by Duktape authors (see AUTHORS.rst) + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb/src/third_party/duktape-1.8.0/LICENSE.txt +------------------------------------------------------------------------------------------------------------------------------------------------ +=============== +Duktape license +=============== + +(http://opensource.org/licenses/MIT) + +Copyright (c) 2013-2017 by Duktape authors (see AUTHORS.rst) + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb/src/third_party/duktape-1.8.0/LICENSE.txt +----------------------------------------------------------------------------------------------------------------------------- +=============== +Duktape license +=============== + +(http://opensource.org/licenses/MIT) + +Copyright (c) 2013-2017 by Duktape authors (see AUTHORS.rst) + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +/pytorch/third_party/kineto/libkineto/third_party/dynolog/LICENSE +----------------------------------------------------------------- +MIT License + +Copyright (c) Facebook, Inc. and its affiliates. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +/pytorch/third_party/opentelemetry-cpp/exporters/etw/include/opentelemetry/exporters/etw/LICENSE +------------------------------------------------------------------------------------------------ +TraceLogging Dynamic for Windows + +Copyright (c) Microsoft Corporation. All rights reserved. + +MIT License + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp/3rd_party/include/opentracing/expected/LICENSE +----------------------------------------------------------------------------------------------------------------- +The MIT License (MIT) + +Copyright (c) 2015 Martin Moene +Copyright (c) 2015 Microsoft Corporation. All rights reserved. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + + +/pytorch/third_party/fbgemm/LICENSE +----------------------------------- +BSD License + +For FBGEMM software + +Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + * Neither the name Facebook nor the names of its contributors may be used to + endorse or promote products derived from this software without specific + prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/ffnvcodec/LICENSE.txt +------------------------------------------------------------------------------ +GNU LESSER GENERAL PUBLIC LICENSE +Version 2.1, February 1999 + +Copyright (C) 1991, 1999 Free Software Foundation, Inc. +51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +Everyone is permitted to copy and distribute verbatim copies +of this license document, but changing it is not allowed. + +[This is the first released version of the Lesser GPL. It also counts + as the successor of the GNU Library Public License, version 2, hence + the version number 2.1.] +Preamble +The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public Licenses are intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. + +This license, the Lesser General Public License, applies to some specially designated software packages--typically libraries--of the Free Software Foundation and other authors who decide to use it. You can use it too, but we suggest you first think carefully about whether this license or the ordinary General Public License is the better strategy to use in any particular case, based on the explanations below. + +When we speak of free software, we are referring to freedom of use, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish); that you receive source code or can get it if you want it; that you can change the software and use pieces of it in new free programs; and that you are informed that you can do these things. + +To protect your rights, we need to make restrictions that forbid distributors to deny you these rights or to ask you to surrender these rights. These restrictions translate to certain responsibilities for you if you distribute copies of the library or if you modify it. + +For example, if you distribute copies of the library, whether gratis or for a fee, you must give the recipients all the rights that we gave you. You must make sure that they, too, receive or can get the source code. If you link other code with the library, you must provide complete object files to the recipients, so that they can relink them with the library after making changes to the library and recompiling it. And you must show them these terms so they know their rights. + +We protect your rights with a two-step method: (1) we copyright the library, and (2) we offer you this license, which gives you legal permission to copy, distribute and/or modify the library. + +To protect each distributor, we want to make it very clear that there is no warranty for the free library. Also, if the library is modified by someone else and passed on, the recipients should know that what they have is not the original version, so that the original author's reputation will not be affected by problems that might be introduced by others. + +Finally, software patents pose a constant threat to the existence of any free program. We wish to make sure that a company cannot effectively restrict the users of a free program by obtaining a restrictive license from a patent holder. Therefore, we insist that any patent license obtained for a version of the library must be consistent with the full freedom of use specified in this license. + +Most GNU software, including some libraries, is covered by the ordinary GNU General Public License. This license, the GNU Lesser General Public License, applies to certain designated libraries, and is quite different from the ordinary General Public License. We use this license for certain libraries in order to permit linking those libraries into non-free programs. + +When a program is linked with a library, whether statically or using a shared library, the combination of the two is legally speaking a combined work, a derivative of the original library. The ordinary General Public License therefore permits such linking only if the entire combination fits its criteria of freedom. The Lesser General Public License permits more lax criteria for linking other code with the library. + +We call this license the "Lesser" General Public License because it does Less to protect the user's freedom than the ordinary General Public License. It also provides other free software developers Less of an advantage over competing non-free programs. These disadvantages are the reason we use the ordinary General Public License for many libraries. However, the Lesser license provides advantages in certain special circumstances. + +For example, on rare occasions, there may be a special need to encourage the widest possible use of a certain library, so that it becomes a de-facto standard. To achieve this, non-free programs must be allowed to use the library. A more frequent case is that a free library does the same job as widely used non-free libraries. In this case, there is little to gain by limiting the free library to free software only, so we use the Lesser General Public License. + +In other cases, permission to use a particular library in non-free programs enables a greater number of people to use a large body of free software. For example, permission to use the GNU C Library in non-free programs enables many more people to use the whole GNU operating system, as well as its variant, the GNU/Linux operating system. + +Although the Lesser General Public License is Less protective of the users' freedom, it does ensure that the user of a program that is linked with the Library has the freedom and the wherewithal to run that program using a modified version of the Library. + +The precise terms and conditions for copying, distribution and modification follow. Pay close attention to the difference between a "work based on the library" and a "work that uses the library". The former contains code derived from the library, whereas the latter must be combined with the library in order to run. + +TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION +0. This License Agreement applies to any software library or other program which contains a notice placed by the copyright holder or other authorized party saying it may be distributed under the terms of this Lesser General Public License (also called "this License"). Each licensee is addressed as "you". + +A "library" means a collection of software functions and/or data prepared so as to be conveniently linked with application programs (which use some of those functions and data) to form executables. + +The "Library", below, refers to any such software library or work which has been distributed under these terms. A "work based on the Library" means either the Library or any derivative work under copyright law: that is to say, a work containing the Library or a portion of it, either verbatim or with modifications and/or translated straightforwardly into another language. (Hereinafter, translation is included without limitation in the term "modification".) + +"Source code" for a work means the preferred form of the work for making modifications to it. For a library, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the library. + +Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running a program using the Library is not restricted, and output from such a program is covered only if its contents constitute a work based on the Library (independent of the use of the Library in a tool for writing it). Whether that is true depends on what the Library does and what the program that uses the Library does. + +1. You may copy and distribute verbatim copies of the Library's complete source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and distribute a copy of this License along with the Library. + +You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. + +2. You may modify your copy or copies of the Library or any portion of it, thus forming a work based on the Library, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: + +a) The modified work must itself be a software library. +b) You must cause the files modified to carry prominent notices stating that you changed the files and the date of any change. +c) You must cause the whole of the work to be licensed at no charge to all third parties under the terms of this License. +d) If a facility in the modified Library refers to a function or a table of data to be supplied by an application program that uses the facility, other than as an argument passed when the facility is invoked, then you must make a good faith effort to ensure that, in the event an application does not supply such function or table, the facility still operates, and performs whatever part of its purpose remains meaningful. +(For example, a function in a library to compute square roots has a purpose that is entirely well-defined independent of the application. Therefore, Subsection 2d requires that any application-supplied function or table used by this function must be optional: if the application does not supply it, the square root function must still compute square roots.) + +These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Library, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Library, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Library. + +In addition, mere aggregation of another work not based on the Library with the Library (or with a work based on the Library) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. + +3. You may opt to apply the terms of the ordinary GNU General Public License instead of this License to a given copy of the Library. To do this, you must alter all the notices that refer to this License, so that they refer to the ordinary GNU General Public License, version 2, instead of to this License. (If a newer version than version 2 of the ordinary GNU General Public License has appeared, then you can specify that version instead if you wish.) Do not make any other change in these notices. + +Once this change is made in a given copy, it is irreversible for that copy, so the ordinary GNU General Public License applies to all subsequent copies and derivative works made from that copy. + +This option is useful when you wish to copy part of the code of the Library into a program that is not a library. + +4. You may copy and distribute the Library (or a portion or derivative of it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange. + +If distribution of object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place satisfies the requirement to distribute the source code, even though third parties are not compelled to copy the source along with the object code. + +5. A program that contains no derivative of any portion of the Library, but is designed to work with the Library by being compiled or linked with it, is called a "work that uses the Library". Such a work, in isolation, is not a derivative work of the Library, and therefore falls outside the scope of this License. + +However, linking a "work that uses the Library" with the Library creates an executable that is a derivative of the Library (because it contains portions of the Library), rather than a "work that uses the library". The executable is therefore covered by this License. Section 6 states terms for distribution of such executables. + +When a "work that uses the Library" uses material from a header file that is part of the Library, the object code for the work may be a derivative work of the Library even though the source code is not. Whether this is true is especially significant if the work can be linked without the Library, or if the work is itself a library. The threshold for this to be true is not precisely defined by law. + +If such an object file uses only numerical parameters, data structure layouts and accessors, and small macros and small inline functions (ten lines or less in length), then the use of the object file is unrestricted, regardless of whether it is legally a derivative work. (Executables containing this object code plus portions of the Library will still fall under Section 6.) + +Otherwise, if the work is a derivative of the Library, you may distribute the object code for the work under the terms of Section 6. Any executables containing that work also fall under Section 6, whether or not they are linked directly with the Library itself. + +6. As an exception to the Sections above, you may also combine or link a "work that uses the Library" with the Library to produce a work containing portions of the Library, and distribute that work under terms of your choice, provided that the terms permit modification of the work for the customer's own use and reverse engineering for debugging such modifications. + +You must give prominent notice with each copy of the work that the Library is used in it and that the Library and its use are covered by this License. You must supply a copy of this License. If the work during execution displays copyright notices, you must include the copyright notice for the Library among them, as well as a reference directing the user to the copy of this License. Also, you must do one of these things: + +a) Accompany the work with the complete corresponding machine-readable source code for the Library including whatever changes were used in the work (which must be distributed under Sections 1 and 2 above); and, if the work is an executable linked with the Library, with the complete machine-readable "work that uses the Library", as object code and/or source code, so that the user can modify the Library and then relink to produce a modified executable containing the modified Library. (It is understood that the user who changes the contents of definitions files in the Library will not necessarily be able to recompile the application to use the modified definitions.) +b) Use a suitable shared library mechanism for linking with the Library. A suitable mechanism is one that (1) uses at run time a copy of the library already present on the user's computer system, rather than copying library functions into the executable, and (2) will operate properly with a modified version of the library, if the user installs one, as long as the modified version is interface-compatible with the version that the work was made with. +c) Accompany the work with a written offer, valid for at least three years, to give the same user the materials specified in Subsection 6a, above, for a charge no more than the cost of performing this distribution. +d) If distribution of the work is made by offering access to copy from a designated place, offer equivalent access to copy the above specified materials from the same place. +e) Verify that the user has already received a copy of these materials or that you have already sent this user a copy. +For an executable, the required form of the "work that uses the Library" must include any data and utility programs needed for reproducing the executable from it. However, as a special exception, the materials to be distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. + +It may happen that this requirement contradicts the license restrictions of other proprietary libraries that do not normally accompany the operating system. Such a contradiction means you cannot use both them and the Library together in an executable that you distribute. + +7. You may place library facilities that are a work based on the Library side-by-side in a single library together with other library facilities not covered by this License, and distribute such a combined library, provided that the separate distribution of the work based on the Library and of the other library facilities is otherwise permitted, and provided that you do these two things: + +a) Accompany the combined library with a copy of the same work based on the Library, uncombined with any other library facilities. This must be distributed under the terms of the Sections above. +b) Give prominent notice with the combined library of the fact that part of it is a work based on the Library, and explaining where to find the accompanying uncombined form of the same work. +8. You may not copy, modify, sublicense, link with, or distribute the Library except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense, link with, or distribute the Library is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. + +9. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Library or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Library (or any work based on the Library), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Library or works based on it. + +10. Each time you redistribute the Library (or any work based on the Library), the recipient automatically receives a license from the original licensor to copy, distribute, link with or modify the Library subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties with this License. + +11. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Library at all. For example, if a patent license would not permit royalty-free redistribution of the Library by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Library. + +If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply, and the section as a whole is intended to apply in other circumstances. + +It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. + +This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. + +12. If the distribution and/or use of the Library is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Library under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. + +13. The Free Software Foundation may publish revised and/or new versions of the Lesser General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Library specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Library does not specify a license version number, you may choose any version ever published by the Free Software Foundation. + +14. If you wish to incorporate parts of the Library into other free programs whose distribution conditions are incompatible with these, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. + +NO WARRANTY + +15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + +16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +END OF TERMS AND CONDITIONS +How to Apply These Terms to Your New Libraries +If you develop a new library, and you want it to be of the greatest possible use to the public, we recommend making it free software that everyone can redistribute and change. You can do so by permitting redistribution under these terms (or, alternatively, under the terms of the ordinary General Public License). + +To apply these terms, attach the following notices to the library. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. + +one line to give the library's name and an idea of what it does. +Copyright (C) year name of author + +This library is free software; you can redistribute it and/or +modify it under the terms of the GNU Lesser General Public +License as published by the Free Software Foundation; either +version 2.1 of the License, or (at your option) any later version. + +This library is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +Lesser General Public License for more details. + +You should have received a copy of the GNU Lesser General Public +License along with this library; if not, write to the Free Software +Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +Also add information on how to contact you by electronic and paper mail. + +You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the library, if necessary. Here is a sample; alter the names: + +Yoyodyne, Inc., hereby disclaims all copyright interest in +the library `Frob' (a library for tweaking knobs) written +by James Random Hacker. + +signature of Ty Coon, 1 April 1990 +Ty Coon, President of Vice +That's all there is to it! + +/pytorch/third_party/flash-attention/LICENSE +-------------------------------------------- +BSD 3-Clause License + +Copyright (c) 2022, the respective contributors, as shown by the AUTHORS file. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/flatbuffers/LICENSE +---------------------------------------- + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +/pytorch/third_party/fmt/LICENSE +-------------------------------- +Copyright (c) 2012 - present, Victor Zverovich and {fmt} contributors + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +--- Optional exception to the license --- + +As an exception, if, as a result of your compiling your source code, portions +of this Software are embedded into a machine-executable object form of such +source code, you may redistribute such embedded portions in such object form +without including the above copyright and permission notices. + + +/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt/LICENSE.rst +------------------------------------------------------------------------------------- +Copyright (c) 2012 - present, Victor Zverovich + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +--- Optional exception to the license --- + +As an exception, if, as a result of your compiling your source code, portions +of this Software are embedded into a machine-executable object form of such +source code, you may redistribute such embedded portions in such object form +without including the above copyright and permission notices. + + +/pytorch/third_party/kineto/libkineto/third_party/fmt/LICENSE +------------------------------------------------------------- +Copyright (c) 2012 - present, Victor Zverovich and {fmt} contributors + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +--- Optional exception to the license --- + +As an exception, if, as a result of your compiling your source code, portions +of this Software are embedded into a machine-executable object form of such +source code, you may redistribute such embedded portions in such object form +without including the above copyright and permission notices. + + +/pytorch/third_party/gemmlowp/gemmlowp/LICENSE +---------------------------------------------- + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest/googlemock/scripts/generator/LICENSE +--------------------------------------------------------------------------------------------------------------------------------------------- + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [2007] Neal Norwitz + Portions Copyright [2007] Google Inc. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest/googlemock/scripts/generator/LICENSE +-------------------------------------------------------------------------------------------------------------------------- + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [2007] Neal Norwitz + Portions Copyright [2007] Google Inc. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +/pytorch/third_party/protobuf/third_party/googletest/googlemock/scripts/generator/LICENSE +----------------------------------------------------------------------------------------- + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [2007] Neal Norwitz + Portions Copyright [2007] Google Inc. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +/pytorch/third_party/tensorpipe/third_party/googletest/googlemock/scripts/generator/LICENSE +------------------------------------------------------------------------------------------- + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [2007] Neal Norwitz + Portions Copyright [2007] Google Inc. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +/pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/gettimeofday/LICENSE +----------------------------------------------------------------------------- +/* + * Copied from PostgreSQL source: + * http://doxygen.postgresql.org/gettimeofday_8c_source.html + * + */ + +/* + * gettimeofday.c + * Win32 gettimeofday() replacement + * + * src/port/gettimeofday.c + * + * Copyright (c) 2003 SRA, Inc. + * Copyright (c) 2003 SKC, Inc. + * + * Permission to use, copy, modify, and distribute this software and + * its documentation for any purpose, without fee, and without a + * written agreement is hereby granted, provided that the above + * copyright notice and this paragraph and the following two + * paragraphs appear in all copies. + * + * IN NO EVENT SHALL THE AUTHOR BE LIABLE TO ANY PARTY FOR DIRECT, + * INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING + * LOST PROFITS, ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS + * DOCUMENTATION, EVEN IF THE UNIVERSITY OF CALIFORNIA HAS BEEN ADVISED + * OF THE POSSIBILITY OF SUCH DAMAGE. + * + * THE AUTHOR SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE. THE SOFTWARE PROVIDED HEREUNDER IS ON AN "AS + * IS" BASIS, AND THE AUTHOR HAS NO OBLIGATIONS TO PROVIDE MAINTENANCE, + * SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. + */ + + +/pytorch/third_party/gloo/LICENSE +--------------------------------- +BSD License + +For Gloo software + +Copyright (c) 2017-present, Facebook, Inc. All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + * Neither the name Facebook nor the names of its contributors may be used to + endorse or promote products derived from this software without specific + prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/protobuf/third_party/googletest/googlemock/LICENSE +----------------------------------------------------------------------- +Copyright 2008, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/tensorpipe/third_party/googletest/googlemock/LICENSE +------------------------------------------------------------------------- +Copyright 2008, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/fbgemm/external/googletest/LICENSE +------------------------------------------------------- +Copyright 2008, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/googletest/LICENSE +--------------------------------------- +Copyright 2008, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest/LICENSE +---------------------------------------------------------------------------------------- +Copyright 2008, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest/LICENSE +---------------------------------------------------------------------------------------------------------------- +Copyright 2008, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/kineto/libkineto/third_party/googletest/LICENSE +-------------------------------------------------------------------- +Copyright 2008, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/opentelemetry-cpp/third_party/googletest/LICENSE +--------------------------------------------------------------------- +Copyright 2008, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest/LICENSE +--------------------------------------------------------------------------------------------- +Copyright 2008, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/protobuf/third_party/googletest/LICENSE +------------------------------------------------------------ +Copyright 2008, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/protobuf/third_party/googletest/googletest/LICENSE +----------------------------------------------------------------------- +Copyright 2008, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/tensorpipe/third_party/googletest/LICENSE +-------------------------------------------------------------- +Copyright 2008, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/tensorpipe/third_party/googletest/googletest/LICENSE +------------------------------------------------------------------------- +Copyright 2008, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/ideep/mkl-dnn/tests/gtests/gtest/LICENSE +------------------------------------------------------------- +Copyright 2008, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/fbgemm/external/hipify_torch/LICENSE.txt +------------------------------------------------------------- +MIT License + +Copyright (c) 2021-2024, Advanced Micro Devices, Inc. All rights reserved. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +/pytorch/third_party/fbgemm/fbgemm_gpu/experimental/hstu/LICENSE +---------------------------------------------------------------- +BSD 3-Clause License + +Copyright (c) 2022, the respective contributors, as shown by the AUTHORS file. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +/* + * SPDX-FileCopyrightText: Copyright (c) <2024> NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: LicenseRef-NvidiaProprietary + * + * NVIDIA CORPORATION, its affiliates and licensors retain all intellectual + * property and proprietary rights in and to this material, related + * documentation and any modifications thereto. Any use, reproduction, + * disclosure or distribution of this material and related documentation + * without an express license agreement from NVIDIA CORPORATION or + * its affiliates is strictly prohibited. + */ + +/pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/hungarian/LICENSE.txt +------------------------------------------------------------------------------ +/******************************************************************** + ******************************************************************** + ** + ** libhungarian by Cyrill Stachniss, 2004 + ** + ** + ** Solving the Minimum Assignment Problem using the + ** Hungarian Method. + ** + ** ** This file may be freely copied and distributed! ** + ** + ** Parts of the used code was originally provided by the + ** "Stanford GraphGase", but I made changes to this code. + ** As asked by the copyright node of the "Stanford GraphGase", + ** I hereby proclaim that this file are *NOT* part of the + ** "Stanford GraphGase" distrubition! + ** + ** This file is distributed in the hope that it will be useful, + ** but WITHOUT ANY WARRANTY; without even the implied + ** warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR + ** PURPOSE. + ** + ******************************************************************** + ********************************************************************/ + + +/pytorch/third_party/ideep/LICENSE +---------------------------------- +Copyright (c) 2018 Intel Corporation. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +/pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/irrlicht/LICENSE.txt +----------------------------------------------------------------------------- +The Irrlicht Engine License +=========================== + +Copyright (C) 2002-2015 Nikolaus Gebhardt + +This software is provided 'as-is', without any express or implied +warranty. In no event will the authors be held liable for any damages +arising from the use of this software. + +Permission is granted to anyone to use this software for any purpose, +including commercial applications, and to alter it and redistribute it +freely, subject to the following restrictions: + +1. The origin of this software must not be misrepresented; you must not + claim that you wrote the original software. If you use this software + in a product, an acknowledgement in the product documentation would be + appreciated but is not required. +2. Altered source versions must be clearly marked as such, and must not be + misrepresented as being the original software. +3. This notice may not be removed or altered from any source distribution. + +/pytorch/third_party/kineto/LICENSE +----------------------------------- +BSD License + +For Kineto software + +Copyright (c) Meta Platforms, Inc. and affiliates. + +All contributions by Microsoft: +Copyright (c) Microsoft Corporation. (The Azure AI Platform team) + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + * Neither the name Meta nor the names of its contributors may be used to + endorse or promote products derived from this software without specific + prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/tensorpipe/third_party/libnop/LICENSE +---------------------------------------------------------- +Copyright 2017 The Native Object Protocols Authors + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + https://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + + +/pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/libstemmer/LICENSE +--------------------------------------------------------------------------- +Snowball - License +Except where explicitly noted, all the software given out on this Snowball site is covered by the 3-clause BSD License: + +Copyright (c) 2001, Dr Martin Porter, +Copyright (c) 2002, Richard Boulton. +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Essentially, all this means is that you can do what you like with the code, except claim another Copyright for it, or claim that it is issued under a different license. The software is also issued without warranties, which means that if anyone suffers through its use, they cannot come back and sue you. You also have to alert anyone to whom you give the Snowball software to the fact that it is covered by the BSD license. + +We have not bothered to insert the licensing arrangement into the text of the Snowball software. + + +/pytorch/third_party/tensorpipe/third_party/libuv/LICENSE +--------------------------------------------------------- +Copyright (c) 2015-present libuv project contributors. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to +deal in the Software without restriction, including without limitation the +rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +sell copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. + + +/pytorch/third_party/mimalloc/LICENSE +------------------------------------- +MIT License + +Copyright (c) 2018-2025 Microsoft Corporation, Daan Leijen + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +/pytorch/third_party/miniz-3.0.2/LICENSE +---------------------------------------- +Copyright 2013-2014 RAD Game Tools and Valve Software +Copyright 2010-2014 Rich Geldreich and Tenacious Software LLC + +All Rights Reserved. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +/pytorch/third_party/ideep/mkl-dnn/LICENSE +------------------------------------------ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + ============================================================================ + + Copyright 2016-2023 Intel Corporation + Copyright 2018 YANDEX LLC + Copyright 2019-2023 FUJITSU LIMITED + Copyright 2020-2023 Arm Ltd. and affiliates + Copyright 2020-2022 Codeplay Software Limited + Copyright 2021 Alanna Tempest + Copyright 2022-2023 IBM Corporation + Copyright 2023 KNS Group LLC (YADRO) + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + This distribution includes third party software ("third party programs"). + This third party software, even if included with the distribution of + the Intel software, may be governed by separate license terms, including + without limitation, third party license terms, other Intel software license + terms, and open source software license terms. These separate license terms + govern your use of the third party programs as set forth in the + "THIRD-PARTY-PROGRAMS" file. + + +/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl/LICENSE +----------------------------------------------------------------- +Copyright (c) 2015 Microsoft Corporation. All rights reserved. + +This code is licensed under the MIT License (MIT). + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +/pytorch/third_party/fbgemm/fbgemm_gpu/src/quantize_ops/mx/LICENSE +------------------------------------------------------------------ + MIT License + + Copyright (c) Microsoft Corporation. + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE + + +/pytorch/third_party/fbgemm/fbgemm_gpu/test/quantize/mx/LICENSE +--------------------------------------------------------------- + MIT License + + Copyright (c) Microsoft Corporation. + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE + + +/pytorch/third_party/onnx/LICENSE +--------------------------------- + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +/pytorch/third_party/opentelemetry-cpp/LICENSE +---------------------------------------------- + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto/LICENSE +------------------------------------------------------------------------------ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp/LICENSE +-------------------------------------------------------------------------- + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright The OpenTracing Authors + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +/pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/pdcurses/LICENSE +------------------------------------------------------------------------- +The core package is in the public domain, but small portions of PDCurses are subject to copyright under various licenses. + +The win32 files are released to the public domain. + +If you use PDCurses in an application, an acknowledgement would be appreciated, but is not mandatory. If you make corrections or enhancements to PDCurses, please forward them to the current maintainer for the benefit of other users. + +This software is provided AS IS with NO WARRANTY whatsoever. + +/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs/LICENSE +--------------------------------------------------------------------------------- + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + Copyright 2020-present Daniel Trugman + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +/pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/physac/LICENSE +----------------------------------------------------------------------- +MIT License + +Copyright (c) 2022 Víctor Fisac + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +/pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/pqp/LICENSE +-------------------------------------------------------------------- +Copyright 1999 University of North Carolina at Chapel Hill. +All rights reserved. + +Permission to use, copy, modify, and distribute this software and its +documentation for educational, research, and non-profit purposes, without fee, +and without a written agreement is hereby granted, provided that the above +copyright notice and the following three paragraphs appear in all copies. + +IN NO EVENT SHALL THE UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL BE LIABLE TO +ANY PARTY FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, +INCLUDING LOST PROFITS, ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS +DOCUMENTATION, EVEN IF THE UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL HAS +BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +THE UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL SPECIFICALLY DISCLAIMS ANY +WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE SOFTWARE PROVIDED +HEREUNDER IS ON AN "AS IS" BASIS, AND THE UNIVERSITY OF NORTH CAROLINA AT +CHAPEL HILL HAS NO OBLIGATIONS TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, +ENHANCEMENTS, OR MODIFICATIONS. + +The authors may be contacted via: + +US Mail: Eric Larsen, Stefan Gottschalk + Department of Computer Science + Sitterson Hall, CB #3175 + University of North Carolina + Chapel Hill, NC 27599-3175 + +Phone: (919) 962-1749 + +Email: geom@cs.unc.edu + +/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/LICENSE +-------------------------------------------------------------------------------------------- +MIT License + +Copyright (c) 2016-2021 Jupp Mueller +Copyright (c) 2017-2022 Gregor Jasny + +And many contributors, see +https://github.com/jupp0r/prometheus-cpp/graphs/contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/LICENSE +------------------------------------------------------------------------- +MIT License + +Copyright (c) 2016-2021 Jupp Mueller +Copyright (c) 2017-2022 Gregor Jasny + +And many contributors, see +https://github.com/jupp0r/prometheus-cpp/graphs/contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +/pytorch/third_party/protobuf/LICENSE +------------------------------------- +Copyright 2008 Google Inc. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Code generated by the Protocol Buffer compiler is owned by the owner +of the input file used when generating it. This code is not +standalone and requires a support library to be linked with it. This +support library is itself covered by the above license. + + +/pytorch/third_party/psimd/LICENSE +---------------------------------- +The MIT License (MIT) + +Copyright (c) 2017 Facebook Inc. +Copyright (c) 2014-2017 Georgia Institute of Technology +Copyright 2019 Google LLC + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +/pytorch/third_party/pthreadpool/LICENSE +---------------------------------------- +Copyright 2019 Google LLC +Copyright (c) 2017 Facebook Inc. +Copyright (c) 2015-2017 Georgia Institute of Technology +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + + +/pytorch/third_party/onnx/third_party/pybind11/LICENSE +------------------------------------------------------ +Copyright (c) 2016 Wenzel Jakob , All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Please also refer to the file .github/CONTRIBUTING.md, which clarifies licensing of +external contributions to this project including patches, pull requests, etc. + + +/pytorch/third_party/pybind11/LICENSE +------------------------------------- +Copyright (c) 2016 Wenzel Jakob , All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Please also refer to the file .github/CONTRIBUTING.md, which clarifies licensing of +external contributions to this project including patches, pull requests, etc. + + +/pytorch/third_party/tensorpipe/third_party/pybind11/LICENSE +------------------------------------------------------------ +Copyright (c) 2016 Wenzel Jakob , All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Please also refer to the file CONTRIBUTING.md, which clarifies licensing of +external contributions to this project including patches, pull requests, etc. + + +/pytorch/third_party/NVTX/python/LICENSE.txt +-------------------------------------------- +============================================================================== +NVTX is under the Apache License v2.0 with LLVM Exceptions: +============================================================================== + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +---- LLVM Exceptions to the Apache 2.0 License ---- + +As an exception, if, as a result of your compiling your source code, portions +of this Software are embedded into an Object form of such source code, you +may redistribute such embedded portions in such Object form without complying +with the conditions of Sections 4(a), 4(b) and 4(d) of the License. + +In addition, if you combine or link compiled forms of this Software with +software that is licensed under the GPLv2 ("Combined Software") and if a +court of competent jurisdiction determines that the patent provision (Section +3), the indemnity provision (Section 9) or other Section of the License +conflicts with the conditions of the GPLv2, you may retroactively and +prospectively choose to deem waived or otherwise exclude such Section(s) of +the License, but only in their entirety and only with respect to the Combined +Software. + + + +/pytorch/third_party/cutlass/python/LICENSE.txt +----------------------------------------------- +Copyright (c) 2017 - 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: BSD-3-Clause + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this +list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, +this list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/fbgemm/external/cutlass/python/LICENSE.txt +--------------------------------------------------------------- +Copyright (c) 2017 - 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: BSD-3-Clause + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this +list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, +this list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/flash-attention/csrc/cutlass/python/LICENSE.txt +-------------------------------------------------------------------- +Copyright (c) 2017 - 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: BSD-3-Clause + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this +list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, +this list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/python-peachpy/LICENSE.rst +----------------------------------------------- +============================== +PeachPy license (2-clause BSD) +============================== + +Copyright (c) 2017, Facebook Inc. +Copyright (c) 2013-2017, Georgia Institute of Technology +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/sigslot/LICENSE +------------------------------------------------------------------------ +License +The sigslot library has been placed in the public domain. This means that you are free to use it however you like. + +The author takes no responsibility or liability of any kind for any use that you may make of this library. + +If you screw up, it's your fault. + +If the library screws up, you got it for free, so you should have tested it better - it's still your responsibility. + +/pytorch/third_party/sleef/LICENSE.txt +-------------------------------------- +Boost Software License - Version 1.0 - August 17th, 2003 + +Permission is hereby granted, free of charge, to any person or organization +obtaining a copy of the software and accompanying documentation covered by +this license (the "Software") to use, reproduce, display, distribute, +execute, and transmit the Software, and to prepare derivative works of the +Software, and to permit third-parties to whom the Software is furnished to +do so, all subject to the following: + +The copyright notices in the Software and this entire statement, including +the above license grant, this restriction and the following disclaimer, +must be included in all copies of the Software, in whole or in part, and +all derivative works of the Software, unless such copies or derivative +works are solely in the form of machine-executable object code generated by +a source language processor. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT +SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE +FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, +ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. + + +/pytorch/third_party/flatbuffers/swift/LICENSE +---------------------------------------------- + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +/pytorch/third_party/kineto/tb_plugin/LICENSE +--------------------------------------------- +BSD License + +For Kineto software + +Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. + +All contributions by Microsoft: +Copyright (c) Microsoft Corporation. (The Azure AI Platform team) + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + * Neither the name Facebook nor the names of its contributors may be used to + endorse or promote products derived from this software without specific + prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/tensorflow-common/LICENSE.txt +-------------------------------------------------------------------------------------- +Copyright (c) Microsoft Corporation + +All rights reserved. + +MIT License + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +/pytorch/third_party/tensorpipe/LICENSE.txt +------------------------------------------- +BSD License + +For TensorPipe software + +Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + * Neither the name Meta nor the names of its contributors may be used to + endorse or promote products derived from this software without specific + prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr/test/LICENSE +-------------------------------------------------------------------------------------- +This license applies to everything inside this directory and all +subdirectories. + + GNU GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + Copyright (C) + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, your program's commands +might be different; for a GUI interface, you would use an "about box". + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU GPL, see +. + + The GNU General Public License does not permit incorporating your program +into proprietary programs. If your program is a subroutine library, you +may consider it more useful to permit linking proprietary applications with +the library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. But first, please read +. + +/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp/3rd_party/include/opentracing/variant/LICENSE +---------------------------------------------------------------------------------------------------------------- +Copyright (c) MapBox +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + +- Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. +- Redistributions in binary form must reproduce the above copyright notice, this + list of conditions and the following disclaimer in the documentation and/or + other materials provided with the distribution. +- Neither the name "MapBox" nor the names of its contributors may be + used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +/pytorch/third_party/opentelemetry-cpp/tools/vcpkg/LICENSE.txt +-------------------------------------------------------------- +MIT License + +Copyright (c) Microsoft Corporation + +Permission is hereby granted, free of charge, to any person obtaining a copy of this +software and associated documentation files (the "Software"), to deal in the Software +without restriction, including without limitation the rights to use, copy, modify, +merge, publish, distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice shall be included in all copies +or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, +INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF +CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE +OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +/pytorch/third_party/opentelemetry-cpp/tools/vcpkg/ports/vulkan/LICENSE.txt +--------------------------------------------------------------------------- +/* +* +* Licensed under the Apache License, Version 2.0 (the "License"); +* you may not use this file except in compliance with the License. +* You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +*/ + + +Apache License +Version 2.0, January 2004 +http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + +"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. + +"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. + +"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. + +"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. + +"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. + +"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. + +"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). + +"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. + +"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." + +"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. + +2. Grant of Copyright License. + +Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. + +3. Grant of Patent License. + +Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. + +4. Redistribution. + +You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: + +You must give any other recipients of the Work or Derivative Works a copy of this License; and +You must cause any modified files to carry prominent notices stating that You changed the files; and +You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and +If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. +You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. + +5. Submission of Contributions. + +Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. + +6. Trademarks. + +This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. + +Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. + +8. Limitation of Liability. + +In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. + +While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. + +END OF TERMS AND CONDITIONS + +=============================================================================================================================================== + +/Copyright (C) 2012 LunarG, Inc. +//All rights reserved. +// +//Redistribution and use in source and binary forms, with or without +//modification, are permitted provided that the following conditions +//are met: +// +// Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// +// Redistributions in binary form must reproduce the above +// copyright notice, this list of conditions and the following +// disclaimer in the documentation and/or other materials provided +// with the distribution. +// +// Neither the name of LunarG Inc. nor the names of its +// contributors may be used to endorse or promote products derived +// from this software without specific prior written permission. +// +//THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +//"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +//LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS +//FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE +//COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +//INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, +//BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +//LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +//CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +//LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN +//ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +//POSSIBILITY OF SUCH DAMAGE. + +=============================================================================================================================================== + +#============================================================================= +# Copyright 2007-2009 Kitware, Inc. +# Copyright 2007-2008 Miguel A. Figueroa-Villanueva +# +# Distributed under the OSI-approved BSD License (the "License"); +# see accompanying file Copyright_cmake.txt for details. +# +# This software is distributed WITHOUT ANY WARRANTY; without even the +# implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. +# See the License for more information. +#============================================================================= +# (To distributed this file outside of CMake, substitute the full +# License text for the above reference.) + + +============================================================================================================================================== + +// +// Copyright (C) 2015-2018 Google, Inc. +// Copyright (C) +// +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions +// are met: +// +// Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// +// Redistributions in binary form must reproduce the above +// copyright notice, this list of conditions and the following +// disclaimer in the documentation and/or other materials provided +// with the distribution. +// +// Neither the name of 3Dlabs Inc. Ltd. nor the names of its +// contributors may be used to endorse or promote products derived +// from this software without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS +// FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE +// COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +// INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, +// BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +// LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +// CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN +// ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +// POSSIBILITY OF SUCH DAMAGE. +// + +========================================================================================================================================== + +Note: This license has also been called the "New BSD License" or "Modified BSD License". See also the 2-clause BSD License. +Copyright +Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: +1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. +3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +========================================================================================================================================== + +/* +* xxHash - Fast Hash algorithm +* Copyright (C) 2012-2016, Yann Collet +* +* BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) +* +* Redistribution and use in source and binary forms, with or without +* modification, are permitted provided that the following conditions are +* met: +* +* * Redistributions of source code must retain the above copyright +* notice, this list of conditions and the following disclaimer. +* * Redistributions in binary form must reproduce the above +* copyright notice, this list of conditions and the following disclaimer +* in the documentation and/or other materials provided with the +* distribution. +* +* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +* +* You can contact the author at : +* - xxHash homepage: http://www.xxhash.com +* - xxHash source repository : https://github.com/Cyan4973/xxHash +*/ + + +=========================================================================================================================================== + +# Copyright (C) 2018 Google, Inc. +# +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following +# disclaimer in the documentation and/or other materials provided +# with the distribution. +# +# Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS +# FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE +# COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, +# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN +# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +========================================================================================================================================== + +/* A Bison parser, made by GNU Bison 3.0.4. */ + +/* Bison implementation for Yacc-like parsers in C +Copyright (C) 1984, 1989-1990, 2000-2015 Free Software Foundation, Inc. +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. +You should have received a copy of the GNU General Public License +along with this program. If not, see . */ + +/* As a special exception, you may create a larger work that contains +part or all of the Bison parser skeleton and distribute that work +under terms of your choice, so long as that work isn't itself a +parser generator using the skeleton or a modified version thereof +as a parser skeleton. Alternatively, if you modify or redistribute +the parser skeleton itself, you may (at your option) remove this +special exception, which will cause the skeleton and the resulting +Bison output files to be licensed under the GNU General Public +License without this special exception. +This special exception was added by the Free Software Foundation in +version 2.2 of Bison. */ + +/* C LALR(1) parser skeleton written by Richard Stallman, by +simplifying the original so-called "semantic" parser. */ + +/* All symbols defined below should begin with yy or YY, to avoid +infringing on user name space. This should be done even for local +variables, as they might otherwise be expanded by user macros. +There are some unavoidable exceptions within include files to +define necessary library symbols; they are noted "INFRINGES ON +USER NAME SPACE" below. */ + +============================================================================================================================================== + +copyright : [ +Copyright (c) 2017 The Khronos Group Inc., +, +Permission is hereby granted, free of charge, to any person obtaining a copy, +of this software and/or associated documentation files (the \Materials\"),", +to deal in the Materials without restriction, including without limitation, +the rights to use, copy, modify, merge, publish, distribute, sublicense,, +and/or sell copies of the Materials, and to permit persons to whom the, +Materials are furnished to do so, subject to the following conditions:, +, +The above copyright notice and this permission notice shall be included in, +all copies or substantial portions of the Materials., +, +MODIFICATIONS TO THIS FILE MAY MEAN IT NO LONGER ACCURATELY REFLECTS KHRONOS, +STANDARDS. THE UNMODIFIED, NORMATIVE VERSIONS OF KHRONOS SPECIFICATIONS AND, +HEADER INFORMATION ARE LOCATED AT https://www.khronos.org/registry/ , +, +THE MATERIALS ARE PROVIDED \AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS", +OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL, +THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER, +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING, +FROM,OUT OF OR IN CONNECTION WITH THE MATERIALS OR THE USE OR OTHER DEALINGS, +IN THE MATERIALS. + +============================================================================================================================================= + +CMake - Cross Platform Makefile Generator +Copyright 2000-2009 Kitware, Inc., Insight Software Consortium +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +* Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +* Neither the names of Kitware, Inc., the Insight Software Consortium, +nor the names of their contributors may be used to endorse or promote +products derived from this software without specific prior written +permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +AS IS AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +------------------------------------------------------------------------------ + +The above copyright and license notice applies to distributions of +CMake in source and binary form. Some source files contain additional +notices of original copyright by their contributors; see each source +for details. Third-party software packages supplied with CMake under +compatible licenses provide their own copyright notices documented in +corresponding subdirectories. + +------------------------------------------------------------------------------ + +CMake was initially developed by Kitware with the following sponsorship: + +* National Library of Medicine at the National Institutes of Health +as part of the Insight Segmentation and Registration Toolkit (ITK). + +* US National Labs (Los Alamos, Livermore, Sandia) ASC Parallel +Visualization Initiative. + +* National Alliance for Medical Image Computing (NAMIC) is funded by the +National Institutes of Health through the NIH Roadmap for Medical Research, +Grant U54 EB005149. + +* Kitware, Inc. + +======================================================================================================================================== + +The authors of this software are Rob Pike and Ken Thompson. +* Copyright (c) 2002 by Lucent Technologies. +* Permission to use, copy, modify, and distribute this software for any +* purpose without fee is hereby granted, provided that this entire notice +* is included in all copies of any software which is or includes a copy +* or modification of this software and in all copies of the supporting +* documentation for such software. +* THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED +* WARRANTY. IN PARTICULAR, NEITHER THE AUTHORS NOR LUCENT TECHNOLOGIES MAKE ANY +* REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE MERCHANTABILITY +* OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR PURPOSE. + + +======================================================================================================================================== + +Copyright (c) 2015-2018 Baldur Karlsson + +Copyright (c) 2014 Crytek + +Copyright (c) 1998-2018 Third party code and tools + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +========================================================================================================================================= + +/* +Copyright (c) 2009 Dave Gamble +Copyright (c) 2015-2016 The Khronos Group Inc. +Copyright (c) 2015-2016 Valve Corporation +Copyright (c) 2015-2016 LunarG, Inc. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. +*/ + +=========================================================================================================================================== + +Copyright (c) 2005 - 2017 G-Truc Creation + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + + +========================================================================================================================================== + +/* +The JsonCpp library's source code, including accompanying documentation, +tests and demonstration applications, are licensed under the following +conditions... +The author (Baptiste Lepilleur) explicitly disclaims copyright in all +jurisdictions which recognize such a disclaimer. In such jurisdictions, +this software is released into the Public Domain. +In jurisdictions which do not recognize Public Domain property (e.g. Germany as of +2010), this software is Copyright (c) 2007-2010 by Baptiste Lepilleur, and is +released under the terms of the MIT License (see below). +In jurisdictions which recognize Public Domain property, the user of this +software may choose to accept it either as 1) Public Domain, 2) under the +conditions of the MIT License (see below), or 3) under the terms of dual +Public Domain/MIT License conditions described here, as they choose. +The MIT License is about as close to Public Domain as a license can get, and is +described in clear, concise terms at: +http://en.wikipedia.org/wiki/MIT_License + +The full text of the MIT License follows: + +Copyright (c) 2007-2010 Baptiste Lepilleur +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated documentation +files (the "Software"), to deal in the Software without +restriction, including without limitation the rights to use, copy, +modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS +BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN +ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +========================================================================================================================================== + +/** +* `murmurhash.h' - murmurhash +* +* copyright (c) 2014 joseph werle +* Copyright (c) 2015-2016 The Khronos Group Inc. +* Copyright (c) 2015-2016 Valve Corporation +* Copyright (c) 2015-2016 LunarG, Inc. +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and/or associated documentation files (the "Materials"), to +* deal in the Materials without restriction, including without limitation the +* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +* sell copies of the Materials, and to permit persons to whom the Materials are +* furnished to do so, subject to the following conditions: +* +* The above copyright notice(s) and this permission notice shall be included in +* all copies or substantial portions of the Materials. +* +* THE MATERIALS ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +* +* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, +* DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR +* OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE MATERIALS OR THE +* USE OR OTHER DEALINGS IN THE MATERIALS. +*/ + +========================================================================================================================================= + +Licenced as X11: http://www.kryogenix.org/code/browser/licence.html +This basically means: do what you want with it. + +========================================================================================================================================= + +/////////////////////////////////////////////////////////////////////////////////// +/// OpenGL Mathematics (glm.g-truc.net) +/// +/// Copyright (c) 2005 - 2014 G-Truc Creation (www.g-truc.net) +/// Permission is hereby granted, free of charge, to any person obtaining a copy +/// of this software and associated documentation files (the "Software"), to deal +/// in the Software without restriction, including without limitation the rights +/// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +/// copies of the Software, and to permit persons to whom the Software is +/// furnished to do so, subject to the following conditions: +/// +/// The above copyright notice and this permission notice shall be included in +/// all copies or substantial portions of the Software. +/// +/// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +/// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +/// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +/// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +/// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +/// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +/// THE SOFTWARE. +/// +/// @ref core +/// @file glm/common.hpp +/// @date 2013-12-24 / 2013-12-24 +/// @author Christophe Riccio +/////////////////////////////////////////////////////////////////////////////////// + + +========================================================================================================================================== + +// LICENSE +// +// This software is in the public domain. Where that dedication is not +// recognized, you are granted a perpetual, irrevocable license to copy, +// distribute, and modify this file as you see fit. +// + +========================================================================================================================================== + +Simple DirectMedia Layer +Copyright (C) 1997-2018 Sam Lantinga + +This software is provided 'as-is', without any express or implied +warranty. In no event will the authors be held liable for any damages +arising from the use of this software. + +Permission is granted to anyone to use this software for any purpose, +including commercial applications, and to alter it and redistribute it +freely, subject to the following restrictions: + +1. The origin of this software must not be misrepresented; you must not +claim that you wrote the original software. If you use this software +in a product, an acknowledgment in the product documentation would be +appreciated but is not required. +2. Altered source versions must be plainly marked as such, and must not be +misrepresented as being the original software. +3. This notice may not be removed or altered from any source distribution. + +========================================================================================================================================= + +/****************************************************************************\ +Copyright (c) 2002, NVIDIA Corporation. + +NVIDIA Corporation("NVIDIA") supplies this software to you in +consideration of your agreement to the following terms, and your use, +installation, modification or redistribution of this NVIDIA software +constitutes acceptance of these terms. If you do not agree with these +terms, please do not use, install, modify or redistribute this NVIDIA +software. + +In consideration of your agreement to abide by the following terms, and +subject to these terms, NVIDIA grants you a personal, non-exclusive +license, under NVIDIA's copyrights in this original NVIDIA software (the +NVIDIA Software), to use, reproduce, modify and redistribute the +NVIDIA Software, with or without modifications, in source and/or binary +forms; provided that if you redistribute the NVIDIA Software, you must +retain the copyright notice of NVIDIA, this notice and the following +text and disclaimers in all such redistributions of the NVIDIA Software. +Neither the name, trademarks, service marks nor logos of NVIDIA +Corporation may be used to endorse or promote products derived from the +NVIDIA Software without specific prior written permission from NVIDIA. +Except as expressly stated in this notice, no other rights or licenses +express or implied, are granted by NVIDIA herein, including but not +limited to any patent rights that may be infringed by your derivative +works or by other works in which the NVIDIA Software may be +incorporated. No hardware is licensed hereunder. + +THE NVIDIA SOFTWARE IS BEING PROVIDED ON AN "AS IS" BASIS, WITHOUT +WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, +INCLUDING WITHOUT LIMITATION, WARRANTIES OR CONDITIONS OF TITLE, +NON-INFRINGEMENT, MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR +ITS USE AND OPERATION EITHER ALONE OR IN COMBINATION WITH OTHER +PRODUCTS. + +IN NO EVENT SHALL NVIDIA BE LIABLE FOR ANY SPECIAL, INDIRECT, +INCIDENTAL, EXEMPLARY, CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, LOST PROFITS; PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) OR ARISING IN ANY WAY +OUT OF THE USE, REPRODUCTION, MODIFICATION AND/OR DISTRIBUTION OF THE +NVIDIA SOFTWARE, HOWEVER CAUSED AND WHETHER UNDER THEORY OF CONTRACT, +TORT (INCLUDING NEGLIGENCE), STRICT LIABILITY OR OTHERWISE, EVEN IF +NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +\****************************************************************************/ + +================================================================================================================================================== + +This software is provided 'as-is', without any express or implied +warranty. In no event will the authors be held liable for any damages +arising from the use of this software. + +Permission is granted to anyone to use this software for any purpose, +including commercial applications, and to alter it and redistribute it +freely, subject to the following restrictions: + +1. The origin of this software must not be misrepresented; you must not + claim that you wrote the original software. If you use this software + in a product, an acknowledgment in the product documentation would be + appreciated but is not required. +2. Altered source versions must be plainly marked as such, and must not be + misrepresented as being the original software. +3. This notice may not be removed or altered from any source distribution. + + +================================================================================================================================================== + +GNU LESSER GENERAL PUBLIC LICENSE +Version 3, 29 June 2007 + +Copyright (C) 2007 Free Software Foundation, Inc. + +Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. + +This version of the GNU Lesser General Public License incorporates the terms and conditions of version 3 of the GNU General Public License, supplemented by the additional permissions listed below. + +0. Additional Definitions. + +As used herein, "this License" refers to version 3 of the GNU Lesser General Public License, and the "GNU GPL" refers to version 3 of the GNU General Public License. + +"The Library" refers to a covered work governed by this License, other than an Application or a Combined Work as defined below. + +An "Application" is any work that makes use of an interface provided by the Library, but which is not otherwise based on the Library. Defining a subclass of a class defined by the Library is deemed a mode of using an interface provided by the Library. + +A "Combined Work" is a work produced by combining or linking an Application with the Library. The particular version of the Library with which the Combined Work was made is also called the "Linked Version". + +The "Minimal Corresponding Source" for a Combined Work means the Corresponding Source for the Combined Work, excluding any source code for portions of the Combined Work that, considered in isolation, are based on the Application, and not on the Linked Version. + +The "Corresponding Application Code" for a Combined Work means the object code and/or source code for the Application, including any data and utility programs needed for reproducing the Combined Work from the Application, but excluding the System Libraries of the Combined Work. + +1. Exception to Section 3 of the GNU GPL. + +You may convey a covered work under sections 3 and 4 of this License without being bound by section 3 of the GNU GPL. + +2. Conveying Modified Versions. + +If you modify a copy of the Library, and, in your modifications, a facility refers to a function or data to be supplied by an Application that uses the facility (other than as an argument passed when the facility is invoked), then you may convey a copy of the modified version: + +a) under this License, provided that you make a good faith effort to ensure that, in the event an Application does not supply the function or data, the facility still operates, and performs whatever part of its purpose remains meaningful, or +b) under the GNU GPL, with none of the additional permissions of this License applicable to that copy. +3. Object Code Incorporating Material from Library Header Files. + +The object code form of an Application may incorporate material from a header file that is part of the Library. You may convey such object code under terms of your choice, provided that, if the incorporated material is not limited to numerical parameters, data structure layouts and accessors, or small macros, inline functions and templates (ten or fewer lines in length), you do both of the following: + +a) Give prominent notice with each copy of the object code that the Library is used in it and that the Library and its use are covered by this License. +b) Accompany the object code with a copy of the GNU GPL and this license document. +4. Combined Works. + +You may convey a Combined Work under terms of your choice that, taken together, effectively do not restrict modification of the portions of the Library contained in the Combined Work and reverse engineering for debugging such modifications, if you also do each of the following: + +a) Give prominent notice with each copy of the Combined Work that the Library is used in it and that the Library and its use are covered by this License. +b) Accompany the Combined Work with a copy of the GNU GPL and this license document. +c) For a Combined Work that displays copyright notices during execution, include the copyright notice for the Library among these notices, as well as a reference directing the user to the copies of the GNU GPL and this license document. +d) Do one of the following: +0) Convey the Minimal Corresponding Source under the terms of this License, and the Corresponding Application Code in a form suitable for, and under terms that permit, the user to recombine or relink the Application with a modified version of the Linked Version to produce a modified Combined Work, in the manner specified by section 6 of the GNU GPL for conveying Corresponding Source. +1) Use a suitable shared library mechanism for linking with the Library. A suitable mechanism is one that (a) uses at run time a copy of the Library already present on the user's computer system, and (b) will operate properly with a modified version of the Library that is interface-compatible with the Linked Version. +e) Provide Installation Information, but only if you would otherwise be required to provide such information under section 6 of the GNU GPL, and only to the extent that such information is necessary to install and execute a modified version of the Combined Work produced by recombining or relinking the Application with a modified version of the Linked Version. (If you use option 4d0, the Installation Information must accompany the Minimal Corresponding Source and Corresponding Application Code. If you use option 4d1, you must provide the Installation Information in the manner specified by section 6 of the GNU GPL for conveying Corresponding Source.) +5. Combined Libraries. + +You may place library facilities that are a work based on the Library side by side in a single library together with other library facilities that are not Applications and are not covered by this License, and convey such a combined library under terms of your choice, if you do both of the following: + +a) Accompany the combined library with a copy of the same work based on the Library, uncombined with any other library facilities, conveyed under the terms of this License. +b) Give prominent notice with the combined library that part of it is a work based on the Library, and explaining where to find the accompanying uncombined form of the same work. +6. Revised Versions of the GNU Lesser General Public License. + +The Free Software Foundation may publish revised and/or new versions of the GNU Lesser General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Library as you received it specifies that a certain numbered version of the GNU Lesser General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that published version or of any later version published by the Free Software Foundation. If the Library as you received it does not specify a version number of the GNU Lesser General Public License, you may choose any version of the GNU Lesser General Public License ever published by the Free Software Foundation. + +If the Library as you received it specifies that a proxy can decide whether future versions of the GNU Lesser General Public License shall apply, that proxy's public statement of acceptance of any version is permanent authorization for you to choose that version for the Library. + + +torch-optimizer +Apache Software License +https://github.com/jettify/pytorch-optimizer +Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2020 Nikolay Novik (https://github.com/jettify) + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +torch_fidelity +Apache License 2.0 +https://www.github.com/toshas/torch-fidelity +Copyright 2020 Anton Obukhov + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + + +torchcodec +BSD 3-Clause License + +Copyright 2024 Meta + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice,this list +of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, this +list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its contributors may +be used to endorse or promote products derived from this software without specific +prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY +EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES +OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT +SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR +BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN +ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH +DAMAGE. + +UNKNOWN +BSD 3-Clause License + +Copyright 2024 Meta + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice,this list +of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, this +list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its contributors may +be used to endorse or promote products derived from this software without specific +prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY +EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES +OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT +SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR +BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN +ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH +DAMAGE. + + +torchdata +BSD License +https://github.com/pytorch/data +BSD 3-Clause License + +Copyright (c) 2021-present, Facebook, Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +torchmetrics +Apache Software License +https://github.com/Lightning-AI/torchmetrics + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2020-2022 Lightning-AI team + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +torchtitan +BSD 3-Clause License + +(c) Meta Platforms, Inc. and affiliates. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice,this list +of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, this +list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its contributors may +be used to endorse or promote products derived from this software without specific +prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY +EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES +OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT +SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR +BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN +ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH +DAMAGE. + +UNKNOWN +BSD 3-Clause License + +(c) Meta Platforms, Inc. and affiliates. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice,this list +of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, this +list of conditions and the following disclaimer in the documentation +and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its contributors may +be used to endorse or promote products derived from this software without specific +prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY +EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES +OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT +SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR +BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN +ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH +DAMAGE. + + +torchvision +BSD +https://github.com/pytorch/vision +BSD 3-Clause License + +Copyright (c) Soumith Chintala 2016, +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +tornado +Apache Software License +http://www.tornadoweb.org/ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +tqdm +MPL-2.0 AND MIT +https://tqdm.github.io +`tqdm` is a product of collaborative work. +Unless otherwise stated, all authors (see commit logs) retain copyright +for their respective work, and release the work under the MIT licence +(text below). + +Exceptions or notable authors are listed below +in reverse chronological order: + +* files: * + MPL-2.0 2015-2026 (c) Casper da Costa-Luis + [casperdcl](https://github.com/casperdcl). +* files: tqdm/_tqdm.py + MIT 2016 (c) [PR #96] on behalf of Google Inc. +* files: tqdm/_tqdm.py README.rst .gitignore + MIT 2013 (c) Noam Yorav-Raphael, original author. + +[PR #96]: https://github.com/tqdm/tqdm/pull/96 + + +Mozilla Public Licence (MPL) v. 2.0 - Exhibit A +----------------------------------------------- + +This Source Code Form is subject to the terms of the +Mozilla Public License, v. 2.0. +If a copy of the MPL was not distributed with this project, +You can obtain one at https://mozilla.org/MPL/2.0/. + + +MIT License (MIT) +----------------- + +Copyright (c) 2013 noamraph + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +traitlets +BSD License +https://github.com/ipython/traitlets +BSD 3-Clause License + +- Copyright (c) 2001-, IPython Development Team + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +transformer_engine +UNKNOWN +UNKNOWN + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + + +transformers +Apache Software License +https://github.com/huggingface/transformers +Copyright 2018- The Hugging Face team. All rights reserved. + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +trimesh +MIT License +https://github.com/mikedh/trimesh +The MIT License (MIT) + +Copyright (c) 2023 Michael Dawson-Haggerty + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +triton +MIT License +https://github.com/triton-lang/triton/ +/* +* Copyright 2018-2020 Philippe Tillet +* Copyright 2020-2022 OpenAI +* +* Permission is hereby granted, free of charge, to any person obtaining +* a copy of this software and associated documentation files +* (the "Software"), to deal in the Software without restriction, +* including without limitation the rights to use, copy, modify, merge, +* publish, distribute, sublicense, and/or sell copies of the Software, +* and to permit persons to whom the Software is furnished to do so, +* subject to the following conditions: +* +* The above copyright notice and this permission notice shall be +* included in all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +* CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, +* TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +*/ + + +trove-classifiers +Apache Software License +https://github.com/pypa/trove-classifiers + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +typeguard +MIT +UNKNOWN +This is the MIT license: http://www.opensource.org/licenses/mit-license.php + +Copyright (c) Alex Grönholm + +Permission is hereby granted, free of charge, to any person obtaining a copy of this +software and associated documentation files (the "Software"), to deal in the Software +without restriction, including without limitation the rights to use, copy, modify, merge, +publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons +to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or +substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, +INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR +PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE +FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR +OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. + + +typer +MIT License +https://github.com/fastapi/typer +The MIT License (MIT) + +Copyright (c) 2019 Sebastián Ramírez + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +typing-inspect +MIT License +https://github.com/ilevkivskyi/typing_inspect +The MIT License (MIT) + +Copyright (c) 2017-2019 Ivan Levkivskyi + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +typing-inspection +MIT +https://github.com/pydantic/typing-inspection +MIT License + +Copyright (c) Pydantic Services Inc. 2025 to present + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +typing_extensions +PSF-2.0 +https://github.com/python/typing_extensions +A. HISTORY OF THE SOFTWARE +========================== + +Python was created in the early 1990s by Guido van Rossum at Stichting +Mathematisch Centrum (CWI, see https://www.cwi.nl) in the Netherlands +as a successor of a language called ABC. Guido remains Python's +principal author, although it includes many contributions from others. + +In 1995, Guido continued his work on Python at the Corporation for +National Research Initiatives (CNRI, see https://www.cnri.reston.va.us) +in Reston, Virginia where he released several versions of the +software. + +In May 2000, Guido and the Python core development team moved to +BeOpen.com to form the BeOpen PythonLabs team. In October of the same +year, the PythonLabs team moved to Digital Creations, which became +Zope Corporation. In 2001, the Python Software Foundation (PSF, see +https://www.python.org/psf/) was formed, a non-profit organization +created specifically to own Python-related Intellectual Property. +Zope Corporation was a sponsoring member of the PSF. + +All Python releases are Open Source (see https://opensource.org for +the Open Source Definition). Historically, most, but not all, Python +releases have also been GPL-compatible; the table below summarizes +the various releases. + + Release Derived Year Owner GPL- + from compatible? (1) + + 0.9.0 thru 1.2 1991-1995 CWI yes + 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes + 1.6 1.5.2 2000 CNRI no + 2.0 1.6 2000 BeOpen.com no + 1.6.1 1.6 2001 CNRI yes (2) + 2.1 2.0+1.6.1 2001 PSF no + 2.0.1 2.0+1.6.1 2001 PSF yes + 2.1.1 2.1+2.0.1 2001 PSF yes + 2.1.2 2.1.1 2002 PSF yes + 2.1.3 2.1.2 2002 PSF yes + 2.2 and above 2.1.1 2001-now PSF yes + +Footnotes: + +(1) GPL-compatible doesn't mean that we're distributing Python under + the GPL. All Python licenses, unlike the GPL, let you distribute + a modified version without making your changes open source. The + GPL-compatible licenses make it possible to combine Python with + other software that is released under the GPL; the others don't. + +(2) According to Richard Stallman, 1.6.1 is not GPL-compatible, + because its license has a choice of law clause. According to + CNRI, however, Stallman's lawyer has told CNRI's lawyer that 1.6.1 + is "not incompatible" with the GPL. + +Thanks to the many outside volunteers who have worked under Guido's +direction to make these releases possible. + + +B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON +=============================================================== + +Python software and documentation are licensed under the +Python Software Foundation License Version 2. + +Starting with Python 3.8.6, examples, recipes, and other code in +the documentation are dual licensed under the PSF License Version 2 +and the Zero-Clause BSD license. + +Some software incorporated into Python is under different licenses. +The licenses are listed with code falling under that license. + + +PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 +-------------------------------------------- + +1. This LICENSE AGREEMENT is between the Python Software Foundation +("PSF"), and the Individual or Organization ("Licensee") accessing and +otherwise using this software ("Python") in source or binary form and +its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, PSF hereby +grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, +analyze, test, perform and/or display publicly, prepare derivative works, +distribute, and otherwise use Python alone or in any derivative version, +provided, however, that PSF's License Agreement and PSF's notice of copyright, +i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, +2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023 Python Software Foundation; +All Rights Reserved" are retained in Python alone or in any derivative version +prepared by Licensee. + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python. + +4. PSF is making Python available to Licensee on an "AS IS" +basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. Nothing in this License Agreement shall be deemed to create any +relationship of agency, partnership, or joint venture between PSF and +Licensee. This License Agreement does not grant permission to use PSF +trademarks or trade name in a trademark sense to endorse or promote +products or services of Licensee, or any third party. + +8. By copying, installing or otherwise using Python, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0 +------------------------------------------- + +BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1 + +1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an +office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the +Individual or Organization ("Licensee") accessing and otherwise using +this software in source or binary form and its associated +documentation ("the Software"). + +2. Subject to the terms and conditions of this BeOpen Python License +Agreement, BeOpen hereby grants Licensee a non-exclusive, +royalty-free, world-wide license to reproduce, analyze, test, perform +and/or display publicly, prepare derivative works, distribute, and +otherwise use the Software alone or in any derivative version, +provided, however, that the BeOpen Python License is retained in the +Software, alone or in any derivative version prepared by Licensee. + +3. BeOpen is making the Software available to Licensee on an "AS IS" +basis. BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE +SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS +AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY +DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +5. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +6. This License Agreement shall be governed by and interpreted in all +respects by the law of the State of California, excluding conflict of +law provisions. Nothing in this License Agreement shall be deemed to +create any relationship of agency, partnership, or joint venture +between BeOpen and Licensee. This License Agreement does not grant +permission to use BeOpen trademarks or trade names in a trademark +sense to endorse or promote products or services of Licensee, or any +third party. As an exception, the "BeOpen Python" logos available at +http://www.pythonlabs.com/logos.html may be used according to the +permissions granted on that web page. + +7. By copying, installing or otherwise using the software, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1 +--------------------------------------- + +1. This LICENSE AGREEMENT is between the Corporation for National +Research Initiatives, having an office at 1895 Preston White Drive, +Reston, VA 20191 ("CNRI"), and the Individual or Organization +("Licensee") accessing and otherwise using Python 1.6.1 software in +source or binary form and its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, CNRI +hereby grants Licensee a nonexclusive, royalty-free, world-wide +license to reproduce, analyze, test, perform and/or display publicly, +prepare derivative works, distribute, and otherwise use Python 1.6.1 +alone or in any derivative version, provided, however, that CNRI's +License Agreement and CNRI's notice of copyright, i.e., "Copyright (c) +1995-2001 Corporation for National Research Initiatives; All Rights +Reserved" are retained in Python 1.6.1 alone or in any derivative +version prepared by Licensee. Alternately, in lieu of CNRI's License +Agreement, Licensee may substitute the following text (omitting the +quotes): "Python 1.6.1 is made available subject to the terms and +conditions in CNRI's License Agreement. This Agreement together with +Python 1.6.1 may be located on the internet using the following +unique, persistent identifier (known as a handle): 1895.22/1013. This +Agreement may also be obtained from a proxy server on the internet +using the following URL: http://hdl.handle.net/1895.22/1013". + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python 1.6.1 or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python 1.6.1. + +4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS" +basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1 WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. This License Agreement shall be governed by the federal +intellectual property law of the United States, including without +limitation the federal copyright law, and, to the extent such +U.S. federal law does not apply, by the law of the Commonwealth of +Virginia, excluding Virginia's conflict of law provisions. +Notwithstanding the foregoing, with regard to derivative works based +on Python 1.6.1 that incorporate non-separable material that was +previously distributed under the GNU General Public License (GPL), the +law of the Commonwealth of Virginia shall govern this License +Agreement only as to issues arising under or with respect to +Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this +License Agreement shall be deemed to create any relationship of +agency, partnership, or joint venture between CNRI and Licensee. This +License Agreement does not grant permission to use CNRI trademarks or +trade name in a trademark sense to endorse or promote products or +services of Licensee, or any third party. + +8. By clicking on the "ACCEPT" button where indicated, or by copying, +installing or otherwise using Python 1.6.1, Licensee agrees to be +bound by the terms and conditions of this License Agreement. + + ACCEPT + + +CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2 +-------------------------------------------------- + +Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, +The Netherlands. All rights reserved. + +Permission to use, copy, modify, and distribute this software and its +documentation for any purpose and without fee is hereby granted, +provided that the above copyright notice appear in all copies and that +both that copyright notice and this permission notice appear in +supporting documentation, and that the name of Stichting Mathematisch +Centrum or CWI not be used in advertising or publicity pertaining to +distribution of the software without specific, written prior +permission. + +STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO +THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND +FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE +FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT +OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + +ZERO-CLAUSE BSD LICENSE FOR CODE IN THE PYTHON DOCUMENTATION +---------------------------------------------------------------------- + +Permission to use, copy, modify, and/or distribute this software for any +purpose with or without fee is hereby granted. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH +REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY +AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, +INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM +LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR +OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR +PERFORMANCE OF THIS SOFTWARE. + + +tyro +MIT License +UNKNOWN +MIT License + +Copyright (c) 2024 Brent Yi + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +tzdata +Apache-2.0 +https://github.com/python/tzdata +Apache Software License 2.0 + +Copyright (c) 2020, Paul Ganssle (Google) + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + + +uri-template +MIT License +https://gitlab.linss.com/open-source/python/uri-template +MIT License + +Copyright (c) 2020 Peter Linss + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +urllib3 +MIT +https://github.com/urllib3/urllib3/blob/main/CHANGES.rst +MIT License + +Copyright (c) 2008-2020 Andrey Petrov and contributors. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +userpath +MIT +https://github.com/ofek/userpath +MIT License + +Copyright (c) 2017-present Ofek Lev + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +uv +Apache Software License; MIT License +https://pypi.org/project/uv/ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +uvicorn +BSD-3-Clause +https://uvicorn.dev/ +Copyright © 2017-present, [Encode OSS Ltd](https://www.encode.io/). +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +uvloop +Apache Software License; MIT License +UNKNOWN +Copyright (C) 2016-present the uvloop authors and contributors. + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright (c) 2015-present MagicStack Inc. http://magic.io + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +virtualenv +MIT +https://github.com/pypa/virtualenv +Copyright (c) 2020-202x The virtualenv developers + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +wandb +MIT License +https://github.com/wandb/wandb +MIT License + +Copyright (c) 2021 Weights and Biases, Inc. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +watchfiles +MIT License +https://github.com/samuelcolvin/watchfiles +The MIT License (MIT) + +Copyright (c) 2017 to present Samuel Colvin + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +wcmatch +MIT +https://github.com/facelessuser/wcmatch +MIT License + +Copyright (c) 2018 - 2025 Isaac Muse + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +webcolors +BSD License +UNKNOWN +Copyright (c) James Bennett, and contributors. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided + with the distribution. + * Neither the name of the author nor the names of other + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +webdataset +BSD-3-Clause +http://github.com/webdataset/webdataset +Copyright 2020 NVIDIA CORPORATION. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +1. Redistributions of source code must retain the above copyright notice, +this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +webencodings +BSD License +https://github.com/SimonSapin/python-webencodings +UNKNOWN + +websocket-client +Apache Software License +https://github.com/websocket-client/websocket-client.git + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2025 engn33r + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +websockets +BSD-3-Clause +https://github.com/python-websockets/websockets +Copyright (c) Aymeric Augustin and contributors + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + * Neither the name of the copyright holder nor the names of its contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +widgetsnbextension +BSD License +http://jupyter.org +Copyright (c) 2015 Project Jupyter Contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +wrapt +BSD License +https://github.com/GrahamDumpleton/wrapt +Copyright (c) 2013-2023, Graham Dumpleton +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + + +xatlas +MIT License + + Copyright (c) 2021 Markus Worchel + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + +https://github.com/mworchel/xatlas-python +MIT License + +Copyright (c) 2021 Markus Worchel + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +xattr +MIT +https://github.com/xattr/xattr +This is the MIT license. This software may also be distributed under the same terms as Python (the PSF license). + +Copyright (c) 2004 Bob Ippolito. + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +xmltodict +MIT +https://github.com/martinblech/xmltodict +Copyright (C) 2012 Martin Blech and individual contributors. + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +xxhash +BSD License +https://github.com/ifduyue/python-xxhash +Copyright (c) 2014-2024, Yue Du +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +yacs +Apache Software License +https://github.com/rbgirshick/yacs +Apache License +Version 2.0, January 2004 +http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + +"License" shall mean the terms and conditions for use, reproduction, +and distribution as defined by Sections 1 through 9 of this document. + +"Licensor" shall mean the copyright owner or entity authorized by +the copyright owner that is granting the License. + +"Legal Entity" shall mean the union of the acting entity and all +other entities that control, are controlled by, or are under common +control with that entity. For the purposes of this definition, +"control" means (i) the power, direct or indirect, to cause the +direction or management of such entity, whether by contract or +otherwise, or (ii) ownership of fifty percent (50%) or more of the +outstanding shares, or (iii) beneficial ownership of such entity. + +"You" (or "Your") shall mean an individual or Legal Entity +exercising permissions granted by this License. + +"Source" form shall mean the preferred form for making modifications, +including but not limited to software source code, documentation +source, and configuration files. + +"Object" form shall mean any form resulting from mechanical +transformation or translation of a Source form, including but +not limited to compiled object code, generated documentation, +and conversions to other media types. + +"Work" shall mean the work of authorship, whether in Source or +Object form, made available under the License, as indicated by a +copyright notice that is included in or attached to the work +(an example is provided in the Appendix below). + +"Derivative Works" shall mean any work, whether in Source or Object +form, that is based on (or derived from) the Work and for which the +editorial revisions, annotations, elaborations, or other modifications +represent, as a whole, an original work of authorship. For the purposes +of this License, Derivative Works shall not include works that remain +separable from, or merely link (or bind by name) to the interfaces of, +the Work and Derivative Works thereof. + +"Contribution" shall mean any work of authorship, including +the original version of the Work and any modifications or additions +to that Work or Derivative Works thereof, that is intentionally +submitted to Licensor for inclusion in the Work by the copyright owner +or by an individual or Legal Entity authorized to submit on behalf of +the copyright owner. For the purposes of this definition, "submitted" +means any form of electronic, verbal, or written communication sent +to the Licensor or its representatives, including but not limited to +communication on electronic mailing lists, source code control systems, +and issue tracking systems that are managed by, or on behalf of, the +Licensor for the purpose of discussing and improving the Work, but +excluding communication that is conspicuously marked or otherwise +designated in writing by the copyright owner as "Not a Contribution." + +"Contributor" shall mean Licensor and any individual or Legal Entity +on behalf of whom a Contribution has been received by Licensor and +subsequently incorporated within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of +this License, each Contributor hereby grants to You a perpetual, +worldwide, non-exclusive, no-charge, royalty-free, irrevocable +copyright license to reproduce, prepare Derivative Works of, +publicly display, publicly perform, sublicense, and distribute the +Work and such Derivative Works in Source or Object form. + +3. Grant of Patent License. Subject to the terms and conditions of +this License, each Contributor hereby grants to You a perpetual, +worldwide, non-exclusive, no-charge, royalty-free, irrevocable +(except as stated in this section) patent license to make, have made, +use, offer to sell, sell, import, and otherwise transfer the Work, +where such license applies only to those patent claims licensable +by such Contributor that are necessarily infringed by their +Contribution(s) alone or by combination of their Contribution(s) +with the Work to which such Contribution(s) was submitted. If You +institute patent litigation against any entity (including a +cross-claim or counterclaim in a lawsuit) alleging that the Work +or a Contribution incorporated within the Work constitutes direct +or contributory patent infringement, then any patent licenses +granted to You under this License for that Work shall terminate +as of the date such litigation is filed. + +4. Redistribution. You may reproduce and distribute copies of the +Work or Derivative Works thereof in any medium, with or without +modifications, and in Source or Object form, provided that You +meet the following conditions: + +(a) You must give any other recipients of the Work or +Derivative Works a copy of this License; and + +(b) You must cause any modified files to carry prominent notices +stating that You changed the files; and + +(c) You must retain, in the Source form of any Derivative Works +that You distribute, all copyright, patent, trademark, and +attribution notices from the Source form of the Work, +excluding those notices that do not pertain to any part of +the Derivative Works; and + +(d) If the Work includes a "NOTICE" text file as part of its +distribution, then any Derivative Works that You distribute must +include a readable copy of the attribution notices contained +within such NOTICE file, excluding those notices that do not +pertain to any part of the Derivative Works, in at least one +of the following places: within a NOTICE text file distributed +as part of the Derivative Works; within the Source form or +documentation, if provided along with the Derivative Works; or, +within a display generated by the Derivative Works, if and +wherever such third-party notices normally appear. The contents +of the NOTICE file are for informational purposes only and +do not modify the License. You may add Your own attribution +notices within Derivative Works that You distribute, alongside +or as an addendum to the NOTICE text from the Work, provided +that such additional attribution notices cannot be construed +as modifying the License. + +You may add Your own copyright statement to Your modifications and +may provide additional or different license terms and conditions +for use, reproduction, or distribution of Your modifications, or +for any such Derivative Works as a whole, provided Your use, +reproduction, and distribution of the Work otherwise complies with +the conditions stated in this License. + +5. Submission of Contributions. Unless You explicitly state otherwise, +any Contribution intentionally submitted for inclusion in the Work +by You to the Licensor shall be under the terms and conditions of +this License, without any additional terms or conditions. +Notwithstanding the above, nothing herein shall supersede or modify +the terms of any separate license agreement you may have executed +with Licensor regarding such Contributions. + +6. Trademarks. This License does not grant permission to use the trade +names, trademarks, service marks, or product names of the Licensor, +except as required for reasonable and customary use in describing the +origin of the Work and reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. Unless required by applicable law or +agreed to in writing, Licensor provides the Work (and each +Contributor provides its Contributions) on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +implied, including, without limitation, any warranties or conditions +of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A +PARTICULAR PURPOSE. You are solely responsible for determining the +appropriateness of using or redistributing the Work and assume any +risks associated with Your exercise of permissions under this License. + +8. Limitation of Liability. In no event and under no legal theory, +whether in tort (including negligence), contract, or otherwise, +unless required by applicable law (such as deliberate and grossly +negligent acts) or agreed to in writing, shall any Contributor be +liable to You for damages, including any direct, indirect, special, +incidental, or consequential damages of any character arising as a +result of this License or out of the use or inability to use the +Work (including but not limited to damages for loss of goodwill, +work stoppage, computer failure or malfunction, or any and all +other commercial damages or losses), even if such Contributor +has been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. While redistributing +the Work or Derivative Works thereof, You may choose to offer, +and charge a fee for, acceptance of support, warranty, indemnity, +or other liability obligations and/or rights consistent with this +License. However, in accepting such obligations, You may act only +on Your own behalf and on Your sole responsibility, not on behalf +of any other Contributor, and only if You agree to indemnify, +defend, and hold each Contributor harmless for any liability +incurred by, or claims asserted against, such Contributor by reason +of your accepting any such warranty or additional liability. + +END OF TERMS AND CONDITIONS + +APPENDIX: How to apply the Apache License to your work. + +To apply the Apache License to your work, attach the following +boilerplate notice, with the fields enclosed by brackets "[]" +replaced with your own identifying information. (Don't include +the brackets!) The text should be enclosed in the appropriate +comment syntax for the file format. We also recommend that a +file or class name and description of purpose be included on the +same "printed page" as the copyright notice for easier +identification within third-party archives. + +Copyright [yyyy] [name of copyright owner] + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + + +yarl +Apache-2.0 +https://github.com/aio-libs/yarl + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +zarr +MIT +https://github.com/zarr-developers/zarr-python +The MIT License (MIT) + +Copyright (c) 2015-2025 Zarr Developers + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + +zipp +MIT +https://github.com/jaraco/zipp +MIT License + +Copyright (c) 2025 + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and +associated documentation files (the "Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the +following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial +portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT +LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO +EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE +USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..7f78763 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,33 @@ +# Change log + +## Unreleased + +- New features + +- Breaking changes + +## 1.2.2 (May 14, 2026) + +- New features + - Add [action policy closed-loop evaluation](./docs/action_policy_closed_loop_eval.md). + +## 1.2.1 (May 08, 2026) + +- New features + - Add [action policy post-training (SFT)](./docs/training.md). + +## 1.2.0 (May 05, 2026) + +- New features + - Add action modalities (Forward Dynamics, Inverse Dynamics, Policy) for Cosmos3-Nano model. + - Upgrade Cosmos3-Nano checkpoint to improve T2V, I2V quality. + +## 1.1.1 (May 01, 2026) + +- New features + - Add DCP checkpoint conversion/inference. + +## 1.1.0 (April 29, 2026) + +- New features + - Add [Post-Training (Supervised Fine-Tuning)](./docs/training.md). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..3c8fdc4 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,121 @@ +# Contributing + + + +______________________________________________________________________ + +**Table of Contents** + +- [Setup](#setup) +- [Test](#test) + - [Run Linting and Formatting](#run-linting-and-formatting) + - [Run Tests](#run-tests) + - [Run a Single Test](#run-a-single-test) +- [Code Reviews](#code-reviews) +- [Signing Your Work](#signing-your-work) + +______________________________________________________________________ + + + +We'd love to receive your patches and contributions. Please keep your PRs as draft until such time that you would like us to review them. + +## Setup + +Install system dependencies: + +[just](https://just.systems/man/en/pre-built-binaries.html#pre-built-binaries) + +```shell +uv tool install -U rust-just +``` + +To see all available `just` commands, run + +```shell +just +``` + +## Test + +### Run Linting and Formatting + +```shell +just lint +``` + +This will also run auto-fixes and linting. We recommend that you commit your changes first. + +### Run Tests + +```shell +just test +``` + +Test levels (`--levels`): + +0. Smoke tests. Requires >= 1 GPU. +1. Partial E2E tests. Requires >= 8 GPUs. +2. Full E2E tests. Requires >= 8 GPUs. + +Test outputs are saved to `outputs/pytest/`. To monitor a test, open `console.log`/`debug.log`. + +### Run a Single Test + +```shell +# List tests to get the test name +just test-list +# Run the test +just test-single [--pdb] +``` + +## Code Reviews + +All submissions, including submissions by project members, require review. We use GitHub pull requests for this purpose. Consult +[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more information on using pull requests. + +## Signing Your Work + +- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. + + - Any contribution which contains commits that are not Signed-Off will not be accepted. + +- To sign off on a commit you simply use the `--signoff` (or `-s`) option when committing your changes: + + ```bash + git commit -s -m "Add cool feature." + ``` + + This will append the following to your commit message: + + ```text + Signed-off-by: Your Name + ``` + +- Full text of the DCO: + + ```text + Developer Certificate of Origin + Version 1.1 + + Copyright (C) 2004, 2006 The Linux Foundation and its contributors. + 1 Letterman Drive + Suite D4700 + San Francisco, CA, 94129 + + Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. + ``` + + ```text + Developer's Certificate of Origin 1.1 + + By making a contribution to this project, I certify that: + + (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or + + (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or + + (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. + + (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. + ``` diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000..f5251ae --- /dev/null +++ b/Dockerfile @@ -0,0 +1,60 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# Dockerfile using uv environment. + +ARG CUDA_VERSION=13.0.2 +ARG BASE_IMAGE=nvidia/cuda:${CUDA_VERSION}-cudnn-devel-ubuntu24.04 +FROM ${BASE_IMAGE} + +# Set the DEBIAN_FRONTEND environment variable to avoid interactive prompts during apt operations. +ENV DEBIAN_FRONTEND=noninteractive + +# Install packages +RUN --mount=type=cache,target=/var/cache/apt \ + --mount=type=cache,target=/var/lib/apt \ + apt-get update && \ + apt-get install -y --no-install-recommends \ + curl \ + ffmpeg \ + git \ + git-lfs \ + tree \ + wget + +# Install uv: https://docs.astral.sh/uv/getting-started/installation/ +# https://github.com/astral-sh/uv-docker-example/blob/main/Dockerfile +COPY --from=ghcr.io/astral-sh/uv:0.10.8 /uv /uvx /usr/local/bin/ +# Copy from the cache instead of linking since it's a mounted volume +ENV UV_LINK_MODE=copy +# Cache python downloads +ENV UV_PYTHON_CACHE_DIR=/root/.cache/uv/python + +# Install just: https://just.systems/man/en/pre-built-binaries.html +RUN curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin --tag 1.46.0 + +ENV PATH="/root/.local/bin:$PATH" + +WORKDIR /workspace + +# Install python +RUN --mount=type=cache,target=/root/.cache/uv \ + --mount=type=bind,source=.python-version,target=.python-version \ + uv python install + +# Install into virtual environment +RUN echo "$CUDA_VERSION" | sed -E 's/^([0-9]+)\.([0-9]+).*/cu\1\2/' > /root/.cuda-name +RUN --mount=type=cache,target=/root/.cache/uv \ + --mount=type=bind,source=uv.lock,target=uv.lock \ + --mount=type=bind,source=pyproject.toml,target=pyproject.toml \ + --mount=type=bind,source=.python-version,target=.python-version \ + --mount=type=bind,source=packages,target=packages \ + uv sync --locked --no-install-project --no-editable --all-extras --group=$(cat /root/.cuda-name) +ENV PATH="/workspace/.venv/bin:$PATH" + +# Triton bundled ptxas doesn't support latest GPU architectures +ENV TRITON_PTXAS_PATH="/usr/local/cuda/bin/ptxas" + +ENTRYPOINT ["/workspace/docker/entrypoint.sh"] + +CMD ["/bin/bash"] diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..a5b8c94 --- /dev/null +++ b/LICENSE @@ -0,0 +1,52 @@ +Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + + +OpenMDW License Agreement, version 1.1 (OpenMDW-1.1) + +By exercising rights granted to you under this agreement, you accept and agree +to its terms. + +As used in this agreement, "Model Materials" means the materials provided to +you under this agreement, consisting of: (1) one or more machine learning +models (including architecture and parameters); and (2) all related artifacts +(including associated data, documentation and software) that are provided to +you hereunder. + +Subject to your compliance with this agreement, permission is hereby granted, +free of charge, to deal in the Model Materials without restriction, including +under all copyright, patent, database, and trade secret rights included or +embodied therein. + +If you distribute any portion of the Model Materials, you shall retain in your +distribution (1) a copy of this agreement, and (2) all copyright notices and +other notices of origin included in the Model Materials that are applicable to +your distribution. + +If you file, maintain, or voluntarily participate in a lawsuit against any +person or entity asserting that the Model Materials directly or indirectly +infringe any patent or copyright, then all rights and grants made to you +hereunder are terminated, unless that lawsuit was in response to a +corresponding lawsuit first brought against you. + +This agreement does not impose any restrictions or obligations with respect to +any use, modification, or sharing of any outputs generated by using the Model +Materials. + +THE MODEL MATERIALS ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS +OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE, TITLE, NONINFRINGEMENT, ACCURACY, OR THE +ABSENCE OF LATENT OR OTHER DEFECTS OR ERRORS, WHETHER OR NOT DISCOVERABLE, ALL +TO THE GREATEST EXTENT PERMISSIBLE UNDER APPLICABLE LAW. + +YOU ARE SOLELY RESPONSIBLE FOR (1) CLEARING RIGHTS OF OTHER PERSONS THAT MAY +APPLY TO THE MODEL MATERIALS OR ANY USE THEREOF, INCLUDING WITHOUT LIMITATION +ANY PERSON'S COPYRIGHTS OR OTHER RIGHTS INCLUDED OR EMBODIED IN THE MODEL +MATERIALS; (2) OBTAINING ANY NECESSARY CONSENTS, PERMISSIONS OR OTHER RIGHTS +REQUIRED FOR ANY USE OF THE MODEL MATERIALS; OR (3) PERFORMING ANY DUE +DILIGENCE OR UNDERTAKING ANY OTHER INVESTIGATIONS INTO THE MODEL MATERIALS OR +ANYTHING INCORPORATED OR EMBODIED THEREIN. + +IN NO EVENT SHALL THE PROVIDERS OF THE MODEL MATERIALS BE LIABLE FOR ANY CLAIM, +DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR +OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE MODEL MATERIALS, THE +USE THEREOF OR OTHER DEALINGS THEREIN. diff --git a/NOTICE b/NOTICE new file mode 100644 index 0000000..9cbe5c6 --- /dev/null +++ b/NOTICE @@ -0,0 +1,54 @@ +Cosmos + +Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. + +This file contains attribution notices for third-party source code that is +adapted, copied, or vendored in this repository. Dependency notices are listed +separately in ATTRIBUTIONS.md. + +HuggingFace Transformers / Qwen +Source: https://github.com/huggingface/transformers +License: Apache-2.0 +Notices: + Copyright 2024 The Qwen team, Alibaba Group and the HuggingFace Inc. team. + Copyright 2025 The Qwen team, Alibaba Group and the HuggingFace Inc. team. + Copyright 2025 The Qwen Team and The HuggingFace Inc. team. + Copyright 2025 HuggingFace Inc. team. +Files: + cosmos_framework/model/vfm/vlm/qwen3_vl/configuration_qwen3_vl.py + cosmos_framework/model/vfm/vlm/qwen3_vl/qwen3_vl.py + cosmos_framework/model/vfm/vlm/qwen3_vl/utils.py + cosmos_framework/model/vfm/vlm/qwen3_vl/video_processing_qwen3_vl.py + +ByteDance-Seed Bagel / Qwen2 tokenizer +Source: https://github.com/ByteDance-Seed/Bagel +License: Apache-2.0 +Notice: + Copyright 2024 The Qwen Team and The HuggingFace Inc. team. +File: + cosmos_framework/model/vfm/tokenizers/tokenization_qwen2.py + +HuggingFace Diffusers UniPC scheduler +Source: https://github.com/huggingface/diffusers +License: Apache-2.0 +Notices: + Copyright 2024 TSAIL Team and The HuggingFace Team. All rights reserved. + Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved. +File: + cosmos_framework/model/vfm/diffusion/samplers/fm_solvers_unipc.py + +Detectron2 lazy config helpers +Source: https://github.com/facebookresearch/detectron2 +License: Apache-2.0 +Notice: + Copyright (c) Facebook, Inc. and its affiliates. +File: + cosmos_framework/utils/lazy_config/__init__.py + +TaylorSeer +Source: https://github.com/Shenyi-Z/TaylorSeer +License: GPL-3.0 +Notice: + Upstream source files do not contain an explicit copyright notice. +Files: + cosmos_framework/model/vfm/utils/taylorseer.py diff --git a/README.md b/README.md index 56b69ec..ac531c0 100644 --- a/README.md +++ b/README.md @@ -1,154 +1,102 @@ -# __NVIDIA_OSS__ Standard Repo Template - -This README file is from the NVIDIA_OSS standard repo template of [PLC-OSS-Template](https://github.com/NVIDIA-GitHub-Management/PLC-OSS-Template?tab=readme-ov-file). It provides a list of files in the PLC-OSS-Template and guidelines on how to use (clone and customize) them. - -**Upon completing the customization for the project repo, the repo admin should replace this README template with the project specific README file.** - -- Files (org-wide templates in the NVIDIA .github org repo; per-repo overrides allowed) in [PLC-OSS-Template](https://github.com/NVIDIA-GitHub-Management/PLC-OSS-Template?tab=readme-ov-file) - - - Root - - README.md skeleton (CTA + Quickstart + Support/Security/Governance links) - - LICENSE (Apache 2.0 by default) - - For other licenses, see the [Confluence page](https://confluence.nvidia.com/pages/viewpage.action?pageId=788418816) for other licenses - - CLA.md file (delete if not using MIT or BSD licenses) - - CODE_OF_CONDUCT.md - - SECURITY.md (vuln reporting path) - - CONTRIBUTING.md (base; repo can add specifics) - - SUPPORT.md (Support levels/channels) - - GOVERNANCE.md (baseline; repo may extend) - - CITATION.md (for projects that need citation) - - - .github/ - - ISSUE_TEMPLATE/ () - - bug.yml, feature.yml, task.yml, config.yml - - PULL_REQUEST_TEMPLATE.md () - - workflows/ - - Note: workflow-templates/ for starter workflows should live in the org-level .github repo, not per-repo - - - Repo-specific (not org-template, maintained by the team) - - CODEOWNERS (place at .github/CODEOWNERS or repo root) - - CHANGELOG.md (or RELEASE.md) - - ROADMAP.md - - MAINTAINERS.md - - NOTICE or THIRD_PARTY_NOTICES / THIRD_PARTY_LICENSES (dependency specific) - - Build/package files (CMake, pyproject, Dockerfile, etc.) - - - Recommended structure and hygiene - - docs/ - - examples/ - - tests/ - - scripts/ - - Container/dev env: Dockerfile, docker/, .devcontainer/ (optional) - - Build/package (language-specific): - - Python: pyproject.toml, setup.cfg/setup.py, requirements.txt, environment.yml - - C++: CMakeLists.txt, cmake/, vcpkg.json - - Repo hygiene: .gitignore, .gitattributes, .editorconfig, .pre-commit-config.yaml, .clang-format - - -## Usage of [PLC-OSS-Template](https://github.com/NVIDIA-GitHub-Management/PLC-OSS-Template?tab=readme-ov-file) for NEW NVIDIA OSS repos - -1. Clone the [PLC-OSS-Template](https://github.com/NVIDIA-GitHub-Management/PLC-OSS-Template?tab=readme-ov-file) -2. Find/replace all in the clone of `___PROJECT___` and `__PROJECT_NAME__` with the name of the specific project. -3. Inspect all files to make sure all replacements work and update text as needed - - -**What you can reuse immediately** -- CODE_OF_CONDUCT.md -- SECURITY.md -- CONTRIBUTING.md (base) -- .github/ISSUE_TEMPLATE/.yml (bug/feature/task + config.yml) -- .github/PULL_REQUEST_TEMPLATE.md -- Reusable workflows - -**What you must customize per repo** -- README.md: copy the skeleton and fill in product-specific details (Quickstart, Requirements, Usage, Support level, links) -- LICENSE: check file is correct, update year, consult Confluence for alternatives https://confluence.nvidia.com/pages/viewpage.action?pageId=788418816, add CLA.md only if your license/process requires it -- CODEOWNERS: replace with your GitHub team handle(s). Place at .github/CODEOWNERS (or repo root) -- MAINTAINERS.md: list maintainers names/roles, escalation path -- CHANGELOG.md (or RELEASE.md): track releases/changes -- SUPPORT.md: Update for your project -- ROADMAP.md (optional): upcoming milestones -- NOTICE / THIRD_PARTY_NOTICES (if you ship third‑party content) -- Build/package files (CMake/pyproject/Dockerfile/etc.), tests/, docs/, examples/, scripts/ as appropriate -- Workflows: Edit if you need custom behavior - - -4. Change git origin to point to new repo and push -5. Remove the line break below and everything above it - -## Usage for existing NVIDIA OSS repos - -1. Follow the steps above, but add the files to your existing repo and merge - - ------------------------------------------ -# [Project Title] -One-sentence value proposition for users. Who is it for, and why it matters. - -# Overview -What the project does? Why the project is useful? -Provide a brief overview, highlighting key features or problem-solving capabilities. - -# Getting Started -Guide users on how they can get started with the project. This should include basic installation step, quick-start examples -```bash -# Option A: Package manager (pip/conda/npm/etc.) - - -# Option B: Container -docker run - -# Verify (hello world) - +

+ NVIDIA Cosmos +

+ +

🤗 Hugging Face

+ +# Cosmos-Framework + +**Cosmos-Framework** is an end-to-end framework for training and serving world foundation models, including the **Cosmos3** model family. Everything lives in a single top-level [`cosmos_framework/`](./cosmos_framework) Python package: + +- **Training** — distributed FSDP / TP / CP / PP trainer, native DCP checkpoints with HuggingFace `safetensors` import/export, JSONL / WebDataset / LeRobot dataset adapters. Entry point: `cosmos_framework.scripts.train`. See [`docs/training.md`](./docs/training.md). +- **Inference** — Diffusers / Transformers / vLLM backends with offline batch generation and online serving (Ray + Gradio). Entry point: `cosmos_framework.scripts.inference`. Ecosystem-facing shim libraries (lightweight standalone wrappers for downstream projects) live under [`packages/`](./packages). + +## Documentation + +- [Gallery](./docs/gallery.md) +- [Quickstart](#setup) +- [Setup](./docs/setup.md) +- [Training (Supervised Fine-Tuning)](./docs/training.md) + - [JSONL Dataset](./docs/dataset_jsonl.md) + - [Action Policy Closed-Loop Evaluation on LIBERO](./docs/action_policy_closed_loop_eval.md) +- [Prompting](./docs/prompting.md) +- [Inference](./docs/inference.md) +- Reference + - [Code Structure](./docs/code_structure.md) + - [Environment Variables](./docs/environment_variables.md) + - [FAQ](./docs/faq.md) + - [AGENTS.md](./AGENTS.md) + +## Overview + +**Cosmos3** is a world foundation model that unifies understanding and generation within a single Mixture-of-Transformer (MoT) architecture. Two tightly coupled towers—a **Reasoner** (vision-language model) and a **Generator** (world simulator)—share latent representations so that structured perception directly grounds realistic, temporally consistent simulation. + +

Image

+ +One model, many capabilities: + +| Input Modality | Output Modality | Application | Status | +| ----------------------- | --------------- | --------------------- | ------------ | +| Video \| Text | Video | Video Generator | ✅ | +| Video \| Text | Text | Vision Language Model | ✅ | +| Action \| Video \| Text | Video | World Model | ✅ | +| Video \| Text | Video & Action | Policy Model | ✅ | + +## Setup + +For more details and alternative installation methods, see [Setup](./docs/setup.md#installation). Before installing, make sure your machine meets the [System Requirements](./docs/setup.md#system-requirements). If you want a curated PyTorch + CUDA environment, start from the [recommended NVIDIA NGC base image](./docs/setup.md#recommended-base-image). + +Install system dependencies: + +```shell +sudo apt-get install -y --no-install-recommends curl ffmpeg git-lfs libx11-dev tree wget ``` -# Requirements -Include a list of pre-requisites. -- OS/Arch: -- Runtime/Compiler: -- GPU/Drivers (if applicable): CUDA , driver , etc. - -# Usage -```bash -# Minimal runnable snippet (≤20 lines) - + +Install the package with `uv` (pick the dependency group that matches your CUDA toolkit — see [CUDA Variants](./docs/setup.md#cuda-variants)): + +```shell +# CUDA 13.0 (recommended) +uv sync --all-extras --group=cu130-train +# Or, for CUDA 12.8: +# uv sync --all-extras --group=cu128-train +source .venv/bin/activate && export LD_LIBRARY_PATH= ``` -- More examples/tutorials: -- API reference: - -# Performance (Optional) -Summary of benchmarks; link to detailed results and hardware used. - -## Releases & Roadmap -- Releases/Changelog: -- (Optional) Next milestones or link to `ROADMAP.md`. - -# Contribution Guidelines -- Start here: `CONTRIBUTING.md` -- Code of Conduct: `CODE_OF_CONDUCT.md` -- Development quickstart (build/test): -```bash - && && + +If you are starting from the recommended NGC image (`nvcr.io/nvidia/pytorch:25.09-py3`), see the [one-shot quickstart](./docs/setup.md#quickstart-from-the-recommended-base-image). + +## Training + +For the full guide (data preparation, base-checkpoint conversion, parallelism strategies, mixed precision, resuming), see [Training](./docs/training.md). A minimal single-GPU training launch looks like: + +```shell +python -m cosmos_framework.scripts.train --sft-toml=examples/toml/sft_config/.toml ``` -## Governance & Maintainers -- Governance: `GOVERNANCE.md` -- Maintainers: -- Labeling/triage policy: -## Security -- Vulnerability disclosure: `SECURITY.md` -- Do not file public issues for security reports. +## Prompting + +See [Prompting](./docs/prompting.md). -## Support -- Level: -- How to get help: Issues/Discussions/ -- Response expectations (if any). +## Inference -# Community -Provide the channel for community communications. +See [Inference](./docs/inference.md) for the full guide — launch commands, supported modes, parallelism presets, and troubleshooting. + +Quick single-GPU launch: + +```shell +python -m cosmos_framework.scripts.inference \ + --parallelism-preset=latency \ + -i "inputs/omni/t2v.json" \ + -o outputs/omni_nano \ + --checkpoint-path Cosmos3-Nano \ + --seed=0 +``` -# References -Provide a list of related references +## Reference -# License -This project is licensed under the [NAME HERE] License - see the LICENSE.md file for details -- License: +| Topic | What it covers | +| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------ | +| [Setup](./docs/setup.md) | Hardware/software prerequisites, `uv` install paths, CUDA variants, Docker base image, and base-checkpoint downloading. | +| [Code Structure](./docs/code_structure.md) | Repository layout and a per-subpackage tour of `cosmos_framework/` — where each concern lives and where to add new code. | +| [Training](./docs/training.md) | Launching single-GPU, multi-GPU, and multi-node runs; parallelism strategies; mixed precision; resuming. | +| [Inference (from a trained checkpoint)](./docs/inference.md) | Loading a trained checkpoint into one of the inference backends. | +| [FAQ](./docs/faq.md) | Troubleshooting (OOM, NCCL hangs, slow training), environment variables, and common pitfalls. | diff --git a/RELEASE.md b/RELEASE.md new file mode 100644 index 0000000..2581899 --- /dev/null +++ b/RELEASE.md @@ -0,0 +1,10 @@ +# Release Cadence + +Release notes will be published here as the framework reaches tagged releases. + +## Previous Releases + +| Version | Description | Date | +| ------- | ------------------------------------------------- | ---------- | +| v1.0 | Initial diffusion and autoregressive WFMs release | 2025-01-06 | +| v0.1 | Initial tokenizer release | 2024-11-06 | diff --git a/SECURITY.md b/SECURITY.md deleted file mode 100644 index 9d1a711..0000000 --- a/SECURITY.md +++ /dev/null @@ -1,24 +0,0 @@ -## Security - -NVIDIA is dedicated to the security and trust of our software products and services, including all source code repositories managed through our organization. - -If you need to report a security issue, please use the appropriate contact points outlined below. **Please do not report security vulnerabilities through GitHub.** If a potential security issue is inadvertently reported via a public issue or pull request, NVIDIA maintainers may limit public discussion and redirect the reporter to the appropriate private disclosure channels. - -## Reporting Potential Security Vulnerability in an NVIDIA Product - -To report a potential security vulnerability in any NVIDIA product: -- Web: [Security Vulnerability Submission Form](https://www.nvidia.com/object/submit-security-vulnerability.html) -- E-Mail: psirt@nvidia.com - - We encourage you to use the following PGP key for secure email communication: [NVIDIA public PGP Key for communication](https://www.nvidia.com/en-us/security/pgp-key) - - Please include the following information: - - Product/Driver name and version/branch that contains the vulnerability - - Type of vulnerability (code execution, denial of service, buffer overflow, etc.) - - Instructions to reproduce the vulnerability - - Proof-of-concept or exploit code - - Potential impact of the vulnerability, including how an attacker could exploit the vulnerability - -While NVIDIA currently does not have a bug bounty program, we do offer acknowledgement when an externally reported security issue is addressed under our coordinated vulnerability disclosure policy. Please visit our [Product Security Incident Response Team (PSIRT)](https://www.nvidia.com/en-us/security/psirt-policies/) policies page for more information. - -## NVIDIA Product Security - -For all security-related concerns, please visit NVIDIA's Product Security portal at https://www.nvidia.com/en-us/security diff --git a/ci/.link-check.json b/ci/.link-check.json new file mode 100644 index 0000000..3546a99 --- /dev/null +++ b/ci/.link-check.json @@ -0,0 +1,10 @@ +{ + "ignorePatterns": [ + { + "pattern": "localhost" + }, + { + "pattern": "^https://github-production-user-asset" + } + ] +} diff --git a/ci/.markdown-toc-creator.toml b/ci/.markdown-toc-creator.toml new file mode 100644 index 0000000..f2a7879 --- /dev/null +++ b/ci/.markdown-toc-creator.toml @@ -0,0 +1,7 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +[tool.markdown_toc_creator] +proactive = false +exclude = '/_src/' +quiet = true diff --git a/ci/.pre-commit-config-base.yaml b/ci/.pre-commit-config-base.yaml new file mode 100644 index 0000000..aa3a5bb --- /dev/null +++ b/ci/.pre-commit-config-base.yaml @@ -0,0 +1,14 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +repos: + - repo: https://github.com/pre-commit/pre-commit-hooks + rev: v6.0.0 + hooks: + - id: check-added-large-files + args: ['--maxkb=10000'] # 10MB + - id: forbid-submodules + - repo: https://github.com/gitleaks/gitleaks + rev: v8.30.0 + hooks: + - id: gitleaks diff --git a/ci/license.txt b/ci/license.txt new file mode 100644 index 0000000..8d8e828 --- /dev/null +++ b/ci/license.txt @@ -0,0 +1,2 @@ +SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: OpenMDW-1.1 diff --git a/ci/uv_lock.sh b/ci/uv_lock.sh new file mode 100755 index 0000000..6538879 --- /dev/null +++ b/ci/uv_lock.sh @@ -0,0 +1,15 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# Generate uv lock files for projects. + +set -euo pipefail + +for file in "$@"; do + project_dir="$(dirname "$file")" + if ! uv lock -q --check --project "$project_dir" &>/dev/null; then + echo "Updating lock file for '$project_dir'" >&2 + uv lock -q --project "$project_dir" + fi +done diff --git a/ci/uv_lock_script.sh b/ci/uv_lock_script.sh new file mode 100755 index 0000000..ebae722 --- /dev/null +++ b/ci/uv_lock_script.sh @@ -0,0 +1,16 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# Generate uv lock files for scripts. + +set -euo pipefail + +for file in "$@"; do + if head -n1 "$file" | grep -q '^#!/usr/bin/env -S uv run --script'; then + if ! uv lock -q --check --script "$file" &>/dev/null; then + echo "Updating lock file for '$file'" >&2 + uv lock -q --script "$file" + fi + fi +done diff --git a/conftest.py b/conftest.py new file mode 100644 index 0000000..7d89bdb --- /dev/null +++ b/conftest.py @@ -0,0 +1,270 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from cosmos_framework.utils.lazy_config import lazy_call + + +lazy_call._CONVERT_TARGET_TO_STRING = True + +import gc +import os +from functools import cache +from pathlib import Path + +import pytest + +from cosmos_framework.inference.fixtures.args import ALL_LEVELS, ALL_NUM_GPUS, ALLOWED_GPUS_BY_LEVEL, Args, get_args, init_args + + +@pytest.fixture(scope="module") +def original_datadir(request: pytest.FixtureRequest) -> Path: + root_dir = request.config.rootpath + relative_path = request.path.with_suffix("").relative_to(root_dir) + return root_dir / "tests/data" / relative_path + + +@cache +def _get_available_gpus() -> int: + import pynvml + + try: + pynvml.nvmlInit() + device_count = pynvml.nvmlDeviceGetCount() + pynvml.nvmlShutdown() + return device_count + except pynvml.NVMLError as e: + print(f"WARNING: Failed to get available GPUs: {e}") + return 0 + + +def pytest_addoption(parser: pytest.Parser): + parser.addoption("--manual", action="store_true", default=False, help="Run manual tests") + parser.addoption( + "--num-gpus", + default=None, + type=int, + choices=ALL_NUM_GPUS, + help="Run tests with the specified number of GPUs", + ) + parser.addoption("--levels", default=None, help="Run tests with the specified levels (comma-separated list)") + + +def pytest_xdist_auto_num_workers(config: pytest.Config) -> int | None: + num_gpus: int | None = config.option.num_gpus + if num_gpus is None: + return 1 + if num_gpus == 0: + return None + + available_gpus = _get_available_gpus() + if available_gpus < num_gpus: + raise ValueError(f"Not enough GPUs available. Required: {num_gpus}, Available: {available_gpus}") + return available_gpus // num_gpus + + +def pytest_configure(config: pytest.Config): + args = Args.from_config(config) + init_args(args) + + if ( + args.num_gpus is not None + and args.levels is not None + and all(args.num_gpus not in ALLOWED_GPUS_BY_LEVEL[level] for level in args.levels) + ): + pytest.exit(f"No tests for {args.num_gpus} GPUs and levels {args.levels}.", returncode=0) + + if args.worker_id == "master": + return + + if args.worker_index > 1: + if args.num_gpus is None: + raise NotImplementedError(f"Running parallel tests requires --num-gpus to be set.") + + # Check if there are enough GPUs available. + if args.num_gpus is not None and args.num_gpus > 0: + required_gpus = args.num_gpus * (args.worker_index + 1) + else: + required_gpus = 1 + available_gpus = _get_available_gpus() + if available_gpus < required_gpus: + raise ValueError(f"Not enough GPUs available. Required: {required_gpus}, Available: {available_gpus}") + + # Limit threading to reduce contention + import torch + + torch.set_num_threads(1) + torch.set_num_interop_threads(1) + + +def _get_marker(item: pytest.Item, name: str) -> pytest.Mark | None: + markers = list(item.iter_markers(name=name)) + if not markers: + return None + marker = markers[0] + for other_marker in markers[1:]: + if other_marker != marker: + raise ValueError(f"Multiple different markers found for {name}: {markers}") + return marker + + +def _parse_level_marker(mark: pytest.Mark) -> int: + if len(mark.args) != 1: + raise ValueError(f"Invalid arguments: {mark.args}") + if mark.kwargs: + raise ValueError(f"Invalid keyword arguments: {mark.kwargs}") + level = mark.args[0] + if level not in ALL_LEVELS: + raise ValueError(f"Invalid level {level} not in {ALL_LEVELS}") + return level + + +def _parse_gpus_marker(mark: pytest.Mark) -> int: + if len(mark.args) != 1: + raise ValueError(f"Invalid arguments: {mark.args}") + if mark.kwargs: + raise ValueError(f"Invalid keyword arguments: {mark.kwargs}") + required_gpus = int(mark.args[0]) + if required_gpus not in ALL_NUM_GPUS: + raise ValueError(f"Invalid number of GPUs {required_gpus} not in {ALL_NUM_GPUS}") + return required_gpus + + +def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item]): + args = get_args() + + for item in items: + manual_mark = _get_marker(item, "manual") + level_mark = _get_marker(item, "level") + gpus_mark = _get_marker(item, "gpus") + try: + level = _parse_level_marker(level_mark) if level_mark else 0 + gpus = _parse_gpus_marker(gpus_mark) if gpus_mark else 0 + except ValueError as e: + pytest.fail(f"Invalid marker on test {item.name}: {e}") + assert False, "unreachable" + + allowed_gpus = ALLOWED_GPUS_BY_LEVEL[level] + if gpus not in allowed_gpus: + pytest.fail(f"Level {level} tests must have {allowed_gpus} GPUs, but {item.name} has {gpus} GPUs") + + # Check if the test should be skipped + if not args.enable_manual and manual_mark is not None: + item.add_marker(pytest.mark.skip(reason="test requires --manual")) + if args.levels is not None and level not in args.levels: + item.add_marker(pytest.mark.skip(reason=f"test requires --levels={level}")) + if args.num_gpus is not None and gpus != args.num_gpus: + item.add_marker(pytest.mark.skip(reason=f"test requires --num-gpus={gpus}")) + available_gpus = _get_available_gpus() + if gpus > available_gpus: + item.add_marker( + pytest.mark.skip(reason=f"test requires {gpus} GPUs, but only {available_gpus} are available") + ) + + # Exclude skipped tests + selected_items = [] + deselected_items = [] + for item in items: + if item.get_closest_marker("skip"): + deselected_items.append(item) + continue + selected_items.append(item) + items[:] = selected_items + config.hook.pytest_deselected(items=deselected_items) + + +def pytest_runtest_setup(item: pytest.Item): + import torch + + args = get_args() + + gpus_mark = item.get_closest_marker(name="gpus") + try: + gpus = _parse_gpus_marker(gpus_mark) if gpus_mark else 0 + except ValueError as e: + pytest.fail(f"Invalid marker on test {item.name}: {e}") + assert False, "unreachable" + + # Limit the number of GPUs used by the test + if gpus > 0: + device_start = args.worker_index * gpus + device_end = device_start + gpus + os.environ["CUDA_VISIBLE_DEVICES"] = ",".join(map(str, range(device_start, device_end))) + os.environ["NUM_GPUS"] = str(gpus) + else: + device = 0 + os.environ["CUDA_VISIBLE_DEVICES"] = str(device) + os.environ["NUM_GPUS"] = "1" + + test_max_processes = int(os.environ.get("TEST_MAX_PROCESSES", "8")) + device_memory_fraction = 1 / max(args.worker_count, test_max_processes) + os.environ["DEVICE_MEMORY_FRACTION"] = str(device_memory_fraction) + torch.cuda.set_per_process_memory_fraction(device_memory_fraction) + + +@pytest.fixture(autouse=True) +def init_cosmos_test(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + from cosmos_framework.inference.common.init import _init_log_console, _init_log_files + + monkeypatch.setenv("IMAGINAIRE_OUTPUT_ROOT", str(tmp_path / "imaginaire4-output")) + + _init_log_console() + _init_log_files(tmp_path) + + yield + + +@pytest.fixture(autouse=True) +def init_torch_test(): + import torch + + from cosmos_framework.inference.common.init import set_seed + + # Reproducibility + set_seed(0) + + yield + + # Cleanup memory + gc.collect() + if torch.cuda.is_available(): + torch.cuda.empty_cache() + + + +_WHITELIST_ENV_VARS = { + "LD_LIBRARY_PATH", + "QT_QPA_FONTDIR", + "QT_QPA_PLATFORM_PLUGIN_PATH", + "TORCHINDUCTOR_CACHE_DIR", +} + + +@pytest.fixture(autouse=True) +def detect_env_modifications(): + original_env = dict(os.environ) + + yield + + new_env = dict(os.environ) + + for env in [original_env, new_env]: + for k in list(env.keys()): + if k.startswith("PYTEST_") or k in _WHITELIST_ENV_VARS: + del env[k] + if new_env != original_env: + added, removed, modified = _compare_dict(new_env, original_env) + os.environ.clear() + os.environ.update(original_env) + raise ValueError( + f"Environment variables modified by test! Use 'monkeypatch.setenv' to temporarily modify environment variables. \n" + f"Added: {added}\n" + f"Removed: {removed}\n" + f"Modified: {modified}" + ) + + +def _compare_dict(actual: dict[str, str], expected: dict[str, str]) -> tuple[set[str], set[str], set[str]]: + added = set(actual) - set(expected) + removed = set(expected) - set(actual) + modified = {k for k in expected if k in actual and expected[k] != actual[k]} + return added, removed, modified diff --git a/cosmos_framework/__init__.py b/cosmos_framework/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/callbacks/__init__.py b/cosmos_framework/callbacks/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/callbacks/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/callbacks/compile_tokenizer.py b/cosmos_framework/callbacks/compile_tokenizer.py new file mode 100644 index 0000000..84efa16 --- /dev/null +++ b/cosmos_framework/callbacks/compile_tokenizer.py @@ -0,0 +1,106 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Training callback that defers AOT compilation of the VAE tokenizer. + +The actual compilation logic lives in +:meth:`~projects.cosmos3.vfm.tokenizers.wan2pt2_vae_4x16x16.Wan2pt2VAEInterface.compile_encode`. +This module provides a :class:`CompileTokenizer` callback that invokes it +at the right point during training (after ``compile_after_iterations`` +steps, to avoid NCCL timeouts during CUDA/cuDNN warm-up). + +Typical config usage +-------------------- +.. code-block:: python + + CompileTokenizer( + enabled=True, + compile_after_iterations=3, + warmup_resolutions=["256", "480", "720"], + ) +""" + +from collections.abc import Sequence + +import torch + +from cosmos_framework.utils import log +from cosmos_framework.utils.callback import Callback +from cosmos_framework.model.vfm.omni_mot_model import OmniMoTModel + + +class CompileTokenizer(Callback): + """Training callback that defers AOT compilation of the VAE tokenizer. + + Hooks into ``on_training_step_start``. On the + ``compile_after_iterations``-th step it calls + ``Wan2pt2VAEInterface.compile_encode`` to compile and load all chunk + variants. Every subsequent step is a no-op. + """ + + def __init__( + self, + enabled: bool = False, + compile_after_iterations: int = 3, + warmup_resolutions: Sequence[str] | None = None, + ): + """ + Args: + enabled: Master switch. When ``False`` the callback is a + complete no-op and no compilation occurs. + compile_after_iterations: How many training steps to skip + before triggering compilation. The default (3) lets CUDA + context setup and Transformer compilation finish first. + warmup_resolutions: Resolution keys (e.g. ``["256", "480", "720"]``) + to AOT-compile. Should include every resolution used in + training. Must be a non-empty list when *enabled* is ``True``. + """ + super().__init__() + self.enabled: bool = enabled + self.compile_after_iterations: int = compile_after_iterations + self.skip_counter: int = 0 + self.warmup_resolutions: Sequence[str] | None = warmup_resolutions + + if self.enabled: + if self.warmup_resolutions is None: + raise ValueError("warmup_resolutions must be provided when enabled, got None") + if len(self.warmup_resolutions) == 0: + raise ValueError("warmup_resolutions must be a non-empty list when enabled, got an empty list") + + def on_training_step_start( + self, model: OmniMoTModel, data_batch: dict[str, torch.Tensor], iteration: int = 0 + ) -> None: + """Called at the start of every training step. + + On the ``compile_after_iterations``-th call, triggers AOT compilation + via ``tokenizer.compile_encode``. + + Args: + model: The OmniMoTModel whose ``tokenizer_vision_gen`` will be compiled. + data_batch: Current training batch (unused, required by Callback API). + iteration: Current training iteration (unused; we track our own counter + via ``skip_counter`` because this callback may be registered after + iteration 0). + """ + if not self.enabled: + return + + tokenizer = model.tokenizer_vision_gen + + if isinstance(tokenizer, torch.jit.ScriptModule): + log.critical( + f"The Tokenizer model {type(tokenizer)} is a JIT model, " + "which is not compilable. The Tokenizer will not be compiled.", + rank0_only=False, + ) + self.enabled = False + return + + if self.skip_counter == self.compile_after_iterations: + if self.warmup_resolutions is not None: + tokenizer.compile_encode( + self.warmup_resolutions, + output_dir=self.config.job.path_local, + ) + + self.skip_counter += 1 diff --git a/cosmos_framework/callbacks/data_stats.py b/cosmos_framework/callbacks/data_stats.py new file mode 100644 index 0000000..20e3161 --- /dev/null +++ b/cosmos_framework/callbacks/data_stats.py @@ -0,0 +1,207 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from __future__ import annotations + +import torch +import torch.distributed as dist +import torch.utils.data +import wandb + +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.utils import distributed +from cosmos_framework.utils.callback import Callback + + +class DataStatsCallback(Callback): + def __init__( + self, + logging_iter_multipler: int = 1, + save_s3: bool = False, + ) -> None: + super().__init__() + self.logging_iter_multipler = logging_iter_multipler + assert self.logging_iter_multipler > 0, "logging_iter_multipler should be greater than 0" + self.save_s3 = save_s3 + self.wandb_extra_tag = f"@{logging_iter_multipler}" if logging_iter_multipler > 1 else "" + self.name = "data_stats" + self.wandb_extra_tag + self.data_freq_current = {} + self.data_freq_acc = {} + self.avg_num_assistant_tokens = [] + self.avg_num_real_tokens = [] + self.max_num_real_tokens = [] + self.min_num_real_tokens = [] + + # Per-dataset token length tracking + self.dataset_token_lengths = {} # dataset_name -> list of avg_num_real_tokens + self.dataset_assistant_tokens = {} # dataset_name -> list of avg_num_assistant_tokens + self.num_log_current = 0 + self.total_count_acc = {} + + def on_training_step_end( + self, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int = 0, + ) -> None: + self.num_log_current += 1 + dataset_name = data_batch.get("dataset_name", "default") + + # Handle case where dataset_name gets batched into a list + if isinstance(dataset_name, list): + + assert len(dataset_name) == 1, "dataset_name should be a list of 1" + dataset_name = dataset_name[0] + + if dataset_name in ["default"] and "__url__" in data_batch: + # try to get the name from url + dataset_name = "/".join(data_batch["__url__"][0].split("/")[:-1]) + + if dataset_name not in self.data_freq_current: + self.data_freq_current[dataset_name] = torch.tensor(0, device="cuda") # [] + self.data_freq_current[dataset_name] += 1 + + if dataset_name not in self.data_freq_acc: + self.data_freq_acc[dataset_name] = torch.tensor(0, device="cuda") # [] + self.data_freq_acc[dataset_name] += 1 + + if "avg_num_assistant_tokens" in output_batch: + self.avg_num_assistant_tokens.append(output_batch["avg_num_assistant_tokens"]) + # Track per-dataset assistant tokens + if dataset_name not in self.dataset_assistant_tokens: + self.dataset_assistant_tokens[dataset_name] = [] + self.dataset_assistant_tokens[dataset_name].append(output_batch["avg_num_assistant_tokens"]) + if "avg_num_real_tokens" in output_batch: + self.avg_num_real_tokens.append(output_batch["avg_num_real_tokens"]) + # Track per-dataset token lengths + if dataset_name not in self.dataset_token_lengths: + self.dataset_token_lengths[dataset_name] = [] + self.dataset_token_lengths[dataset_name].append(output_batch["avg_num_real_tokens"]) + if "max_num_real_tokens" in output_batch: + self.max_num_real_tokens.append(output_batch["max_num_real_tokens"]) + if "min_num_real_tokens" in output_batch: + self.min_num_real_tokens.append(output_batch["min_num_real_tokens"]) + + if iteration % (self.config.trainer.logging_iter * self.logging_iter_multipler) == 0: + # Step 1: Gather all dataset names across ranks + local_dataset_names = list(self.data_freq_current.keys()) + all_dataset_names = [None for _ in range(dist.get_world_size())] + dist.all_gather_object(all_dataset_names, local_dataset_names) + + # Step 2: Create the union of all dataset names + union_dataset_names = set() + for names in all_dataset_names: + union_dataset_names.update(names) + union_dataset_names = sorted(list(union_dataset_names)) + + # Step 3: For any missing dataset name, add dummy _LossRecord with NaN loss + for dataset_name in union_dataset_names: + if dataset_name not in self.data_freq_acc: + self.data_freq_acc[dataset_name] = torch.tensor(0, device="cuda") # [] + if dataset_name not in self.data_freq_current: + self.data_freq_current[dataset_name] = torch.tensor(0, device="cuda") # [] + + # Step 4: Calculate the total count of each dataset across all ranks + total_count_current = {} + for dataset_name in union_dataset_names: + acc_tensor = self.data_freq_acc[dataset_name].clone() + current_tensor = self.data_freq_current[dataset_name].clone() + + dist.all_reduce(acc_tensor, op=dist.ReduceOp.SUM) + dist.all_reduce(current_tensor, op=dist.ReduceOp.SUM) + + self.total_count_acc[dataset_name] = acc_tensor.item() + total_count_current[dataset_name] = current_tensor.item() + + if distributed.is_rank0() and wandb.run is not None: + info = {} + if len(self.avg_num_assistant_tokens) > 0: + info["data_stats_tokens/avg_num_assistant_tokens"] = sum(self.avg_num_assistant_tokens) / len( + self.avg_num_assistant_tokens + ) + self.avg_num_assistant_tokens = [] + if len(self.avg_num_real_tokens) > 0: + info["data_stats_tokens/avg_num_real_tokens"] = sum(self.avg_num_real_tokens) / len( + self.avg_num_real_tokens + ) + self.avg_num_real_tokens = [] + if len(self.max_num_real_tokens) > 0: + info["data_stats_tokens/max_num_real_tokens"] = max(self.max_num_real_tokens) + self.max_num_real_tokens = [] + + if len(self.min_num_real_tokens) > 0: + info["data_stats_tokens/min_num_real_tokens"] = min(self.min_num_real_tokens) + self.min_num_real_tokens = [] + + # Log per-dataset average token lengths + for dataset_name in union_dataset_names: + if dataset_name in self.dataset_token_lengths and len(self.dataset_token_lengths[dataset_name]) > 0: + avg_token_length = sum(self.dataset_token_lengths[dataset_name]) / len( + self.dataset_token_lengths[dataset_name] + ) + info[f"data_stats_avg_tokens_per_dataset/{dataset_name}"] = avg_token_length + if ( + dataset_name in self.dataset_assistant_tokens + and len(self.dataset_assistant_tokens[dataset_name]) > 0 + ): + avg_assistant_tokens = sum(self.dataset_assistant_tokens[dataset_name]) / len( + self.dataset_assistant_tokens[dataset_name] + ) + info[f"data_stats_avg_assistant_tokens_per_dataset/{dataset_name}"] = avg_assistant_tokens + + # Reset per-dataset token lengths after logging + self.dataset_token_lengths = {} + self.dataset_assistant_tokens = {} + + # Log the valid count per dataset + for dataset_name in union_dataset_names: + info[f"data_stats_count_acc/{dataset_name}"] = self.total_count_acc[dataset_name] + info[f"data_stats_count_current/{dataset_name}"] = total_count_current[dataset_name] + self.num_log_current = 0 + + wandb.log(info, step=iteration) + + # Create a table of the data stats, columns: Dataset, Accumulated frequency, Current frequency, Accumulated Count, Current Count + table_html = "" + total_count_acc_sum = sum(self.total_count_acc.values()) + total_count_current_sum = sum(total_count_current.values()) + # Sort union_dataset_names by total_count_acc, from highest to lowest + union_dataset_names = sorted(union_dataset_names, key=lambda x: self.total_count_acc[x], reverse=True) + acc_freq_list = [] + current_freq_list = [] + for name in union_dataset_names: + acc_freq = self.total_count_acc[name] / total_count_acc_sum + acc_freq_list.append(acc_freq) + current_freq = total_count_current[name] / total_count_current_sum + current_freq_list.append(current_freq) + table_html += f"" + # Sum over all dataset for each column + acc_freq_sum = sum(acc_freq_list) + current_freq_sum = sum(current_freq_list) + table_html += f"" + + table_html += "
DatasetAccumulated frequencyCurrent frequencyAccumulated CountCurrent Count
{name}{acc_freq}{current_freq}{self.total_count_acc[name]}{total_count_current[name]}
Total ({len(union_dataset_names)}){acc_freq_sum}{current_freq_sum}{total_count_acc_sum}{total_count_current_sum}
" + wandb.log({"table_data_stats/html": wandb.Html(table_html)}, step=iteration) + # Reset self.data_freq_current + self.data_freq_current = {k: v * 0 for k, v in self.data_freq_current.items()} + if ( + distributed.is_rank0() + and wandb.run is not None + and iteration in [100, 1000, 2000, 5000, 15000, 30000] + and len(self.total_count_acc) + ): + # log a table of the total_count_acc + # Sort self.total_count_acc by value, from highest to lowest + sorted_total_count_acc = sorted(self.total_count_acc.items(), key=lambda x: x[1], reverse=True) + table = wandb.Table(data=[[k, v] for k, v in sorted_total_count_acc], columns=["Dataset", "Count"]) + + wandb.log( + { + f"data_counts_bar_{iteration:09d}": wandb.plot.bar( + table, "Dataset", "Count", title=f"Count per Dataset iter {iteration:09d}" + ) + }, + step=iteration, + ) diff --git a/cosmos_framework/callbacks/dataloader_state.py b/cosmos_framework/callbacks/dataloader_state.py new file mode 100644 index 0000000..fee45b9 --- /dev/null +++ b/cosmos_framework/callbacks/dataloader_state.py @@ -0,0 +1,217 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from __future__ import annotations + +import os +from dataclasses import dataclass +from typing import Any + +import torch + +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.utils import log +from cosmos_framework.utils.callback import Callback + + +@dataclass +class NoReplaceShardlistState: + epoch: int = 0 + index: int = 0 + + +class DataLoaderStateCallback(Callback): + checkpoint_component: str = "dataloader" + + def __init__( + self, + distributor_type: str | None = None, + name: str = "", + ) -> None: + super().__init__() + self.distributor_type = distributor_type + self.name = name + self.config: Any = None + self.state: dict[int, NoReplaceShardlistState] = {} + self.verbose = True + + def _update_state_from_batch(self, data_batch: dict[str, torch.Tensor]) -> None: + if "sample_worker_id" not in data_batch: + return # batch has no position metadata (shuffle=False or iterable data_source) + worker_ids = data_batch["sample_worker_id"].tolist() # [B] + epochs = data_batch["sample_epoch"].tolist() # [B] + indices = data_batch["sample_index"].tolist() # [B] + for worker_id, epoch, index in zip(worker_ids, epochs, indices, strict=True): + if worker_id not in self.state: + self.state[worker_id] = NoReplaceShardlistState(epoch=epoch, index=index) + + elif self.state[worker_id].epoch < epoch or ( + self.state[worker_id].index < index and self.state[worker_id].epoch == epoch + ): + self.state[worker_id] = NoReplaceShardlistState(epoch=epoch, index=index) + + _ACTIVE_DISTRIBUTOR_TYPES = ("no_replace", "data_packer") + + def on_training_step_batch_end( + self, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int = 0, + ) -> None: + if self.distributor_type in self._ACTIVE_DISTRIBUTOR_TYPES: + self._update_state_from_batch(data_batch) + + def on_training_step_end( + self, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int = 0, + ) -> None: + if self.distributor_type in self._ACTIVE_DISTRIBUTOR_TYPES: + if self.verbose: + if iteration % self.config.trainer.logging_iter == 0: + msg = "\n" + for wid, state in self.state.items(): + msg += f"worker {wid}: epoch={state.epoch}, index={state.index}\n" + log.info(msg) + + def has_checkpoint_state(self) -> bool: + return self.distributor_type in self._ACTIVE_DISTRIBUTOR_TYPES + + def state_dict(self) -> dict[int, dict[str, int]]: + if self.distributor_type not in self._ACTIVE_DISTRIBUTOR_TYPES: + return {} + + state_dict: dict[int, dict[str, int]] = {} + for worker_id, per_worker_state in self.state.items(): + state_dict[worker_id] = {"epoch": per_worker_state.epoch, "index": per_worker_state.index} + log.info( + f"Saved dataloader state for worker {worker_id}: " + f"epoch={per_worker_state.epoch}, index={per_worker_state.index}" + ) + return state_dict + + def load_state_dict(self, state_dict: dict[int, dict[str, int]]) -> None: + if self.distributor_type not in self._ACTIVE_DISTRIBUTOR_TYPES: + return + + if not state_dict: + log.info("No dataloader state found in checkpoint") + return + + self.state = {} + # Build env var prefix. For data_packer, namespacing avoids conflicts + # when multiple DataPackerDataLoader instances share the same process + # (e.g. inside JointDataPackerDataLoader). name="" → original format. + _dp_pfx = f"DP_STATE_{self.name}_" if self.name else "DP_STATE_" + for worker_id, per_worker_state in state_dict.items(): + epoch = per_worker_state["epoch"] + index = per_worker_state["index"] + self.state[worker_id] = NoReplaceShardlistState(epoch=epoch, index=index) + if self.distributor_type == "data_packer": + os.environ[f"{_dp_pfx}WORKER_{worker_id}_EPOCH"] = str(epoch) + os.environ[f"{_dp_pfx}WORKER_{worker_id}_INDEX"] = str(index) + log.info(f"Loaded data_packer dataloader state for worker {worker_id}: epoch={epoch}, index={index}") + else: + os.environ[f"NSL_STATE_WORKER_{worker_id}_EPOCH"] = str(epoch) + os.environ[f"NSL_STATE_WORKER_{worker_id}_INDEX"] = str(index) + log.info(f"Loaded no_replace dataloader state for worker {worker_id}: epoch={epoch}, index={index}") + + +class JointDataLoaderStateCallback(Callback): + """Checkpoint/resume state for ``JointDataPackerDataLoader``. + + Manages two levels of state in a single DCP checkpoint entry + (``checkpoint_component = "dataloader"``): + + 1. **Outer** ``global_id`` — the number of batches the outer loader has + yielded. Restored via ``outer_loader.set_start_iteration(global_id)`` + so the deterministic dataset-selection sequence resumes from the correct + step. + + 2. **Inner** per-dataset, per-worker ``(epoch, index)`` — one + ``DataLoaderStateCallback`` per inner loader, keyed by the dataset name. + Each inner callback sets namespaced env vars on ``load_state_dict`` so + workers fast-forward to the saved sample position. + + Usage in experiment configs:: + + joint_loader = JointDataPackerDataLoader(dataloaders={...}, seed=42) + exp["dataloader_train"] = joint_loader + exp["trainer"]["callbacks"]["dataloader_state"] = JointDataLoaderStateCallback( + outer_loader=joint_loader, + distributor_type="data_packer", + ) + + The ``checkpoint_component = "dataloader"`` class attribute ensures the DCP + checkpointer's ``_DataloaderWrapper`` discovers exactly this callback (it + picks the first matching callback). Do **not** also register standalone + ``DataLoaderStateCallback`` instances for the inner loaders — this class + already handles them all. + """ + + checkpoint_component: str = "dataloader" + + def __init__( + self, + outer_loader: Any, + distributor_type: str = "data_packer", + ) -> None: + super().__init__() + self._outer = outer_loader + self._inner: dict[str, DataLoaderStateCallback] = { + name: DataLoaderStateCallback(distributor_type=distributor_type, name=name) + for name in outer_loader._names + } + self.config: Any = None + + def _update_state_from_batch(self, batch: dict) -> None: + name = batch.get("dataset_name") + if name in self._inner: + self._inner[name]._update_state_from_batch(batch) + + def on_training_step_batch_end( + self, + model: Any, + data_batch: dict, + output_batch: dict, + loss: Any, + iteration: int = 0, + ) -> None: + self._update_state_from_batch(data_batch) + + def on_training_step_end( + self, + model: Any, + data_batch: dict, + output_batch: dict, + loss: Any, + iteration: int = 0, + ) -> None: + if self.config and iteration % self.config.trainer.logging_iter == 0: + msg = f"\nJointDataPackerDataLoader global_id={self._outer._global_id}\n" + for name, cb in self._inner.items(): + for wid, state in cb.state.items(): + msg += f" [{name}] worker {wid}: epoch={state.epoch}, index={state.index}\n" + log.info(msg) + + def has_checkpoint_state(self) -> bool: + return True + + def state_dict(self) -> dict: + return { + "global_id": self._outer._global_id, + **{name: cb.state_dict() for name, cb in self._inner.items()}, + } + + def load_state_dict(self, state: dict) -> None: + global_id = state.get("global_id", 0) + self._outer.set_start_iteration(global_id) + log.info(f"JointDataLoaderStateCallback: resumed outer global_id={global_id}") + for name, cb in self._inner.items(): + if name in state: + cb.load_state_dict(state[name]) diff --git a/cosmos_framework/callbacks/dataloading_monitor.py b/cosmos_framework/callbacks/dataloading_monitor.py new file mode 100644 index 0000000..3f69754 --- /dev/null +++ b/cosmos_framework/callbacks/dataloading_monitor.py @@ -0,0 +1,673 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import time +from collections import defaultdict + +import psutil +import torch +import torch.distributed as dist +import wandb + +from imaginaire.datasets.webdataset.utils.stream import ( + ENABLE_STREAM_WANDB, + WATCHDOG_ENABLED, + collect_throughput_ipc_stats, +) +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.utils import distributed +from cosmos_framework.utils.callback import Callback +from cosmos_framework.utils.easy_io import easy_io +from cosmos_framework.data.vfm.joint_dataloader import _PackingMetrics + +_AGG_COUNT, _AGG_SUM, _AGG_MIN, _AGG_MAX = 0, 1, 2, 3 +_AGG_COLS = 4 + + +class DetailedDataLoadingSpeedMonitor(Callback): + def __init__( + self, + every_n: int, + step_size: int = 1, + save_s3: bool = False, + action_family_names: list[str] | None = None, + ): + """Detailed dataloading + memory diagnostics callback. + + Args: + every_n: How often (in training iterations) to flush stats to W&B. + step_size: Trainer step granularity (passed-through; usually 1). + save_s3: If True, snapshot ``wandb_info`` to ``s3://rundir/...`` on each flush. + action_family_names: Set of dataset names that are *action families* + (e.g. ``["umi_20260501", "fractal_20260501", ...]``). When + supplied, every flush also emits per-rank labeled metrics + ``action_dl_per_rank/{family}_rank_{NNN:03d}_{mean,max}`` — + which is exactly the per-rank dataloading wait under + Hare-Niemeyer (each rank serves one family) restricted to + **action iterations only** (i.e. iterations where the + ``JointDataLoader`` selected ``action_data``, not VFM). + Useful to disambiguate per-rank stalls in VFM+action mixed + runs without scrolling through 2048 ``dataloading_NNN`` + metrics whose rank ↔ family mapping is implicit. + If ``None`` (default), the per-rank labeled metrics are + skipped — the existing global ``dl_wait_time_per_dataset/{ds}_*`` + cross-rank aggregates remain available for back-compat. + """ + self.every_n = every_n + self.step_size = step_size + self.should_run = False + self.start_dataloading_time = None + self.dataloading_time = None + self.name = self.__class__.__name__ + self.save_s3 = save_s3 + self.time_delta_list = [] + # Parallel to ``time_delta_list``: the dataset name attached to the + # batch consumed at iter N. Populated at on_training_step_end (when + # ``data_batch["dataset_name"]`` is available), so its length tracks + # ``time_delta_list`` exactly under normal flow. Used to filter the + # per-rank dataloading metric to action-only iterations when + # ``action_family_names`` is supplied. + self._iter_dataset_names: list[str] = [] + # Node-wide RAM stats (psutil.virtual_memory()): the same value for all + # local ranks on the same host, so the per-rank suffix indexes nodes + # not ranks. Useful for tracking absolute host pressure / page-cache + # headroom but cannot attribute memory to a specific action family. + self.memory_consumption_list = [] + self.memory_consumption_percentage_list = [] + self._action_family_set: set[str] | None = set(action_family_names) if action_family_names else None + # Deterministic sorted list of action family names — same on every + # rank. Used to encode each rank's family identity as an integer + # index that travels alongside (mean, max) inside a single + # all_gather_tensor call, avoiding the slower all_gather_object that + # the per-rank label metric used to need. + self._action_family_list: list[str] = sorted(self._action_family_set) if self._action_family_set else [] + self._action_family_index: dict[str, int] = {f: i for i, f in enumerate(self._action_family_list)} + self._pending_time_delta = None + self.dataloading_time_per_dataset = {} + self._worker_batch_times = [] + self._worker_aug_times = [] + self._worker_io_times = [] + self._worker_aug_step_times: dict[str, list[float]] = defaultdict(list) + self._worker_times_by_ds_wid: dict[tuple[str, int], list[float]] = defaultdict(list) + self._dataset_scalar_stats: dict[str, dict[str, int]] = defaultdict(lambda: defaultdict(int)) + self._dataset_list_stats: dict[str, dict[str, list[int]]] = defaultdict(lambda: defaultdict(list)) + + def on_before_dataloading(self, iteration: int = 0) -> None: + # We want to run it one iteration before on_training_step_start should_run is set to True. + global_step = iteration // self.step_size + self.should_run = (global_step + 1) % self.every_n == 0 + self.start_dataloading_time = time.time() + + def on_after_dataloading(self, iteration: int = 0) -> None: + self._pending_time_delta = time.time() - self.start_dataloading_time + self.time_delta_list.append(self._pending_time_delta) + memory = psutil.virtual_memory() + self.memory_consumption_list.append(memory.used / (1024**3)) + self.memory_consumption_percentage_list.append(memory.percent) + + def on_training_step_end( + self, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int = 0, + ) -> None: + dataset_name = data_batch.get("dataset_name", ["default"])[0] + # ``_action_family`` is stamped per-sample by + # :class:`~projects.cosmos3.vfm.datasets.action.unified_dataset.ActionUnifiedIterableDataset` + # and survives the outer ``JointDataLoader`` (which overwrites + # ``dataset_name`` with the *stream* name e.g. ``"action_data"``). + # When present, it is the per-rank action family identity needed by + # the ``action_dl_per_rank/{family}_rank_{NNN:03d}_{mean,max}`` + # filter below. Falls back to ``dataset_name`` for VFM streams that + # have no inner family stamp. + action_family = data_batch["_action_family"][0] if "_action_family" in data_batch else None + family_or_stream = action_family if action_family is not None else dataset_name + # Always tag this iteration with its family-or-stream name (even + # when _pending_time_delta is None due to a missed + # on_after_dataloading call) so that ``_iter_dataset_names`` and + # ``time_delta_list`` stay the same length for the action-only + # filter below. + if len(self._iter_dataset_names) < len(self.time_delta_list): + self._iter_dataset_names.append(family_or_stream) + if self._pending_time_delta is not None: + if dataset_name not in self.dataloading_time_per_dataset: + self.dataloading_time_per_dataset[dataset_name] = [] + self.dataloading_time_per_dataset[dataset_name].append(self._pending_time_delta) + self._pending_time_delta = None + + for batch_key, _, agg_type in _PackingMetrics.STATS_SPEC: + if batch_key not in data_batch: + continue + val = int(data_batch[batch_key]) + if agg_type == "scalar": + self._dataset_scalar_stats[batch_key][dataset_name] += val + else: + self._dataset_list_stats[batch_key][dataset_name].append(val) + + if "_worker_batch_time" in data_batch: + bt = float(data_batch["_worker_batch_time"]) + self._worker_batch_times.append(bt) + wid = int(data_batch.get("_worker_id", 0)) + self._worker_times_by_ds_wid[(dataset_name, wid)].append(bt) + if "_worker_aug_time" in data_batch: + self._worker_aug_times.append(float(data_batch["_worker_aug_time"])) + if "_worker_io_time" in data_batch: + self._worker_io_times.append(float(data_batch["_worker_io_time"])) + if "_worker_aug_step_times" in data_batch: + for step_name, t in data_batch["_worker_aug_step_times"].items(): + self._worker_aug_step_times[step_name].append(float(t)) + + if self.should_run: + # Convert list to tensor on GPU for gathering + local_times = torch.tensor(self.time_delta_list).cuda() # [num_iters] + local_memory_consumption = torch.tensor(self.memory_consumption_list).cuda() # [num_iters] + local_memory_consumption_percentage = torch.tensor( + self.memory_consumption_percentage_list + ).cuda() # [num_iters] + iteration_count = len(self.time_delta_list) + # Compute action-only per-iter mask BEFORE clearing buffers — paired + # with ``time_delta_list`` 1:1 so we can derive each rank's action + # iteration mean/max in the per-rank labeled metric below. + iter_dataset_names_local = list(self._iter_dataset_names[:iteration_count]) + self.time_delta_list = [] # Reset the list + self.memory_consumption_list = [] + self.memory_consumption_percentage_list = [] + self._iter_dataset_names = self._iter_dataset_names[iteration_count:] + + # Gather all times from all ranks + # Each tensor in the list has shape (num_iterations,) + gathered_times_list = distributed.all_gather_tensor(local_times) # list of [num_iters], len=world_size + + # Stack to get shape (world_size, num_iterations) + all_times = torch.stack(gathered_times_list) # [world_size,num_iters] + + # Calculate per-rank statistics + # dim=1 is across iterations + rank_means = torch.mean(all_times, dim=1) # [world_size] + rank_maxes = torch.max(all_times, dim=1).values # [world_size] + + wandb_info = {f"{self.name}_mean/dataloading_{k:03d}": v.item() for k, v in enumerate(rank_means)} + wandb_info.update({f"{self.name}_max/dataloading_{k:03d}": v.item() for k, v in enumerate(rank_maxes)}) + + # Per-rank action-only labeled dataloading metric. Unlike the + # generic ``dataloading_NNN`` above (which mixes action + VFM in + # joint runs), this filters ``time_delta_list`` to iterations + # whose batch came from one of ``self._action_family_set``, then + # publishes each rank's mean/max under a name that includes the + # family it served — e.g. + # ``action_dl_per_rank/umi_20260501_rank_000_mean``. Because + # under Hare-Niemeyer each rank serves exactly one action family, + # a rank's filtered-times all share a single dataset name, and + # we use the most recent observed name as the family label. + # + # Family identity is encoded as an integer index into the + # deterministic ``_action_family_list`` (same on every rank, + # built once at __init__ from the constructor-supplied + # ``action_family_names``) so it can travel inside the same + # ``all_gather_tensor`` call as (mean, max) — no + # ``all_gather_object`` needed. + if self._action_family_set is not None and iter_dataset_names_local: + action_mask_local = torch.tensor( + [(ds in self._action_family_set) for ds in iter_dataset_names_local], + dtype=torch.bool, + ).cuda() # [num_iters] + # Pad to iteration_count if dataset_names were missed (shouldn't + # happen with normal flow but guards against edge cases). + if action_mask_local.numel() < iteration_count: + pad = torch.zeros( + iteration_count - action_mask_local.numel(), + dtype=torch.bool, + ).cuda() + action_mask_local = torch.cat([action_mask_local, pad]) + # Most recent action family observed locally — under + # Hare-Niemeyer this is the SAME family on every action + # iter for this rank, so picking the last one is exact. + local_family_label = next( + (ds for ds in reversed(iter_dataset_names_local) if ds in self._action_family_set), + None, + ) + local_family_idx = ( + self._action_family_index.get(local_family_label, -1) if local_family_label is not None else -1 + ) + if action_mask_local.any(): + action_times_local = all_times[distributed.get_rank()][action_mask_local] # [num_action_iters] + local_action_mean = action_times_local.mean().to(torch.float64) + local_action_max = action_times_local.max().to(torch.float64) + else: + local_action_mean = torch.tensor(float("nan"), dtype=torch.float64).cuda() + local_action_max = torch.tensor(float("nan"), dtype=torch.float64).cuda() + # Pack [mean, max, family_idx] into one tensor and gather + # once — single all_gather_tensor, no object exchange. + local_action_stats = torch.stack( + [ + local_action_mean, + local_action_max, + torch.tensor(float(local_family_idx), dtype=torch.float64).cuda(), + ] + ) # [3] + all_action_stats = self._gather_list_stats(local_action_stats) # [world_size, 3] + num_families = len(self._action_family_list) + for rank_id in range(all_action_stats.shape[0]): + fam_idx = int(all_action_stats[rank_id, 2].item()) + if fam_idx < 0 or fam_idx >= num_families: + continue # rank served no action data this window + m = all_action_stats[rank_id, 0].item() + mx = all_action_stats[rank_id, 1].item() + if m != m or mx != mx: # NaN check + continue + fam = self._action_family_list[fam_idx] + wandb_info[f"action_dl_per_rank/{fam}_rank_{rank_id:03d}_mean"] = m + wandb_info[f"action_dl_per_rank/{fam}_rank_{rank_id:03d}_max"] = mx + + gathered_memory_consumption = distributed.all_gather_tensor( + local_memory_consumption + ) # list of [num_iters], len=world_size + gathered_memory_consumption_percentage = distributed.all_gather_tensor( + local_memory_consumption_percentage + ) # list of [num_iters], len=world_size + + wandb_info.update( + { + f"{self.name}_mean/memory_consumption_gb_{k:03d}": v.mean().item() + for k, v in enumerate(gathered_memory_consumption) + } + ) + wandb_info.update( + { + f"{self.name}_mean/memory_consumption_percentage_{k:03d}": v.mean().item() + for k, v in enumerate(gathered_memory_consumption_percentage) + } + ) + # Per-rank max over the window — surfaces transient peaks (heavy + # families on lonely ranks can spike between mean-sampled steps). + wandb_info.update( + { + f"{self.name}_max/memory_consumption_gb_{k:03d}": v.max().item() + for k, v in enumerate(gathered_memory_consumption) + } + ) + wandb_info.update( + { + f"{self.name}_max/memory_consumption_percentage_{k:03d}": v.max().item() + for k, v in enumerate(gathered_memory_consumption_percentage) + } + ) + + wandb_info[f"{self.name}_mean/memory_consumption_gb_mean"] = ( + torch.stack(gathered_memory_consumption).mean().item() # [world_size,num_iters] + ) + wandb_info[f"{self.name}_mean/memory_consumption_percentage_mean"] = ( + torch.stack(gathered_memory_consumption_percentage).mean().item() # [world_size,num_iters] + ) + wandb_info[f"{self.name}_max/memory_consumption_gb_max"] = ( + torch.stack(gathered_memory_consumption).max().item() # [world_size,num_iters] + ) + wandb_info[f"{self.name}_max/memory_consumption_percentage_max"] = ( + torch.stack(gathered_memory_consumption_percentage).max().item() # [world_size,num_iters] + ) + + # Identify slowest rank based on mean time + slowest_dataloading_rank_id = torch.argmax(rank_means) # [] + max_dataloading = torch.max(rank_means) # [] + + # Calculate sum of max times across all iterations (new metric) + # Max across ranks for each iteration (dim=0) + max_per_iter = torch.max(all_times, dim=0).values # [num_iters] + sum_of_max_times = torch.sum(max_per_iter).item() / iteration_count + + wandb_info.update( + { + "slowest_rank/slowest_dataloading_rank": slowest_dataloading_rank_id.item(), + "slowest_rank/slowest_dataloading_time": max_dataloading.item(), + "slowest_rank/sum_of_max_dataloading_time_per_iteration": sum_of_max_times, + } + ) + + # 1. Gather and log stream throughput and watchdog reconnect stats for `stream_throughput` metrics + self._gather_and_log_stream_throughput(wandb_info) + + # Only all_gather_object to get name indices (dataset names, aug-step names, worker-balance keys) across all ranks + # Later methods 2-4 use efficient all_gather_tensor to gather tensor data, then compute statistics and log metrics + ds_index, aug_index, dswid_index = self._discover_name_indices() + + # 2.Gather and log per-dataset dataloading wait times for `dl_wait_time_per_dataset` metrics + self._gather_and_log_per_dataset_time(wandb_info, ds_index) + + # 3. Gather and log per-dataset sampling stats for `dl_packing_stats` metrics + self._gather_and_log_packing_stats(wandb_info, ds_index) + + # 4. Gather and log worker timing metrics for `dl_worker_batch_time`, `dl_worker_balance_per_dataset`, `dl_worker_augmentation` metrics + self._gather_and_log_worker_timing(wandb_info, dswid_index, aug_index) + + if wandb.run: + wandb.log(wandb_info, step=iteration) + + if self.save_s3 and distributed.is_rank0(): + easy_io.dump( + wandb_info, + f"s3://rundir/{self.name}/iter_{iteration:09d}.yaml", + ) + + def _discover_name_indices( + self, + ) -> tuple[dict[str, int], dict[str, int], dict[str, int]]: + """Discover the global union of dataset, aug-step, and worker-balance names. + + Performs a single ``all_gather_object`` call to exchange short string + lists across all ranks and returns deterministic index mappings. + + Returns: + ds_index: ``{dataset_name: col_idx}`` for per-dataset tensors. + aug_index: ``{step_name: col_idx}`` for augmentation step tensors. + dswid_index: ``{"ds|wid": col_idx}`` for worker-balance tensors. + """ + local_ds_names: set[str] = set(self.dataloading_time_per_dataset.keys()) + for key_dict in self._dataset_scalar_stats.values(): + local_ds_names.update(key_dict.keys()) + for key_dict in self._dataset_list_stats.values(): + local_ds_names.update(key_dict.keys()) + + local_names = { + "datasets": sorted(local_ds_names), + "aug_steps": sorted(self._worker_aug_step_times.keys()), + "ds_wid": sorted(f"{ds}|{wid}" for ds, wid in self._worker_times_by_ds_wid.keys()), + } + all_names: list[dict] = [{} for _ in range(dist.get_world_size())] # len=world_size + dist.all_gather_object(all_names, local_names) + + union_ds = sorted({n for r in all_names for n in r.get("datasets", [])}) + ds_index = {name: i for i, name in enumerate(union_ds)} + + union_aug = sorted({n for r in all_names for n in r.get("aug_steps", [])}) + aug_index = {name: i for i, name in enumerate(union_aug)} + + union_dswid = sorted({n for r in all_names for n in r.get("ds_wid", [])}) + dswid_index = {name: i for i, name in enumerate(union_dswid)} + + return ds_index, aug_index, dswid_index + + def _gather_and_log_per_dataset_time(self, wandb_info: dict, ds_index: dict[str, int]) -> None: + """Gather per-dataset dataloading wait times via ``all_gather_tensor``.""" + N = len(ds_index) + if N == 0: + self.dataloading_time_per_dataset = {} + return + + local_ds_time = torch.full((N,), float("nan"), dtype=torch.float64).cuda() # [num_datasets] + for ds, times in self.dataloading_time_per_dataset.items(): + if ds in ds_index: + local_ds_time[ds_index[ds]] = sum(times) / len(times) + + all_ds_time = self._gather_list_stats(local_ds_time) # [world_size, num_datasets] + for ds, i in ds_index.items(): + col = all_ds_time[:, i] # [world_size] + valid = col[~col.isnan()] # [<=world_size] + if len(valid) > 0: + wandb_info[f"dl_wait_time_per_dataset/{ds}_mean"] = valid.mean().item() + wandb_info[f"dl_wait_time_per_dataset/{ds}_max"] = valid.max().item() + + self.dataloading_time_per_dataset = {} + + def _gather_and_log_packing_stats(self, wandb_info: dict, ds_index: dict[str, int]) -> None: + """Gather packing diagnostics via ``all_gather_tensor``, driven by ``_PackingMetrics.STATS_SPEC``.""" + _STATS = "dl_packing_stats" + N = len(ds_index) + if N == 0: + self._dataset_scalar_stats = defaultdict(lambda: defaultdict(int)) + self._dataset_list_stats = defaultdict(lambda: defaultdict(list)) + return + + for batch_key, wandb_suffix, _ in _PackingMetrics.STATS_SPEC: + if batch_key == "_num_tokens": + # Token fraction: gather per-rank token sums, compute each dataset's share of total + local_v = torch.zeros(N, dtype=torch.float64).cuda() # [num_datasets] + for ds, i in ds_index.items(): + local_v[i] = self._dataset_scalar_stats.get(batch_key, {}).get(ds, 0) + all_v = self._gather_list_stats(local_v) # [world_size, num_datasets] + global_tokens = all_v.sum(dim=0) # [num_datasets] + total = global_tokens.sum().item() + for ds, i in ds_index.items(): + wandb_info[f"{_STATS}/{ds}_{wandb_suffix}"] = global_tokens[i].item() / total if total > 0 else 0.0 + + elif batch_key == "_dropped_count": + # Dropped samples: gather per-rank counts, report global total per dataset + local_v = torch.zeros(N, dtype=torch.float64).cuda() # [num_datasets] + for ds, i in ds_index.items(): + local_v[i] = self._dataset_scalar_stats.get(batch_key, {}).get(ds, 0) + all_v = self._gather_list_stats(local_v) # [world_size, num_datasets] + for ds, i in ds_index.items(): + wandb_info[f"{_STATS}/{ds}_{wandb_suffix}_total"] = int(all_v[:, i].sum().item()) + + else: + # Per-batch distributions (_num_samples, _from_buffer, _from_workers, _buffer_size). + # Each rank packs [count, sum, min, max]; reduce to weighted global mean/min/max. + local_t = torch.full( + (N, _AGG_COLS), float("nan"), dtype=torch.float64 + ).cuda() # [num_datasets, _AGG_COLS] + for ds, i in ds_index.items(): + vals = self._dataset_list_stats.get(batch_key, {}).get(ds, []) + if vals: + local_t[i] = torch.tensor([len(vals), sum(vals), min(vals), max(vals)], dtype=torch.float64) + all_t = self._gather_list_stats(local_t) # [world_size, num_datasets, _AGG_COLS] + for ds, i in ds_index.items(): + result = self._reduce_agg_column(all_t[:, i, :]) + if result: + mean_val, min_val, max_val = result + wandb_info[f"{_STATS}/{ds}_{wandb_suffix}_mean"] = mean_val + wandb_info[f"{_STATS}/{ds}_{wandb_suffix}_min"] = min_val + wandb_info[f"{_STATS}/{ds}_{wandb_suffix}_max"] = max_val + + self._dataset_scalar_stats = defaultdict(lambda: defaultdict(int)) + self._dataset_list_stats = defaultdict(lambda: defaultdict(list)) + + def _gather_and_log_stream_throughput(self, wandb_info: dict) -> None: + """Gather stream throughput and watchdog reconnect stats via IPC files.""" + if not ENABLE_STREAM_WANDB: + return + + tp_keys = ["MBps"] + if WATCHDOG_ENABLED: + tp_keys.append("watchdog_reconnects") + tp_stats = collect_throughput_ipc_stats() + local_tp = torch.tensor([tp_stats.get(k, 0.0) for k in tp_keys]).cuda() # [num_metrics] + gathered_tp = distributed.all_gather_tensor(local_tp) # list of [num_metrics], len=world_size + all_tp = torch.stack(gathered_tp) # [world_size, num_metrics] + + for ki, k in enumerate(tp_keys): + col = all_tp[:, ki] # [world_size] + wandb_info[f"stream_throughput/{k}_mean"] = col.mean().item() + wandb_info[f"stream_throughput/{k}_min"] = col.min().item() + wandb_info[f"stream_throughput/{k}_max"] = col.max().item() + if k == "watchdog_reconnects": + wandb_info[f"stream_throughput/{k}_sum"] = col.sum().item() + + mbps_col = all_tp[:, 0] # [world_size] + slowest_throughput_rank = mbps_col.argmin().item() + wandb_info["slowest_rank/slowest_stream_throughput_rank"] = slowest_throughput_rank + + @staticmethod + def _gather_list_stats(local: torch.Tensor) -> torch.Tensor: + """all_gather_tensor + stack, returning [world_size, *local.shape].""" + return torch.stack(distributed.all_gather_tensor(local)) + + @staticmethod + def _reduce_agg_column(col: torch.Tensor) -> tuple[float, float, float] | None: + """From a [world_size, _AGG_COLS] slice, return (mean, min, max) or None if empty. + + Each row is [count, sum, min, max] from one rank. Rows with NaN count + are ranks that had no data for this key. + + Used for metrics where each rank accumulates a variable-length list of + values (e.g. samples_per_batch, buffer_size, per-aug-step times) and we + need a correct weighted global mean rather than a simple average of + per-rank means. The sum/count columns make this possible. + + Callers: ``_gather_and_log_packing_stats`` (list-type metrics) and + ``_gather_and_log_worker_timing`` (per-aug-step breakdown). + """ + valid = col[~col[:, _AGG_COUNT].isnan()] + if len(valid) == 0: + return None + total_count = valid[:, _AGG_COUNT].sum().item() + total_sum = valid[:, _AGG_SUM].sum().item() + if total_count == 0: + return None + return ( + total_sum / total_count, + valid[:, _AGG_MIN].min().item(), + valid[:, _AGG_MAX].max().item(), + ) + + def _gather_and_log_worker_timing( + self, wandb_info: dict, dswid_index: dict[str, int], aug_index: dict[str, int] + ) -> None: + """Gather worker timing from all ranks and log percentile metrics. + + All metrics here are worker-side measurements — time spent inside + DataLoader worker processes producing batches. This is different from + DetailedDataLoadingSpeedMonitor or dl_wait_time_per_dataset/ metrics which measure main-process wall-clock time, + This can help identify if the bottleneck is in the dataloader worker processes or in the main process, + for example waiting for a packed output batch from the JointDataLoader + + Logged metrics: + Section 1 – dl_worker_batch_time/ + Every individual batch time from every worker from every rank, all + thrown into one pool. One data point = one batch produced by one + worker at one step. Computes p50/p90/p99/max/mean of that pool. + Answers: What is the tail latency to produce a batch? + + Section 2 – dl_worker_balance_per_dataset/ + First computes each worker's average batch time over the logging + window. One data point = one worker's mean over several batches. + Then gathers these per-worker averages across all ranks, grouped by + dataset. Computes mean/std/min/max of those averages. + Answers: Are some workers consistently slower than others within + each sub-dataloader? + + Section 3 – dl_worker_augmentation/ + Unified augmentation profiling. Contains: + - total_aug_mean|min|max – total augmentation time per batch + - total_io_mean|min|max – I/O time per batch (batch_time minus aug_time) + - aug_fraction_mean, io_fraction_mean – what fraction of batch time is spent in augmentation vs I/O + - aug_steps/{StepName}_mean|min|max – per-augmentor-step breakdown + (e.g. VideoParsingWithFullFrames for video decode, + TextTokenizerTransform for text tokenization). + All use mean/min/max globally across all ranks. + Answers: Is the bottleneck in augmentations or downloads, and + which augmentor step dominates? + + Note: dl_packing_stats/ is logged from on_training_step_end (not here). + It reports token_fraction, samples_per_batch, from_buffer, from_workers, buffer_size, and dropped_total per dataset — useful for tuning num_workers/batch_size/prefetch per dataloader. + """ + if not self._worker_batch_times: + self._worker_aug_times = [] + self._worker_io_times = [] + self._worker_aug_step_times = defaultdict(list) + self._worker_times_by_ds_wid = defaultdict(list) + return + + _PERCENTILES = [0.50, 0.90, 0.99] + _PNAMES = ["p50", "p90", "p99"] + + # Gather raw batch times across all ranks + local_bt = torch.tensor(self._worker_batch_times, dtype=torch.float32).cuda() # [num_batches_local] + gathered_bt = distributed.all_gather_tensor(local_bt) # list of [num_batches_local], len=world_size + all_bt = torch.cat(gathered_bt) # [num_batches_all_ranks] + + # Section 1: global batch time percentiles + _BATCH_PREFIX = "dl_worker_batch_time" + for pval, pname in zip(_PERCENTILES, _PNAMES): + wandb_info[f"{_BATCH_PREFIX}/{pname}"] = all_bt.quantile(pval).item() + wandb_info[f"{_BATCH_PREFIX}/max"] = all_bt.max().item() + wandb_info[f"{_BATCH_PREFIX}/mean"] = all_bt.mean().item() + + # Section 2: per-dataloader worker balance + # Each rank fills its (dataset, worker_id) slots with that worker's + # mean batch time; NaN marks absent slots. After all_gather we group + # by dataset and compute cross-rank statistics. + + _BALANCE_PREFIX = "dl_worker_balance_per_dataset" + if dswid_index: + N_dswid = len(dswid_index) + local_pw = torch.full((N_dswid,), float("nan"), dtype=torch.float64).cuda() # [num_ds_worker_pairs] + for (ds_name, wid), ts in self._worker_times_by_ds_wid.items(): + key = f"{ds_name}|{wid}" + if key in dswid_index: + local_pw[dswid_index[key]] = sum(ts) / len(ts) + + all_pw = self._gather_list_stats(local_pw) # [world_size, num_ds_worker_pairs] + + # Pass 1: collect all valid per-worker means, grouped by dataset + ds_worker_vals: dict[str, list[float]] = defaultdict(list) + for key, idx in dswid_index.items(): + ds_name = key.rsplit("|", 1)[0] + col = all_pw[:, idx] # [world_size] + valid = col[~col.isnan()] # [<=world_size] + ds_worker_vals[ds_name].extend(valid.tolist()) + + # Pass 2: log per-dataset worker balance statistics + for ds_name in sorted(ds_worker_vals): + pw_means = ds_worker_vals[ds_name] + if not pw_means: + continue + pw_t = torch.tensor(pw_means, dtype=torch.float32).cuda() # [num_workers_for_ds] + wandb_info[f"{_BALANCE_PREFIX}/{ds_name}_mean"] = pw_t.mean().item() + wandb_info[f"{_BALANCE_PREFIX}/{ds_name}_std"] = pw_t.std().item() + wandb_info[f"{_BALANCE_PREFIX}/{ds_name}_min"] = pw_t.min().item() + wandb_info[f"{_BALANCE_PREFIX}/{ds_name}_max"] = pw_t.max().item() + + # Section 3: augmentation profiling (total aug/io + per-step breakdown) + _AUG_PREFIX = "dl_worker_augmentation" + + if self._worker_aug_times: + local_aug = torch.tensor(self._worker_aug_times, dtype=torch.float32).cuda() # [num_batches_local] + all_aug = torch.cat(distributed.all_gather_tensor(local_aug)) # [num_batches_all_ranks] + wandb_info[f"{_AUG_PREFIX}/total_aug_mean"] = all_aug.mean().item() + wandb_info[f"{_AUG_PREFIX}/total_aug_min"] = all_aug.min().item() + wandb_info[f"{_AUG_PREFIX}/total_aug_max"] = all_aug.max().item() + + if self._worker_io_times: + local_io = torch.tensor(self._worker_io_times, dtype=torch.float32).cuda() # [num_batches_local] + all_io = torch.cat(distributed.all_gather_tensor(local_io)) # [num_batches_all_ranks] + wandb_info[f"{_AUG_PREFIX}/total_io_mean"] = all_io.mean().item() + wandb_info[f"{_AUG_PREFIX}/total_io_min"] = all_io.min().item() + wandb_info[f"{_AUG_PREFIX}/total_io_max"] = all_io.max().item() + + if self._worker_aug_times and self._worker_batch_times: + aug_fracs = [ + a / b for a, b in zip(self._worker_aug_times, self._worker_batch_times) if b > 0 + ] # [num_valid_batches_local] + if aug_fracs: + local_fracs = torch.tensor(aug_fracs, dtype=torch.float32).cuda() # [num_valid_batches_local] + all_fracs = torch.cat(distributed.all_gather_tensor(local_fracs)) # [num_valid_batches_all_ranks] + wandb_info[f"{_AUG_PREFIX}/aug_fraction_mean"] = all_fracs.mean().item() + wandb_info[f"{_AUG_PREFIX}/io_fraction_mean"] = 1.0 - all_fracs.mean().item() + + # Per-augmentor-step breakdown (converted to all_gather_tensor) + if aug_index: + N_aug = len(aug_index) + local_aug_steps = torch.full( + (N_aug, _AGG_COLS), float("nan"), dtype=torch.float64 + ).cuda() # [num_aug_steps, _AGG_COLS] + for step_name, ts in self._worker_aug_step_times.items(): + if step_name in aug_index and ts: + local_aug_steps[aug_index[step_name]] = torch.tensor( + [len(ts), sum(ts), min(ts), max(ts)], dtype=torch.float64 + ) + + all_aug_steps = self._gather_list_stats(local_aug_steps) # [world_size, num_aug_steps, _AGG_COLS] + for step_name, idx in aug_index.items(): + result = self._reduce_agg_column(all_aug_steps[:, idx, :]) + if result: + mean_val, min_val, max_val = result + wandb_info[f"{_AUG_PREFIX}/aug_steps/{step_name}_mean"] = mean_val + wandb_info[f"{_AUG_PREFIX}/aug_steps/{step_name}_min"] = min_val + wandb_info[f"{_AUG_PREFIX}/aug_steps/{step_name}_max"] = max_val + + self._worker_batch_times = [] + self._worker_aug_times = [] + self._worker_io_times = [] + self._worker_aug_step_times = defaultdict(list) + self._worker_times_by_ds_wid = defaultdict(list) diff --git a/cosmos_framework/callbacks/device_monitor.py b/cosmos_framework/callbacks/device_monitor.py new file mode 100644 index 0000000..65fd1d4 --- /dev/null +++ b/cosmos_framework/callbacks/device_monitor.py @@ -0,0 +1,189 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import os +from typing import Any, Dict, List, Tuple + +import pandas as pd +import psutil +import pynvml +import torch +import wandb + +from cosmos_framework.callbacks.every_n import EveryN +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.trainer import ImaginaireTrainer +from cosmos_framework.utils import distributed, log +from cosmos_framework.utils.easy_io import easy_io + + +def log_prof_data( + data_list: List[Dict[str, Any]], + iteration: int, +) -> Tuple[pd.DataFrame]: + # Create a table to log data with rank information + columns = ["iteration", "rank"] + list(data_list[0].keys()) + data = [] + + # Initialize dictionaries to store min and max values for each metric + min_values = {key: float("inf") for key in columns[2:]} + max_values = {key: float("-inf") for key in columns[2:]} + sum_values = {key: 0.0 for key in columns[2:]} + + count = 0 + + for _rank, prof_data in enumerate(data_list): + row = [iteration, _rank] + [prof_data[key] for key in columns[2:]] + data.append(row) + count += 1 + + # Update min, max, and sum values + for key in columns[2:]: + min_values[key] = min(min_values[key], prof_data[key]) + max_values[key] = max(max_values[key], prof_data[key]) + sum_values[key] += prof_data[key] + + # Calculate average values + avg_values = {key: sum_values[key] / count for key in columns[2:]} + + df = pd.DataFrame(data, columns=columns) + summary_df = pd.DataFrame({"Avg": avg_values, "Max": max_values, "Min": min_values}) + + if wandb.run: + # Log the table + table = wandb.Table(dataframe=df) + wandb.log({"DeviceMonitor/prof_data": table}, step=iteration) + + # Log summary statistics + summary = {} + for key in columns[2:]: + summary[f"DeviceMonitor/min_{key}"] = min_values[key] + summary[f"DeviceMonitor/max_{key}"] = max_values[key] + summary[f"DeviceMonitor/avg_{key}"] = avg_values[key] + + wandb.log(summary, step=iteration) + return df, summary_df + + +class DeviceMonitor(EveryN): + """ + A callback to monitor device (CPU/GPU) usage and log it at regular intervals. + + Args: + every_n (int, optional): The frequency at which the callback is invoked. Defaults to 200. + step_size (int, optional): The step size for the callback. Defaults to 1. + save_s3 (bool, optional): Whether to save the monitoring data to S3. Defaults to False. + """ + + def __init__( + self, + every_n: int = 200, + step_size: int = 1, + save_s3: bool = False, + upload_every_n_mul: int = 1, + log_memory_detail: bool = True, + ): + super().__init__(every_n=every_n, step_size=step_size) + self.name = self.__class__.__name__ + self.save_s3 = save_s3 + self.s3_save_fp = f"s3://rundir/{self.name}" + self.upload_every_n = upload_every_n_mul * every_n + + self.log_memory_detail = log_memory_detail + + def on_train_start(self, model, iteration=0): + torch.cuda.reset_peak_memory_stats() + self.world_size = distributed.get_world_size() + self.rank = distributed.get_rank() + config_job = self.config.job + self.local_dir = f"{config_job.path_local}/{self.name}" + if self.rank == 0: + os.makedirs(self.local_dir, exist_ok=True) + log.info(f"{self.name} callback: local_dir: {self.local_dir}") + + local_rank = int(os.getenv("LOCAL_RANK", 0)) + self.handle = pynvml.nvmlDeviceGetHandleByIndex(local_rank) + + def every_n_impl( + self, + trainer: ImaginaireTrainer, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int, + ) -> None: + cur_process = psutil.Process(os.getpid()) + # cur_process.children(recursive=True) can crash if the dataloader is constantly creating and destroying processes (e.g. calling FFmpeg). + try: + cpu_memory_usage = sum(p.memory_info().rss for p in [cur_process] + cur_process.children(recursive=True)) + except Exception as e: # e.g. psutil.NoSuchProcess + log.warning(f"Failed to get CPU memory usage with error {e}") + cpu_memory_usage = 0 + cpu_mem_gb = cpu_memory_usage / (1024**3) + + peak_gpu_mem_gb = torch.cuda.max_memory_allocated() / (1024**3) + peak_gpu_mem_reserved_gb = torch.cuda.max_memory_reserved() / (1024**3) + temp = torch.cuda.temperature() + try: + power = torch.cuda.power_draw() + except Exception as e: + log.warning(f"Failed to get power draw with error {e}") + power = 0 + util = torch.cuda.utilization() + clock = torch.cuda.clock_rate() + + memory_info = pynvml.nvmlDeviceGetMemoryInfo(self.handle) + nvml_used_gpu_mem_gb = memory_info.used / (1024**3) + nvml_free_gpu_mem_gb = memory_info.free / (1024**3) + + prof_data = { + "cpu_mem_gb": cpu_mem_gb, + "peak_gpu_mem_gb": peak_gpu_mem_gb, + "peak_gpu_mem_reserved_gb": peak_gpu_mem_reserved_gb, + "nvml_used_gpu_mem_gb": nvml_used_gpu_mem_gb, + "nvml_free_gpu_mem_gb": nvml_free_gpu_mem_gb, + "temp": temp, + "power": power, + "util": util, + "clock": clock, + } + + data_list = [prof_data] * self.world_size + # this is blocking by default + if self.world_size > 1: + torch.distributed.all_gather_object(data_list, prof_data) + torch.distributed.barrier() + + df, summary_df = log_prof_data(data_list, iteration) + if self.save_s3 and self.rank == 0: + global_step = iteration // self.step_size + should_run = global_step % self.upload_every_n == 0 + if should_run: + df.to_csv(os.path.join(self.local_dir, f"prof_data_{iteration:09d}.csv"), index=False) + summary_df.to_csv(os.path.join(self.local_dir, f"summary_{iteration:09d}.csv"), index=True) + easy_io.copyfile_from_local( + os.path.join(self.local_dir, f"prof_data_{iteration:09d}.csv"), + os.path.join(self.s3_save_fp, f"prof_data_{iteration:09d}.csv"), + ) + easy_io.copyfile_from_local( + os.path.join(self.local_dir, f"summary_{iteration:09d}.csv"), + os.path.join(self.s3_save_fp, f"summary_{iteration:09d}.csv"), + ) + if self.rank == 0: + log.info(f"{self.name} Stats:\n{summary_df.to_string()}") + if self.log_memory_detail: + memory_stats = torch.cuda.memory_stats() + if wandb.run: + wandb_memory_info = {f"mem/{key}": memory_stats[key] for key in memory_stats.keys()} + wandb.log(wandb_memory_info, step=iteration) + if self.save_s3: + global_step = iteration // self.step_size + should_run = global_step % self.upload_every_n == 0 + if should_run: + easy_io.dump( + memory_stats, + os.path.join(self.s3_save_fp, f"memory_stats_{iteration:09d}.yaml"), + ) + + torch.cuda.reset_peak_memory_stats() diff --git a/cosmos_framework/callbacks/every_n.py b/cosmos_framework/callbacks/every_n.py new file mode 100644 index 0000000..0fb675d --- /dev/null +++ b/cosmos_framework/callbacks/every_n.py @@ -0,0 +1,73 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from abc import abstractmethod +from typing import Optional + +import torch + +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.trainer import ImaginaireTrainer +from cosmos_framework.utils import distributed, log +from cosmos_framework.utils.callback import Callback + + +class EveryN(Callback): + def __init__( + self, + every_n: Optional[int] = None, + step_size: int = 1, + barrier_after_run: bool = True, + run_at_start: bool = False, + ) -> None: + """Constructor for `EveryN`. + + Args: + every_n (int): Frequency with which callback is run during training. + step_size (int): Size of iteration step count. Default 1. + barrier_after_run (bool): Whether to have a distributed barrier after each execution. Default True, to avoid timeouts. + run_at_start (bool): Whether to run at the beginning of training. Default False. + """ + self.every_n = every_n + if self.every_n == 0: + log.warning( + f"every_n is set to 0. Callback {self.__class__.__name__} will be invoked only once in the beginning of the training. Calls happens on_training_step_end will be skipped." + ) + + self.step_size = step_size + self.barrier_after_run = barrier_after_run + self.run_at_start = run_at_start + + def on_training_step_end( + self, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int = 0, + ) -> None: + # every_n = 0 is a special case which means every_n_impl will be called only once in the beginning of the training + if self.every_n != 0: + trainer = self.trainer + global_step = iteration // self.step_size + should_run = (iteration == 1 and self.run_at_start) or ( + global_step % self.every_n == 0 + ) # (self.every_n - 1) + if should_run: + log.debug(f"Callback {self.__class__.__name__} fired on train_batch_end step {global_step}") + self.every_n_impl(trainer, model, data_batch, output_batch, loss, iteration) + log.debug(f"Callback {self.__class__.__name__} finished on train_batch_end step {global_step}") + # add necessary barrier to avoid timeout + if self.barrier_after_run: + distributed.barrier() + + @abstractmethod + def every_n_impl( + self, + trainer: ImaginaireTrainer, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int, + ) -> None: ... diff --git a/cosmos_framework/callbacks/every_n_draw_sample.py b/cosmos_framework/callbacks/every_n_draw_sample.py new file mode 100644 index 0000000..baf1ffc --- /dev/null +++ b/cosmos_framework/callbacks/every_n_draw_sample.py @@ -0,0 +1,426 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import math +import os +from contextlib import nullcontext +from functools import partial +from typing import List, Optional + +import numpy as np +import torch +import torch.distributed as dist +import torch.nn.functional as F +import torchvision +import torchvision.transforms.functional as torchvision_F +import wandb +from einops import rearrange + +from cosmos_framework.callbacks.every_n import EveryN +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.utils import distributed, log, misc +from cosmos_framework.utils.easy_io import easy_io +from cosmos_framework.tools.visualize.video import save_img_or_video +from cosmos_framework.utils.vfm.data_utils import slice_data_batch + + +def resize_image(image: torch.Tensor, size: int = 1024) -> torch.Tensor: + """ + Resize the image to the given size. This is done so that wandb can display the image correctly. + """ + _, h, w = image.shape + ratio = size / max(h, w) + new_h, new_w = int(ratio * h), int(ratio * w) + return torchvision_F.resize(image, (new_h, new_w)) + + +def is_primitive(value): + return isinstance(value, (int, float, str, bool, type(None))) + + +def convert_to_primitive(value): + if isinstance(value, (list, tuple)): + return [convert_to_primitive(v) for v in value if is_primitive(v) or isinstance(v, (list, dict))] + elif isinstance(value, dict): + return {k: convert_to_primitive(v) for k, v in value.items() if is_primitive(v) or isinstance(v, (list, dict))} + elif is_primitive(value): + return value + else: + return "non-primitive" # Skip non-primitive types + + +def pad_images_and_cat(images: List[torch.Tensor], max_w: int, max_h: int, t_crop: int = 1) -> torch.Tensor: + """ + Pad images to a common size and concatenate them along the batch dimension. + + This function is needed because different samples in a batch can have different resolutions. + To create a unified visualization grid, all images must be padded to the same dimensions. + Images are center-padded to preserve their visual content in the middle. + + Args: + images: List of image/video tensors with shape [B, C, T, H, W]. + max_w: Target width to pad all images to. + max_h: Target height to pad all images to. + t_crop: Number of temporal frames to keep for videos. If > 1 and the image + has more than 1 frame, only the first t_crop frames are retained. + + Returns: + Concatenated tensor of padded images with shape [total_B, C, T, max_h, max_w]. + """ + padded_images = [] + for image in images: + # Pad the image to the center + padding_h = (max_h - image.shape[-2]) // 2 + padding_w = (max_w - image.shape[-1]) // 2 + padded_image = torch.nn.functional.pad( + image, (padding_w, max_w - image.shape[-1] - padding_w, padding_h, max_h - image.shape[-2] - padding_h) + ) # [B,C,T,max_h,max_w] + # Handle video case + if image.shape[2] > 1 and t_crop > 1: + padded_image = padded_image[:, :, 0:t_crop, :, :] + + padded_images.append(padded_image) + return torch.cat(padded_images, dim=0) # [total_B,C,T,max_h,max_w] (total_B = sum of batch dims) + + +class EveryNDrawSample(EveryN): + """ + This callback sample condition inputs from training data, run inference and save the results to wandb and s3. + + Args: + every_n (int): The frequency at which the callback is invoked. + step_size (int, optional): The step size for the callback. Defaults to 1. + n_viz_sample (int, optional): for each batch, min(n_viz_sample, batch_size) samples will be saved to wandb. Defaults to 3. + n_sample_to_save (int, optional): number of samples to save. The actual number of samples to save is min(n_sample_to_save, data parallel instances). Defaults to 128. + num_sampling_step (int, optional): number of sampling steps. Defaults to 35. + guidance (List[float], optional): guidance scale. Defaults to [0.0, 3.0, 7.0]. + do_x0_prediction (bool, optional): whether to do x0 prediction. Defaults to True. + n_sigmas_for_x0_prediction (int, optional): number of sigmas to use for x0 prediction. Defaults to 4. + save_s3 (bool, optional): whether to save to s3. Defaults to False. + is_ema (bool, optional): whether the callback is run for ema model. Defaults to False. + use_negative_prompt (bool, optional): whether to use negative prompt. Defaults to False. + fps (int, optional): frames per second when saving the video. Defaults to 16. + """ + + def __init__( + self, + every_n: int, + step_size: int = 1, + n_viz_sample: int = 2, + n_sample_to_save: int = 128, + num_sampling_step: int = 35, + guidance: List[float] = [0.0, 3.0, 7.0], + do_x0_prediction: bool = True, + n_sigmas_for_x0_prediction: int = 4, + save_s3: bool = False, + save_local: bool = False, + is_ema: bool = False, + use_negative_prompt: bool = False, + prompt_type: str = "t5_xxl", + fps: int = 16, + run_at_start: bool = False, + ): + # s3: # files: min(n_sample_to_save, data instance) # per file: min(batch_size, n_viz_sample) + # wandb: 1 file, # per file: min(batch_size, n_viz_sample) + super().__init__(every_n, step_size, run_at_start=run_at_start) + + self.n_viz_sample = n_viz_sample + self.n_sample_to_save = n_sample_to_save + self.save_s3 = save_s3 + self.save_local = save_local + self.do_x0_prediction = do_x0_prediction + self.n_sigmas_for_x0_prediction = n_sigmas_for_x0_prediction + self.name = self.__class__.__name__ + self.is_ema = is_ema + self.use_negative_prompt = use_negative_prompt + self.prompt_type = prompt_type + self.guidance = guidance + self.num_sampling_step = num_sampling_step + self.rank = distributed.get_rank() + self.fps = fps + + def on_train_start(self, model: ImaginaireModel, iteration: int = 0) -> None: + config_job = self.config.job + self.local_dir = f"{config_job.path_local}/{self.name}" + if distributed.get_rank() == 0: + os.makedirs(self.local_dir, exist_ok=True) + log.info(f"Callback: local_dir: {self.local_dir}") + + self.data_parallel_id = self.rank + + @misc.timer("EveryNDrawSample: x0") + @torch.no_grad() + def x0_pred(self, trainer, model, data_batch, output_batch, loss, iteration): + tag = "ema" if self.is_ema else "reg" + + log.debug("starting data and condition model", rank0_only=False) + + + data_clean = model.get_data_and_condition(data_batch) + raw_data = data_clean.raw_state_vision + x0 = data_clean.x0_tokens_vision + + # Handle model parallelism if available (legacy models) + if hasattr(model, "broadcast_split_for_model_parallelsim"): + _, condition, x0, _ = model.broadcast_split_for_model_parallelsim(None, None, x0, None) + + log.debug("done data and condition model", rank0_only=False) + batch_size = len(x0) + sigmas = np.exp( + np.linspace( + math.log(model.sde.sigma_min), math.log(model.sde.sigma_max), self.n_sigmas_for_x0_prediction + 1 + )[1:] + ) + + to_show = [] + generator = torch.Generator(device="cuda") + generator.manual_seed(0) + random_noise = torch.randn(*x0.shape, generator=generator, **model.tensor_kwargs) # same shape as x0 + _ones = torch.ones(batch_size, **model.tensor_kwargs) # [B] + mse_loss_list = [] + for _, sigma in enumerate(sigmas): + x_sigma = sigma * random_noise + x0 + log.debug(f"starting denoising {sigma}", rank0_only=False) + sample = model.denoise(x_sigma, None).x0 + log.debug(f"done denoising {sigma}", rank0_only=False) + mse_loss = distributed.dist_reduce_tensor(F.mse_loss(sample, x0)) + mse_loss_list.append(mse_loss) + + if hasattr(model, "decode"): + sample = model.decode(sample) + to_show.append(sample.float().cpu()) + to_show.append( + raw_data.float().cpu(), + ) + + base_fp_wo_ext = f"{tag}_ReplicateID{self.data_parallel_id:04d}_x0_Iter{iteration:09d}" + + local_path = self.run_save(to_show, batch_size, base_fp_wo_ext) + return local_path, torch.tensor(mse_loss_list).cuda(), sigmas # [N_sigmas] + + @torch.no_grad() + def every_n_impl(self, trainer, model, data_batch, output_batch, loss, iteration): + if self.is_ema: + if not model.config.ema.enabled: + return + context = partial(model.ema_scope, "every_n_sampling") + else: + context = nullcontext + + tag = "ema" if self.is_ema else "reg" + sample_counter = getattr(trainer, "sample_counter", iteration) + batch_info = { + "data": { + k: convert_to_primitive(v) + for k, v in data_batch.items() + if is_primitive(v) or isinstance(v, (list, dict)) + }, + "sample_counter": sample_counter, + "iteration": iteration, + } + if self.save_s3 and self.data_parallel_id < self.n_sample_to_save: + easy_io.dump( + batch_info, + f"s3://rundir/{self.name}/Iter{iteration:09d}/BatchInfo_ReplicateID{self.data_parallel_id:04d}_Iter{iteration:09d}.json", + ) + + log.debug("entering, every_n_impl", rank0_only=False) + with context(): + if self.do_x0_prediction: + log.debug("entering, x0_pred", rank0_only=False) + x0_img_fp, mse_loss, sigmas = self.x0_pred( + trainer, + model, + data_batch, + output_batch, + loss, + iteration, + ) + log.debug("done, x0_pred", rank0_only=False) + if self.save_s3 and self.rank == 0: + easy_io.dump( + { + "mse_loss": mse_loss.tolist(), + "sigmas": sigmas.tolist(), + "iteration": iteration, + }, + f"s3://rundir/{self.name}/{tag}_MSE_Iter{iteration:09d}.json", + ) + + log.debug("entering, sample", rank0_only=False) + sample_img_fp = self.sample( + trainer, + model, + data_batch, + output_batch, + loss, + iteration, + ) + log.debug("done, sample", rank0_only=False) + + log.debug("waiting for all ranks to finish", rank0_only=False) + dist.barrier() + if wandb.run: + sample_counter = getattr(trainer, "sample_counter", iteration) + data_type = "image" if model.is_image_batch(data_batch) else "video" + tag += f"_{data_type}" + info = { + "trainer/global_step": iteration, + "sample_counter": sample_counter, + } + if self.do_x0_prediction: + info[f"{self.name}/{tag}_x0"] = wandb.Image(x0_img_fp, caption=f"{sample_counter}") + # convert mse_loss to a dict + mse_loss = mse_loss.tolist() + info.update({f"x0_pred_mse_{tag}/Sigma{sigmas[i]:0.5f}": mse_loss[i] for i in range(len(mse_loss))}) + + info[f"{self.name}/{tag}_sample"] = wandb.Image(sample_img_fp, caption=f"{sample_counter}") + wandb.log( + info, + step=iteration, + ) + torch.cuda.empty_cache() + + @misc.timer("EveryNDrawSample: sample") + def sample(self, trainer, model, data_batch, output_batch, loss, iteration): + data_batch = slice_data_batch(data_batch, start=0, limit=self.n_viz_sample) + + tag = "ema" if self.is_ema else "reg" + + # Obtain text embeddings online + text_encoder_config = getattr(model.config, "text_encoder_config", None) + if text_encoder_config is not None and text_encoder_config.compute_online: + text_embeddings = model.text_encoder.compute_text_embeddings_online(data_batch, model.input_caption_key) + data_batch["t5_text_embeddings"] = text_embeddings + data_batch["t5_text_mask"] = torch.ones( + text_embeddings.shape[0], text_embeddings.shape[1], device="cuda" + ) # [B,N_tokens] (all tokens valid) + + data_clean = model.get_data_and_condition(data_batch) + raw_data = data_clean.raw_state_vision + x0 = data_clean.x0_tokens_vision + + # determine the number of visualized samples + n_viz_sample = min(self.n_viz_sample, data_clean.batch_size) + + # Check if this is a multi-item vision batch (image editing) + num_items = data_clean.num_vision_items_per_sample + is_multi_item = num_items is not None + + if is_multi_item: + # Image editing: raw_data is flat [src1, tgt1, src2, tgt2, ...]. + # Split into per-sample condition (source) and GT target images. + condition_images: list[torch.Tensor] = [] + gt_target_images: list[torch.Tensor] = [] + vis_offset = 0 + for sample_idx in range(data_clean.batch_size): + n_vis = num_items[sample_idx] + # First item(s) are condition, last item is generation target + + # but we need to support multiple conditions per sample in the future. Current code + # can handle this without throwing an error. + condition_images.append(raw_data[vis_offset]) # source image (1, C, 1, H, W) + gt_target_images.append(raw_data[vis_offset + n_vis - 1]) # target image (1, C, 1, H, W) + vis_offset += n_vis + + # Use target images for max_w/max_h/t_crop (generated samples match target size) + max_w = max(img.shape[-1] for img in gt_target_images) + max_h = max(img.shape[-2] for img in gt_target_images) + t_crop = min(img.shape[-3] for img in gt_target_images) + else: + max_w = max(image.shape[-1] for image in raw_data) + max_h = max(image.shape[-2] for image in raw_data) + t_crop = min(image.shape[-3] for image in raw_data) + + to_show = [] + + # Row 0 (image editing only): condition (source) images + if is_multi_item: + to_show.append(pad_images_and_cat(condition_images[:n_viz_sample], max_w, max_h, t_crop).float().cpu()) + + for guidance in self.guidance: + sample = model.generate_samples_from_batch( + data_batch, + guidance=guidance, + n_sample=n_viz_sample, + num_steps=self.num_sampling_step, + has_negative_prompt=True if self.use_negative_prompt else False, + seed=list(range(iteration, iteration + n_viz_sample)), + ) + sample_vision = sample["vision"] + assert hasattr(model, "decode") + sample_vision_decoded = [model.decode(sample_vision_i) for sample_vision_i in sample_vision] + assert len(sample_vision_decoded) == n_viz_sample + to_show.append(pad_images_and_cat(sample_vision_decoded, max_w, max_h, t_crop).float().cpu()) + + # Last row: ground truth + if is_multi_item: + # Image editing: show GT target images (not the flat raw_data which mixes src + tgt) + assert len(gt_target_images) == n_viz_sample + to_show.append(pad_images_and_cat(gt_target_images, max_w, max_h, t_crop).float().cpu()) + else: + assert len(raw_data) == n_viz_sample + to_show.append(pad_images_and_cat(raw_data, max_w, max_h, t_crop).float().cpu()) + + base_fp_wo_ext = f"{tag}_ReplicateID{self.data_parallel_id:04d}_Sample_Iter{iteration:09d}" + base_fp_wo_ext = f"Iter{iteration:09d}/{base_fp_wo_ext}" + + batch_size = data_clean.batch_size + local_path = self.run_save(to_show, batch_size, base_fp_wo_ext) + return local_path + + def run_save(self, to_show, batch_size, base_fp_wo_ext) -> Optional[str]: + to_show = (1.0 + torch.stack(to_show, dim=0).clamp(-1, 1)) / 2.0 # [N_rows,B,C,T,H,W] range [0,1] + is_single_frame = to_show.shape[3] == 1 + n_viz_sample = min(self.n_viz_sample, batch_size) + to_show = to_show[:, :n_viz_sample] + + # ! we only save first n_sample_to_save video! + video_grid = rearrange(to_show, "n b c t h w -> c t (n h) (b w)") # [C,T,N_rows*H,B*W] + if self.save_s3 and self.data_parallel_id < self.n_sample_to_save: + save_img_or_video( + video_grid, + f"s3://rundir/{self.name}/{base_fp_wo_ext}", + fps=self.fps, + ) + if self.save_local and self.data_parallel_id < self.n_sample_to_save: + local_video_path = f"{self.local_dir}/{base_fp_wo_ext}" + os.makedirs(os.path.dirname(local_video_path), exist_ok=True) + save_img_or_video(video_grid, local_video_path, fps=self.fps) + + file_base_fp = f"{base_fp_wo_ext}_resize.jpg" + local_path = f"{self.local_dir}/{file_base_fp}" + + if self.rank == 0 and wandb.run: + if is_single_frame: # image case + to_show = rearrange( + to_show[:, :n_viz_sample], + "n b c t h w -> t c (n h) (b w)", + ) # [1,C,N_rows*H,B*W] (t=1 for images) + image_grid = torchvision.utils.make_grid(to_show, nrow=1, padding=0, normalize=False) + # resize so that wandb can handle it + os.makedirs(os.path.dirname(local_path), exist_ok=True) + torchvision.utils.save_image(resize_image(image_grid, 1024), local_path, nrow=1, scale_each=True) + else: + to_show = to_show[:, :n_viz_sample] # [N_rows,B,C,T,H,W] + + # resize 3 frames frames so that we can display them on wandb + _T = to_show.shape[3] + three_frames_list = [0, _T // 2, _T - 1] + to_show = to_show[:, :, :, three_frames_list] # [N_rows,B,C,3,H,W] (3 sampled frames) + log_image_size = 1024 + to_show = rearrange( + to_show, + "n b c t h w -> 1 c (n h) (b t w)", + ) # [1,C,N_rows*H,B*3*W] (t=3 sampled frames) + + os.makedirs(os.path.dirname(local_path), exist_ok=True) + # resize so that wandb can handle it + image_grid = torchvision.utils.make_grid(to_show, nrow=1, padding=0, normalize=False) + os.makedirs(os.path.dirname(local_path), exist_ok=True) + torchvision.utils.save_image( + resize_image(image_grid, log_image_size), local_path, nrow=1, scale_each=True + ) + + return local_path + return None diff --git a/cosmos_framework/callbacks/expert_heatmap.py b/cosmos_framework/callbacks/expert_heatmap.py new file mode 100644 index 0000000..8fb745d --- /dev/null +++ b/cosmos_framework/callbacks/expert_heatmap.py @@ -0,0 +1,108 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import matplotlib.pyplot as plt +import torch +import wandb +from torch.distributed.tensor import DTensor, Partial + +from cosmos_framework.callbacks.every_n import EveryN +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.trainer import ImaginaireTrainer +from cosmos_framework.utils import distributed +from cosmos_framework.model.vfm.vlm.qwen3_vl_moe.qwen3_vl_moe import Qwen3VLMoeTextSparseMoeBlock + + +def compute_expert_heatmap(vfm: torch.nn.Module) -> dict[str, torch.Tensor]: + """ + Compute the heatmap for the MoE blocks in the language model. + + The heatmap is a dictionary with keys set to ["und", "gen"] and values set to + a tensor of shape (num_layers, num_experts). + + Each element of the tensor is the average number of tokens routed to each expert for a + given layer. The sum of the elements in each row should be equal to the average number + of experts per token for the MoE model (config.num_experts_per_tok). + + For dense models, the heatmap is an empty dictionary. + """ + with torch.no_grad(): + num_layers = len(vfm.language_model.model.layers) + + example_dtensor = vfm.language_model.model.layers[0].self_attn.q_proj.weight + if isinstance(example_dtensor, DTensor): + assert hasattr(example_dtensor, "device_mesh") + device_mesh = example_dtensor.device_mesh + else: + device_mesh = None + + expert_heatmaps = {} + for tower in ["und", "gen"]: + expert_heatmaps_per_layer = [] + + for layer_idx in range(num_layers): + layer_module = vfm.language_model.model.layers[layer_idx] + mlp_module = layer_module.mlp if tower == "und" else layer_module.mlp_moe_gen + if isinstance(mlp_module, Qwen3VLMoeTextSparseMoeBlock): + # This is accumulated across all iterations. + total_tokens_per_expert = mlp_module.get_total_tokens_per_expert() + total_tokens = mlp_module.get_total_tokens() + + # Compute the average across all ranks. + assert device_mesh is not None, "MoE models require multiple GPUs." + total_tokens_per_expert = DTensor.from_local( + total_tokens_per_expert, + device_mesh=device_mesh, + placements=[Partial()] * device_mesh.ndim, + ).full_tensor() + total_tokens = DTensor.from_local( + total_tokens, + device_mesh=device_mesh, + placements=[Partial()] * device_mesh.ndim, + ).full_tensor() + + mean_tokens_per_expert = total_tokens_per_expert.float() / total_tokens.float() # [num_experts] + expert_heatmaps_per_layer.append(mean_tokens_per_expert) + + if len(expert_heatmaps_per_layer) > 0: + expert_heatmaps[tower] = torch.stack(expert_heatmaps_per_layer, dim=0) # [num_layers,num_experts] + + return expert_heatmaps + + +class ExpertHeatmap(EveryN): + """ + Plots the expert heatmap for the MoE blocks in the language model. + + Args: + every_n (int): Number of iterations to log the expert heatmap. + """ + + def __init__(self, every_n: int = 1000): + super().__init__(every_n=every_n) + + def every_n_impl( + self, + trainer: ImaginaireTrainer, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int, + ) -> None: + expert_heatmaps = compute_expert_heatmap(model.net) + + if distributed.is_rank0() and wandb.run: + for tower, heatmap in expert_heatmaps.items(): + fig, ax = plt.subplots() + im = ax.imshow(heatmap.cpu().numpy()) + ax.set_xlabel("Experts") + ax.set_ylabel("Layers") + plt.colorbar(im, ax=ax) + wandb.log( + { + f"expert_heatmap/{tower}": fig, + }, + step=iteration, + ) + plt.close(fig) diff --git a/cosmos_framework/callbacks/grad_clip.py b/cosmos_framework/callbacks/grad_clip.py new file mode 100644 index 0000000..151c1bc --- /dev/null +++ b/cosmos_framework/callbacks/grad_clip.py @@ -0,0 +1,325 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import math +from collections import defaultdict + +import torch +import wandb +from torch.distributed.tensor import DTensor + +from cosmos_framework.utils import log +from cosmos_framework.utils.callback import Callback + + +@torch.compile +def _fused_nan_to_num(grads: list[torch.Tensor]) -> None: + """Replace NaN/Inf entries with 0.0 in every floating-point grad in-place. + + The Python-level loop over ``grads`` is wrapped in ``@torch.compile`` so + Inductor can fuse the per-tensor ``nan_to_num`` ops into a single CUDA + kernel. This is NOT the ``torch._foreach_*`` API; it is fusion-via-compile + and depends on the grad list structure (length, dtypes, shapes) staying + stable across iterations so Dynamo can reuse its specialized graph. + """ + grads = [g for g in grads if torch.is_floating_point(g)] + for g in grads: + torch.nan_to_num(g, nan=0.0, posinf=0.0, neginf=0.0, out=g) + + +class _MagnitudeRecord: + def __init__(self) -> None: + self.state: torch.Tensor | None = None + self.iter_count: int = 0 + + def reset(self) -> None: + self.state = None + self.iter_count = 0 + + def update(self, cur_state: torch.Tensor) -> None: + if self.state is None: + self.state = cur_state.detach().clone() + else: + self.state.add_(cur_state) + self.iter_count += 1 + + def get_stat(self) -> float: + if self.state is not None and self.iter_count > 0: + avg_state = (self.state / self.iter_count).item() + else: + avg_state = 0.0 + self.reset() + return avg_state + + +@torch.no_grad() +def _clip_grad( + parameters: list[torch.Tensor], + max_norm: float, + norm_type: float = 2.0, + error_if_nonfinite: bool = False, + foreach: bool | None = None, + return_norm_only: bool = False, +) -> tuple[torch.Tensor, dict[str, torch.Tensor]]: + """ + Clip the gradient norm of an iterable of parameters. + + Gradient norm clipping requires computing the gradient norm over the entire model. + `torch.nn.utils.clip_grad_norm_` only computes gradient norm along DP/FSDP/TP dimensions. + We need to manually reduce the gradient norm across PP stages. + See https://github.com/pytorch/torchtitan/issues/596 for details. + + Params are grouped by their ``device_mesh`` (by mesh-dim-names string — + plain (non-DTensor) params map to ``"default"``). A scalar L2 norm is + computed per mesh group, DTensor results are reduced to local scalars + via ``.full_tensor()``, the per-mesh scalars are combined into one + global norm, and (unless ``return_norm_only=True``) every mesh group + is rescaled with that single global scalar. + + Args: + parameters: an iterable of Tensors or a single Tensor that will have gradients normalized + max_norm (float): max norm of the gradients + norm_type (float): type of the used p-norm. Can be ``'inf'`` for + infinity norm. + error_if_nonfinite (bool): if True, an error is thrown if the total + norm of the gradients from :attr:`parameters` is ``nan``, + ``inf``, or ``-inf``. Default: False (will switch to True in the future) + foreach (bool): use the faster foreach-based implementation. + If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently + fall back to the slow implementation for other device types. + Default: ``None`` + return_norm_only: if True, skip in-place rescaling of grads and only + return the computed norms. + + Returns: + ``(total_norm, per_mesh_norms)`` where ``total_norm`` is the global + scalar norm used for the rescale, and ``per_mesh_norms`` maps each + mesh-dim-names key (or ``"default"`` for plain params) to its + pre-clip per-mesh L2 norm. + + """ + # Group the parameters by their device meshes. + parameters_by_mesh: dict[str, list[torch.Tensor]] = defaultdict(list) + for param in parameters: + if param.grad is None: + raise ValueError( + f"_clip_grad received a parameter with no gradient " + f"(shape={tuple(param.shape)}, dtype={param.dtype}); " + "callers are expected to pre-filter." + ) + + # If one parameter belongs to multiple meshes, use a flattened mesh name + # by concatenating all the mesh-dim names together. ``mesh_dim_names`` + # is ``tuple[str, ...] | None`` on DeviceMesh — fall back to ``default`` + # when names weren't assigned. + if hasattr(param, "device_mesh"): + names = param.device_mesh.mesh_dim_names + device_mesh_str = "-".join(names) if names else "default" + else: + device_mesh_str = "default" + parameters_by_mesh[device_mesh_str].append(param) + + # Compute the norm for each mesh group + per_mesh_norms: dict[str, torch.Tensor] = {} + per_mesh_norm_list = [] + for mesh, params in parameters_by_mesh.items(): + # Every param reached here passed the ``param.grad is None`` check in + # the grouping loop above, so this list comprehension is total. + grads = [p.grad for p in params] + mesh_norm = torch.nn.utils.get_total_norm(grads, norm_type, error_if_nonfinite, foreach) + + # If mesh_norm is a DTensor, the placements must be + # `torch.distributed._tensor.ops.math_ops._NormPartial`. + # We can simply reduce the DTensor to get the total norm in this + # tensor's process group and then convert it to a local tensor. + + # 1. to make sure the total norm is computed correctly when PP is used (see below) + # 2. to return a reduced mesh_norm tensor whose .item() would return the correct value + if isinstance(mesh_norm, DTensor): + # Will reach here if any non-PP parallelism is used. + # If only using PP, mesh_norm will be a local tensor. + + # Remove FT replicate dimension if it exists. + mesh_norm = mesh_norm.full_tensor() + # Expose the (rank-replicated) per-mesh scalar for diagnostic logging. + per_mesh_norms[mesh] = mesh_norm + + # Make the norm to be a 1D tensor so we can call cat() later. + if mesh_norm.ndim == 0: + mesh_norm = mesh_norm.reshape(1) + per_mesh_norm_list.append(mesh_norm) + + # Compute the total norm among all meshes. + if len(per_mesh_norm_list) > 1: + per_mesh_norm_tensor = torch.cat(per_mesh_norm_list) + if math.isinf(norm_type): + total_norm = torch.max(per_mesh_norm_tensor) + else: + per_mesh_norm_tensor **= norm_type + total_norm = torch.sum(per_mesh_norm_tensor) + total_norm **= 1.0 / norm_type + else: + assert per_mesh_norm_list[0].numel() == 1, "total_norm should be a scalar" + total_norm = per_mesh_norm_list[0].view(-1)[0] + + if not return_norm_only: + # Perform clipping on each mesh group + for mesh, params in parameters_by_mesh.items(): + torch.nn.utils.clip_grads_with_norm_(params, max_norm, total_norm, foreach) + + return total_norm, per_mesh_norms + + +class GradClip(Callback): + """Unified gradient-clipping callback for both VFM (diffusion) and VLM training. + + The heavy lifting is delegated to ``_clip_grad``: it groups + params by their ``device_mesh`` (using mesh-dim-names as the key), + computes a scalar L2 norm per mesh group (reducing any DTensor result + via ``.full_tensor()``), combines the per-mesh scalars into ONE global + norm via ``sqrt(sum(per_mesh_norm**2))``, and applies + ``torch.nn.utils.clip_grads_with_norm_`` per mesh group with the SAME + global scalar — a SINGLE GLOBAL rescale across every parameter. + + This is necessary for correctness when parameters live on multiple device + meshes (e.g. dense FSDP-shard + EP-shard MoE experts): clipping each + mesh independently with stock ``torch.nn.utils.clip_grad_norm_`` would + assign a different rescale factor per mesh and distort the relative + magnitudes of dense vs MoE updates. Under VFM's current FSDP-only + setup the math reduces to a single mesh group and is identical to + stock ``clip_grad_norm_``; this implementation is forward-correct + once EP is enabled. + + For diagnostics, the callback ALSO records pre-clip per-mesh sub-norms + alongside the actual global norm. When ``track_per_modality=True`` (VFM), + samples are bucketed by image/video via ``model.is_image_batch(data_batch)``, + producing wandb keys ``clip_grad_norm/{image|video}/{mesh_key}`` plus a + ``.../global`` synthetic key carrying the actual rescale norm. When False + (VLM), keys are ``clip_grad_norm/{mesh_key}`` plus ``clip_grad_norm/global``. + + Param-source semantics: + * ``track_per_modality=True`` (VFM): caller passes the ``OmniMoTModel``; + only ``model.net.parameters()`` is iterated, matching legacy VFM + behavior (the optimizer is built from ``self.net``). + * ``track_per_modality=False`` (VLM): caller passes a single + ``ImaginaireModel`` or a list of model parts; ``parameters()`` is + iterated and filtered by grad-presence. + + Args: + clip_norm: max norm to clip to. + force_finite: if True, NaN/Inf in any grad is zeroed in-place before + the norm computation. + track_per_modality: if True, route stats into image/video buckets via + ``model.is_image_batch(data_batch)``. If False, accumulate into a + single un-bucketed log group. + """ + + def __init__( + self, + clip_norm: float = 1.0, + force_finite: bool = True, + track_per_modality: bool = False, + ): + self.clip_norm = clip_norm + self.force_finite = force_finite + self.track_per_modality = track_per_modality + + # Outer key: modality bucket name. For VLM we use a single bucket "" so + # wandb keys are short (`clip_grad_norm/{mesh}`); for VFM the bucket is + # "image" or "video" (`clip_grad_norm/image/{mesh}`). + # Inner key: mesh string, plus the synthetic "global" key for the + # actual rescale norm returned by _clip_grad. + self._states: dict[str, dict[str, _MagnitudeRecord]] = defaultdict(lambda: defaultdict(_MagnitudeRecord)) + self._state_key: str = "" + + def on_training_step_start( + self, + model: torch.nn.Module, + data_batch: dict[str, torch.Tensor], + iteration: int = 0, + ) -> None: + if not self.track_per_modality: + return + self._state_key = "image" if model.is_image_batch(data_batch) else "video" + + def on_before_optimizer_step( + self, + model: torch.nn.Module | list[torch.nn.Module], + optimizer: torch.optim.Optimizer, + scheduler: torch.optim.lr_scheduler.LRScheduler, + grad_scaler: torch.amp.GradScaler, + iteration: int = 0, + ) -> None: + del optimizer, scheduler, grad_scaler + + # 1. Resolve which parameters to clip. + if self.track_per_modality: + # VFM: only clip `.net` params, matching legacy semantics + the + # optimizer's actual param set. + assert not isinstance(model, list), "track_per_modality=True expects a single OmniMoTModel, not a list" + model_parts = [model.net] + else: + # VLM: frozen vlm/trainer/sft_trainer_cosmos_rl.py types + # model_parts as list[ImaginaireModel] (FSDP + PP, no DDP). + model_parts = model if isinstance(model, list) else [model] + + # 2. Collect params with grads. + all_params: list[torch.Tensor] = [] + for part in model_parts: + for p in part.parameters(): + if p.grad is not None: + all_params.append(p) + + # 3. No-grad / all-frozen step → skip. _clip_grad's empty + # fallback uses torch.cuda.current_device() and would crash on CPU. + if not all_params: + return + + # 4. Optionally zero NaN/Inf in grads. + if self.force_finite: + _fused_nan_to_num([p.grad for p in all_params]) + + # 5. Compute per-mesh norms, the global rescale norm, and clip in + # one call. ``_clip_grad`` groups params by mesh, + # computes per-mesh L2 norms (reducing DTensor results to local + # scalars), combines them into a single global norm, and + # rescales every mesh group with that scalar. + # + # When ``force_finite`` is False we did NOT sanitize the grads, so + # ask ``get_total_norm`` to raise rather than silently producing a + # NaN ``total_norm`` that would taint every parameter on rescale. + global_norm, per_mesh_norms = _clip_grad( + all_params, + self.clip_norm, + error_if_nonfinite=False, + foreach=True, + ) + + # 6. Record diagnostic stats: pre-clip per-mesh sub-norms plus the + # actual global rescale norm. + cur_state = self._states[self._state_key] + for mesh_str, mesh_norm in per_mesh_norms.items(): + cur_state[mesh_str].update(mesh_norm) + cur_state["global"].update(global_norm) + + # 7. Log every logging_iter. The reset is intentionally *outside* + # the ``wandb.run`` gate: ``_MagnitudeRecord.get_stat`` is the + # consumer that flushes the windowed accumulator, so coupling it + # to wandb being live would let stats accumulate unboundedly + # whenever wandb is disabled (smoke tests, ``job.wandb_mode=disabled``, + # wandb init failure) and would back-fill any later wandb enablement + # with the entire pre-enable history. + if iteration % self.config.trainer.logging_iter == 0: + log_dict: dict[str, float | int] = {"iteration": iteration} + for modality, state in self._states.items(): + for mesh_str, record in state.items(): + avg = record.get_stat() + if self.track_per_modality: + key = f"clip_grad_norm/{modality}/{mesh_str}" + else: + key = f"clip_grad_norm/{mesh_str}" + log_dict[key] = avg + if mesh_str == "global": + log.info(f"{key}: {avg:.5f} (iteration {iteration})", rank0_only=False) + if wandb.run: + wandb.log(log_dict, step=iteration) diff --git a/cosmos_framework/callbacks/heart_beat.py b/cosmos_framework/callbacks/heart_beat.py new file mode 100644 index 0000000..16462c1 --- /dev/null +++ b/cosmos_framework/callbacks/heart_beat.py @@ -0,0 +1,94 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import time +from datetime import datetime + +import pytz +import torch + +from cosmos_framework.callbacks.every_n import EveryN +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.trainer import ImaginaireTrainer +from cosmos_framework.utils import distributed +from cosmos_framework.utils.easy_io import easy_io + + +class HeartBeat(EveryN): + """ + A callback that logs a heartbeat message at regular intervals to indicate that the training process is still running. + + Args: + every_n (int): The frequency at which the callback is invoked. + step_size (int, optional): The step size for the callback. Defaults to 1. + update_interval_in_minute (int, optional): The interval in minutes for logging the heartbeat. Defaults to 20 minutes. + save_s3 (bool, optional): Whether to save the heartbeat information to S3. Defaults to False. + """ + + def __init__(self, every_n: int, step_size: int = 1, update_interval_in_minute: int = 20, save_s3: bool = False): + super().__init__(every_n=every_n, step_size=step_size) + self.name = self.__class__.__name__ + self.update_interval_in_minute = update_interval_in_minute + self.save_s3 = save_s3 + self.pst = pytz.timezone("America/Los_Angeles") + self.is_hitted = False + + @distributed.rank0_only + def on_train_start(self, model: ImaginaireModel, iteration: int = 0) -> None: + self.time = time.time() + if self.save_s3: + current_time_pst = datetime.now(self.pst).strftime("%Y_%m_%d-%H_%M_%S") + info = { + "iteration": iteration, + "time": current_time_pst, + } + easy_io.dump(info, f"s3://rundir/{self.name}_start.yaml") + easy_io.dump(info, f"s3://timestamps_rundir/{self.name}_start.yaml") + + def on_training_step_end( + self, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int = 0, + ) -> None: + if not self.is_hitted: + self.is_hitted = True + if distributed.get_rank() == 0: + self.report(iteration) + super().on_training_step_end(model, data_batch, output_batch, loss, iteration) + + @distributed.rank0_only + def every_n_impl( + self, + trainer: ImaginaireTrainer, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int, + ) -> None: + if time.time() - self.time > 60 * self.update_interval_in_minute: + self.report(iteration) + + def report(self, iteration: int = 0): + self.time = time.time() + if self.save_s3: + current_time_pst = datetime.now(self.pst).strftime("%Y_%m_%d-%H_%M_%S") + info = { + "iteration": iteration, + "time": current_time_pst, + } + easy_io.dump(info, f"s3://rundir/{self.name}.yaml") + + @distributed.rank0_only + def on_train_end(self, model: ImaginaireModel, iteration: int = 0) -> None: + if self.save_s3: + current_time_pst = datetime.now(self.pst).strftime("%Y_%m_%d-%H_%M_%S") + info = { + "iteration": iteration, + "time": current_time_pst, + } + easy_io.dump(info, f"s3://rundir/{self.name}_end.yaml") + easy_io.dump(info, f"s3://timestamps_rundir/{self.name}_end.yaml") diff --git a/cosmos_framework/callbacks/hf_export.py b/cosmos_framework/callbacks/hf_export.py new file mode 100644 index 0000000..583c968 --- /dev/null +++ b/cosmos_framework/callbacks/hf_export.py @@ -0,0 +1,459 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 +"""HFExportCallback: export VLM DCP checkpoints to HuggingFace safetensors format. + +Design notes +------------ +- Hooks into ``on_save_checkpoint`` (called by DistributedCheckpointer.save() before I/O). +- All ranks participate in the weight-gather phase (DTensor.full_tensor() all-gathers). +- Rank 0 accumulates CPU tensors, writes shards, and uploads — other ranks exit early. +- File I/O and upload run in a background thread on rank 0 to avoid blocking training. +- Worker exceptions are stored in ``_worker_exception`` and re-raised on the next + checkpoint or at train end, so failures are never silently swallowed. +- Controlled entirely via ``config.checkpoint.hf_export`` (HFExportConfig). + +Phase 2+ note +------------- +Weight parameters are iterated via ``model.model.model.named_parameters()`` where +``model.model`` is the ``HFModel`` wrapper and ``model.model.model`` is the underlying +HuggingFace transformer. Parameter names are already HF-native — no weight_mapper +remapping is required. +""" + +import json +import os +import shutil +import threading +from typing import Any + +import torch + +from cosmos_framework.utils import log +from cosmos_framework.utils.callback import Callback +from cosmos_framework.utils.distributed import is_rank0 + +try: + from safetensors.torch import save_file as _safetensors_save_file +except ImportError: + _safetensors_save_file = None + +try: + from transformers import AutoTokenizer, GenerationConfig +except ImportError: + AutoTokenizer = None + GenerationConfig = None + +# Map string dtype names (as stored in ParallelismConfig.precision) to torch dtypes. +_DTYPE_MAP: dict[str, torch.dtype] = { + "float32": torch.float32, + "float16": torch.float16, + "bfloat16": torch.bfloat16, + "float64": torch.float64, +} + + +def _upload_folder_to_s3(local_folder: str, bucket: str, s3_prefix: str, credential_path: str) -> None: + """Upload every file under *local_folder* to ``s3://{bucket}/{s3_prefix}/...``. + + Uses the i4 ``easy_io`` S3 backend (Boto3Backend), which reads credentials from + *credential_path*. Files are uploaded as streaming transfers via boto3's + ``upload_file()`` — the full shard is never loaded into memory. + """ + from cosmos_framework.utils.easy_io import easy_io + + backend = easy_io.get_file_backend( + backend_args={ + "backend": "s3", + "s3_credential_path": credential_path, + "path_mapping": None, + } + ) + for root, _, files in os.walk(local_folder): + for fname in sorted(files): + local_path = os.path.join(root, fname) + rel = os.path.relpath(local_path, local_folder) + s3_path = f"s3://{bucket}/{s3_prefix}/{rel}" + # Pass the local path string so Boto3Backend uses upload_file() — + # a streaming transfer that avoids reading the whole shard into memory. + backend.put(local_path, s3_path) + log.info(f"[HFExportCallback] Uploaded {local_path} → {s3_path}") + + +class HFExportCallback(Callback): + """Export VLM weights to HuggingFace-compatible safetensors after each DCP checkpoint. + + Enabled / configured via ``config.checkpoint.hf_export`` (HFExportConfig). Disabled + by default — add this callback and set ``hf_export.enabled = True`` to activate. + + Exports written to:: + + {job.path_local}/hf_exports/iter_{iteration:09d}/ + 00000.safetensors + ... + model.safetensors.index.json + config.json + tokenizer.json (if tokenizer can be loaded from model_name_or_path) + + Optionally uploads to: + - S3 (``hf_export.upload_to_object_store``) + - HuggingFace Hub (``hf_export.hf_repo_id``) + + Args: + dtype: Export weight dtype (e.g. ``"bfloat16"``). Use + ``"${model.config.policy.parallelism.precision}"`` in the Hydra callback config to + inherit from the training precision. + """ + + # HuggingFace convention: max 4 GB per shard file. + _MAX_SHARD_BYTES: int = 4 * 1024**3 + + def __init__(self, dtype: str = "bfloat16") -> None: + self._export_dtype: torch.dtype | None = _DTYPE_MAP.get(dtype) + self._current_iteration: int = 0 + self._export_thread: threading.Thread | None = None + # Stores any exception raised inside the background worker so it can be + # re-raised on the main thread at the next checkpoint or train end. + self._worker_exception: BaseException | None = None + + # ------------------------------------------------------------------ + # Callback hooks + # ------------------------------------------------------------------ + + def on_save_checkpoint_start(self, model: Any, iteration: int = 0) -> None: + self._current_iteration = iteration + + def on_save_checkpoint(self, model: Any, state_dict: dict[str, Any]) -> None: + hf_cfg = self.config.checkpoint.hf_export + if not hf_cfg.enabled: + return + + iteration = self._current_iteration + if iteration % hf_cfg.export_every_n != 0: + return + + # Deferred import to avoid circular dependency at module load time. + from cosmos_framework.model.vfm.vlm_model import VLMModel + + if not isinstance(model, VLMModel): + # The legacy vlm/train.py path passes model_parts: list[nn.Module] (raw HF + # models without the VLMModel attribute structure). HF export requires the + # VLMModel wrapper, which is only available via the unified cosmos_framework/scripts/train.py path. + if isinstance(model, list): + log.warning( + "[HFExportCallback] Received model_parts (list) instead of VLMModel. " + "HF export requires the unified training path (cosmos_framework/scripts/train.py). Skipping." + ) + else: + log.warning( + "[HFExportCallback] model is not VLMModel (got %s); skipping HF export.", + type(model).__name__, + ) + return + + if _safetensors_save_file is None: + raise ImportError("safetensors is required for HFExportCallback. Install it with: pip install safetensors") + + output_dir = os.path.join(self.config.job.path_local, "hf_exports", f"iter_{iteration:09d}") + + # ---------------------------------------------------------------- + # Phase 1 (all ranks): gather sharded parameters into CPU chunks. + # full_tensor() is a collective operation — all ranks must participate. + # ---------------------------------------------------------------- + cpu_chunks, manifest, total_size = self._gather_weights(model) + + # ---------------------------------------------------------------- + # Phase 2 (rank 0, background thread): file I/O + optional upload. + # ---------------------------------------------------------------- + if not is_rank0(): + return + + # Block on any still-running export from the previous checkpoint and + # propagate any worker exception before starting a new export. + if self._export_thread is not None and self._export_thread.is_alive(): + log.warning( + "[HFExportCallback] Previous export thread still running; waiting before starting export for iter %d.", + iteration, + ) + self._export_thread.join() + + if self._worker_exception is not None: + exc = self._worker_exception + self._worker_exception = None + raise RuntimeError(f"[HFExportCallback] Previous export failed with: {exc}") from exc + + self._export_thread = threading.Thread( + target=self._save_and_upload, + args=(cpu_chunks, manifest, total_size, model.hf_config, model.model_name_or_path, output_dir, iteration), + daemon=True, + ) + self._export_thread.start() + + def on_train_end(self, model: Any, iteration: int = 0) -> None: + """Wait for the final export thread so the process does not exit prematurely.""" + if self._export_thread is not None and self._export_thread.is_alive(): + log.info("[HFExportCallback] Waiting for export thread to finish...") + self._export_thread.join() + log.info("[HFExportCallback] Export thread done.") + + if self._worker_exception is not None: + exc = self._worker_exception + self._worker_exception = None + raise RuntimeError(f"[HFExportCallback] Export thread failed with: {exc}") from exc + + # ------------------------------------------------------------------ + # Internal helpers + # ------------------------------------------------------------------ + + def _gather_weights(self, model: Any) -> tuple[list[dict[str, torch.Tensor]], dict[str, str], int]: + """Iterate model parameters, all-gather DTensor shards, and build CPU chunks. + + Must be called on **all ranks**. Only rank 0 populates the returned + ``cpu_chunks`` and ``manifest``; other ranks return empty structures but still + participate in the distributed all-gathers. + + Returns: + cpu_chunks: List of ``{weight_name: cpu_tensor}`` dicts, one per shard file. + manifest: Mapping of ``weight_name → shard_filename``. + total_size: Total byte count of all exported tensors (for the index JSON). + """ + cpu_chunks: list[dict[str, torch.Tensor]] = [] + manifest: dict[str, str] = {} + current_chunk: dict[str, torch.Tensor] = {} + current_chunk_bytes: int = 0 + total_size: int = 0 + file_idx: int = 0 + + for name, param in model.model.model.named_parameters(): + # Phase 2+: HFModel initialises _model via AutoModelForImageTextToText / + # AutoModelForCausalLM, so parameter names are HF-native and match the + # safetensors checkpoint keys loaded by _load_vlm_weights(). + # + # MoE note: Qwen3VLMoeTextExpertsGroupedMm stores expert weights in HF-native + # grouped layout — gate_up_proj: [E, H, 2F], down_proj: [E, F, H] — matching + # the checkpoint format exactly. No transposition or per-expert fan-out is + # needed. (The legacy Phase 0 path stored tensors in a transposed internal + # format [E, 2F, H] under the name "gate_and_up_projs" and required + # weight_mapper.policy_map_local_key_for_export_tensor() to transpose back on + # export. Phase 2 uses HFModel and has no such internal reformat.) + # + # torch.compile and gradient-checkpointing wrappers inject prefixes into + # named_parameters() output. Strip them so exported keys are HF-native, + # matching what HFModel._load_vlm_weights() does for the in-memory state dict. + name = name.replace("_orig_mod.", "").replace("_checkpoint_wrapped_module.", "") + + # Gather across FSDP / TP / CP ranks (collective — all ranks must call). + if isinstance(param, torch.distributed.tensor.DTensor): + param = param.full_tensor() + param = param.detach() + if self._export_dtype is not None: + param = param.to(dtype=self._export_dtype) + + tensor_bytes = param.element_size() * param.numel() + + # Flush the current chunk when the shard size limit would be exceeded. + # current_chunk_bytes is tracked on ALL ranks so shard boundaries are + # consistent (the shard_name written into manifest must agree everywhere). + if current_chunk_bytes + tensor_bytes > self._MAX_SHARD_BYTES and current_chunk_bytes > 0: + if is_rank0(): + cpu_chunks.append(current_chunk) + current_chunk = {} + file_idx += 1 + current_chunk_bytes = 0 + + shard_name = f"{file_idx:05d}.safetensors" + if is_rank0(): + current_chunk[name] = param.cpu() + manifest[name] = shard_name + total_size += tensor_bytes + current_chunk_bytes += tensor_bytes + + # Flush the final (possibly partial) chunk. + if current_chunk_bytes > 0 and is_rank0() and current_chunk: + cpu_chunks.append(current_chunk) + + return cpu_chunks, manifest, total_size + + def _save_and_upload( + self, + cpu_chunks: list[dict[str, torch.Tensor]], + manifest: dict[str, str], + total_size: int, + hf_config: Any, + model_name_or_path: str, + output_dir: str, + iteration: int, + ) -> None: + """Write safetensors shards, HF config, tokenizer; upload to S3 / HF Hub. + + Runs on rank 0 inside a background thread. Any exception is stored in + ``self._worker_exception`` so the main thread can re-raise it. + """ + try: + self._do_save_and_upload( + cpu_chunks, manifest, total_size, hf_config, model_name_or_path, output_dir, iteration + ) + except Exception as exc: + log.error( + "[HFExportCallback] Export worker for iter %d raised an exception: %s", + iteration, + exc, + exc_info=True, + ) + self._worker_exception = exc + + def _do_save_and_upload( + self, + cpu_chunks: list[dict[str, torch.Tensor]], + manifest: dict[str, str], + total_size: int, + hf_config: Any, + model_name_or_path: str, + output_dir: str, + iteration: int, + ) -> None: + """Core export logic (called from the background thread via ``_save_and_upload``). + + Error handling is tiered: + - Steps 1-4 (shards, index JSON, HF config, source-model file copy): any exception + propagates to the outer ``_save_and_upload`` wrapper so the main thread is notified. + A failed file copy leaves the checkpoint unusable for trust_remote_code models, so + it is treated as a hard failure like the shard writes. + - Steps 5-7 (tokenizer, generation_config, S3 upload, HF Hub upload): failures are + treated as soft warnings. The tokenizer and generation config are best-effort; upload + failures do not invalidate the local safetensors export, so an outage must not abort + training. + """ + hf_cfg = self.config.checkpoint.hf_export + os.makedirs(output_dir, exist_ok=True) + log.info(f"[HFExportCallback] Writing iter {iteration} export to {output_dir}") + + # 1. Safetensors shards — one file per chunk (ordered by file_idx). + # Each chunk is cleared after writing so its tensors can be GC'd + # incrementally rather than being held until the whole loop completes. + for i in range(len(cpu_chunks)): + chunk = cpu_chunks[i] + shard_path = os.path.join(output_dir, f"{i:05d}.safetensors") + _safetensors_save_file(chunk, shard_path) + log.info(f"[HFExportCallback] Wrote {shard_path}") + cpu_chunks[i] = {} # release tensor references for GC + + # 2. model.safetensors.index.json + # total_size is pre-computed in _gather_weights to avoid needing chunks here. + index_json = {"metadata": {"total_size": total_size}, "weight_map": manifest} + index_path = os.path.join(output_dir, "model.safetensors.index.json") + with open(index_path, "w") as fh: + json.dump(index_json, fh, indent=4) + + # 3. HuggingFace model config. + hf_config.save_pretrained(output_dir) + + # 4. Copy missing .py/.json files for trust_remote_code models. + # Only applicable when model_name_or_path is a local directory. + # The full directory layout is preserved so nested packages referenced by + # auto_map are included (mirroring convert_checkpoint.py's copytree approach). + # Files already present in the export dir (e.g., config.json written by + # hf_config.save_pretrained) are never overwritten. + # HARD failure: a broken copy leaves the checkpoint unloadable, so any I/O error + # propagates to the background-worker wrapper (same as shard writes). + if model_name_or_path and os.path.isdir(model_name_or_path): + real_src = os.path.realpath(model_name_or_path) + real_out = os.path.realpath(output_dir) + copied = [] + for root, dirs, files in os.walk(real_src): + real_root = os.path.realpath(root) + # Prune any subtree that is, leads to, or is inside output_dir. + # This prevents recursing into previously written export dirs when + # output_dir (or a parent of it) lives inside model_name_or_path. + dirs[:] = [ + d + for d in dirs + if not ( + (p := os.path.realpath(os.path.join(real_root, d))) == real_out + or p.startswith(real_out + os.sep) + or real_out.startswith(p + os.sep) + ) + ] + if real_root == real_out or real_root.startswith(real_out + os.sep): + continue + rel_dir = os.path.relpath(real_root, real_src) + for fname in files: + if not (fname.endswith(".py") or fname.endswith(".json")): + continue + src = os.path.join(real_root, fname) + dst_dir = output_dir if rel_dir == "." else os.path.join(output_dir, rel_dir) + dst = os.path.join(dst_dir, fname) + if not os.path.exists(dst): + os.makedirs(dst_dir, exist_ok=True) + shutil.copy2(src, dst) + copied.append(os.path.join(rel_dir, fname) if rel_dir != "." else fname) + if copied: + log.info(f"[HFExportCallback] Copied missing files from source model: {copied}") + + # 5. Tokenizer (best-effort — may fail for custom / gated models). + if AutoTokenizer is not None and model_name_or_path: + try: + tok = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True) + tok.save_pretrained(output_dir) + except Exception as exc: + log.warning(f"[HFExportCallback] Tokenizer save skipped: {exc}") + + # 6. Generation config (best-effort — not all models expose one). + if GenerationConfig is not None and model_name_or_path: + try: + gen_cfg = GenerationConfig.from_pretrained(model_name_or_path, trust_remote_code=True) + gen_cfg.save_pretrained(output_dir) + except Exception as exc: + log.warning(f"[HFExportCallback] generation_config save skipped: {exc}") + + # 7. S3 upload — soft failure: local export is intact regardless of upload outcome. + obj_store = hf_cfg.upload_to_object_store + if obj_store.enabled: + s3_prefix = f"{self.config.job.path}/hf_exports/iter_{iteration:09d}" + try: + _upload_folder_to_s3(output_dir, obj_store.bucket, s3_prefix, obj_store.credentials) + log.info(f"[HFExportCallback] S3 upload done: s3://{obj_store.bucket}/{s3_prefix}") + except Exception as exc: + # Intentionally soft: an upload outage must not crash training. + log.warning(f"[HFExportCallback] S3 upload failed (local export intact): {exc}") + + # 8. HuggingFace Hub upload — soft failure: see comment above. + if hf_cfg.hf_repo_id: + self._upload_to_hf_hub(output_dir, hf_cfg.hf_repo_id) + + log.info(f"[HFExportCallback] Export complete for iter {iteration}.") + + @staticmethod + def _upload_to_hf_hub(output_dir: str, repo_id: str, max_retries: int = 3) -> None: + try: + from huggingface_hub import HfApi + except ImportError: + log.warning("[HFExportCallback] huggingface_hub not installed; skipping HF Hub upload.") + return + + api = HfApi() + for attempt in range(1, max_retries + 1): + try: + api.create_repo(repo_id=repo_id, exist_ok=True) + break + except Exception as exc: + log.warning(f"[HFExportCallback] create_repo attempt {attempt}/{max_retries} failed: {exc}") + if attempt == max_retries: + log.warning( + f"[HFExportCallback] Could not create HF Hub repo '{repo_id}' after " + f"{max_retries} attempts; skipping upload." + ) + return + + for attempt in range(1, max_retries + 1): + try: + api.upload_folder( + folder_path=output_dir, + repo_id=repo_id, + commit_message=f"Upload checkpoint from {os.path.basename(output_dir)}", + ) + log.info(f"[HFExportCallback] Uploaded to HF Hub: {repo_id}") + return + except Exception as exc: + log.warning(f"[HFExportCallback] HF Hub upload attempt {attempt}/{max_retries} failed: {exc}") + + log.warning(f"[HFExportCallback] All {max_retries} HF Hub upload attempts failed for {repo_id}.") diff --git a/cosmos_framework/callbacks/iter_speed.py b/cosmos_framework/callbacks/iter_speed.py new file mode 100644 index 0000000..738721e --- /dev/null +++ b/cosmos_framework/callbacks/iter_speed.py @@ -0,0 +1,109 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import time + +import torch +import wandb +from torch import Tensor + +from cosmos_framework.callbacks.every_n import EveryN +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.trainer import ImaginaireTrainer +from cosmos_framework.utils import log +from cosmos_framework.utils.distributed import rank0_only +from cosmos_framework.utils.easy_io import easy_io + + +class IterSpeed(EveryN): + """ + Args: + hit_thres (int): Number of iterations to wait before logging. + save_s3 (bool): Whether to save to S3. + save_s3_every_log_n (int): Save to S3 every n log iterations, which means save_s3_every_log_n n * every_n global iterations. + """ + + def __init__(self, *args, hit_thres: int = 5, save_s3: bool = True, save_s3_every_log_n: int = 10, **kwargs): + super().__init__(*args, **kwargs) + self.time = None + self.hit_counter = 0 + self.hit_thres = hit_thres + self.save_s3 = save_s3 + self.save_s3_every_log_n = save_s3_every_log_n + self.name = self.__class__.__name__ + self.last_hit_time = time.time() + + def on_training_step_end( + self, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int = 0, + ) -> None: + if self.hit_counter < self.hit_thres: + log.info( + f"Iteration {iteration}: " + f"Hit counter: {self.hit_counter + 1}/{self.hit_thres} | " + f"Loss: {loss.detach().item():.4f} | " + f"Time: {time.time() - self.last_hit_time:.2f}s", + rank0_only=False, + ) + self.hit_counter += 1 + self.last_hit_time = time.time() + #! useful for large scale training and avoid oom crash in the first two iterations!!! + torch.cuda.synchronize() + return + super().on_training_step_end(model, data_batch, output_batch, loss, iteration) + + @rank0_only + def every_n_impl( + self, + trainer: ImaginaireTrainer, + model: ImaginaireModel, + data_batch: dict[str, Tensor], + output_batch: dict[str, Tensor], + loss: Tensor, + iteration: int, + ) -> None: + if self.time is None: + self.time = time.time() + return + cur_time = time.time() + iter_speed = (cur_time - self.time) / self.every_n / self.step_size + + log.info( + f"{iteration} : iter_speed {iter_speed:.2f} seconds per iteration | Loss: {loss.detach().item():.4f}", + rank0_only=False, + ) + + per_sample_batch_counter = dict() + if hasattr(model, "is_image_batch"): + is_image_batch = model.is_image_batch(data_batch) + if is_image_batch: + image_batch_size = len(data_batch[model.input_image_key]) + per_sample_batch_counter["image_batch_size"] = image_batch_size + else: + video_batch_size = len(data_batch[model.input_video_key]) + per_sample_batch_counter["video_batch_size"] = video_batch_size + + if wandb.run: + sample_counter = getattr(trainer, "sample_counter", iteration) + wandb.log( + { + "timer/iter_speed": iter_speed, + "sample_counter": sample_counter, + } + | per_sample_batch_counter, + step=iteration, + ) + self.time = cur_time + if self.save_s3: + if iteration % (self.save_s3_every_log_n * self.every_n) == 0: + easy_io.dump( + { + "iter_speed": iter_speed, + "iteration": iteration, + }, + f"s3://rundir/{self.name}/iter_{iteration:09d}.yaml", + ) diff --git a/cosmos_framework/callbacks/learning_rate_logger.py b/cosmos_framework/callbacks/learning_rate_logger.py new file mode 100644 index 0000000..223d827 --- /dev/null +++ b/cosmos_framework/callbacks/learning_rate_logger.py @@ -0,0 +1,47 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import torch +import wandb + +from cosmos_framework.utils.callback import Callback + + +class LearningRateLogger(Callback): + """Logs per-model-part learning rate every ``every_n × logging_iter`` steps. + + Designed for VLM training where the optimizer is an + ``OptimizersContainer`` exposing ``.optimizers`` (list of single-element + optimizer lists) paired with ``.model_part_names``. Silently no-ops when + those attributes are absent so it can be registered alongside plain + ``torch.optim.Optimizer`` setups without harm. + """ + + def __init__(self, every_n: int = 10): + self.every_n = every_n + + def on_before_optimizer_step( + self, + model: torch.nn.Module | list[torch.nn.Module], + optimizer: torch.optim.Optimizer, + scheduler: torch.optim.lr_scheduler.LRScheduler, + grad_scaler: torch.amp.GradScaler, + iteration: int = 0, + ) -> None: + del model, scheduler, grad_scaler + gate = self.config.trainer.logging_iter * self.every_n + if not (iteration == 1 or (gate > 0 and iteration % gate == 0)): + return + if not wandb.run: + return + if not (hasattr(optimizer, "optimizers") and hasattr(optimizer, "model_part_names")): + return + unique_lr: dict[str, float] = {} + for optim_per_model, name in zip(optimizer.optimizers, optimizer.model_part_names): + if not optim_per_model: + continue + for pg in optim_per_model[0].param_groups: + unique_lr[f"optim/lr_{name}"] = pg["lr"] + if not unique_lr: + return + wandb.log(unique_lr, step=iteration) diff --git a/cosmos_framework/callbacks/load_pretrained.py b/cosmos_framework/callbacks/load_pretrained.py new file mode 100644 index 0000000..45f81d8 --- /dev/null +++ b/cosmos_framework/callbacks/load_pretrained.py @@ -0,0 +1,29 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.utils.callback import Callback + + +class LoadPretrained(Callback): + """Load HF understanding-pathway weights after DCP resume, gated by checkpoint state. + + Decision table (config flags are *intent*; the runtime probes here decide): + * Latest checkpoint exists in load dir → DCP loaded the full model. Skip HF load. + * No latest checkpoint, ``load_path`` set → DCP loaded full model from warm-start. + Reload HF understanding pathway (e.g. swap Qwen3-VL → Cosmos-Reason) but skip + the understanding→generation copy. + * Neither → fresh init: full HF load + understanding→generation copy. + + Reads ``self.config.checkpoint`` / ``self.config.job`` (injected by + ``CallBackGroup`` after instantiate) to build a probe checkpointer. + """ + + def on_train_start(self, model: ImaginaireModel, iteration: int = 0) -> None: + from cosmos_framework.checkpoint.dcp import DistributedCheckpointer + + probe = DistributedCheckpointer(self.config.checkpoint, self.config.job, callbacks=None, disable_async=True) + model.load_pretrained_model_if_needed( + has_resumable_checkpoint=probe.has_resumable_checkpoint(), + has_load_path=probe.load_path is not None, + ) diff --git a/cosmos_framework/callbacks/log_tensor_shape.py b/cosmos_framework/callbacks/log_tensor_shape.py new file mode 100644 index 0000000..50cd941 --- /dev/null +++ b/cosmos_framework/callbacks/log_tensor_shape.py @@ -0,0 +1,34 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from __future__ import annotations + +import torch + +from cosmos_framework.utils import log +from cosmos_framework.utils.callback import Callback + + +class LogTensorShapeCallback(Callback): + """Log the shape and dtype of every tensor in ``data_batch`` for the first + ``num_log`` training iterations, on every rank. Used to verify dataloader + geometry at the start of a run. + """ + + def __init__(self, num_log: int = 10): + self.num_log = num_log + + def on_training_step_start(self, model_parts, data_batch, iteration): + if iteration > self.num_log: + return + summary_str = f"[Tensor Shape] Iteration {iteration}" + for key in data_batch.keys(): + if isinstance(data_batch[key], torch.Tensor): + summary_str += f" | {key} shape: {data_batch[key].shape}, dtype: {data_batch[key].dtype} " + summary_str += f"data_batch: {data_batch.keys()}" + if iteration < 1000: + # Only log the first 1000 iterations + for key in ["__url__", "__key__", "image_grid_thw", "video_grid_thw"]: + if key in data_batch: + summary_str += f" | {key}: {data_batch[key]}" + log.info(summary_str, rank0_only=False) diff --git a/cosmos_framework/callbacks/manual_gc.py b/cosmos_framework/callbacks/manual_gc.py new file mode 100644 index 0000000..a9883d8 --- /dev/null +++ b/cosmos_framework/callbacks/manual_gc.py @@ -0,0 +1,38 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import gc + +from cosmos_framework.callbacks.every_n import EveryN +from cosmos_framework.utils import log + + +class ManualGarbageCollection(EveryN): + """ + Disable auto gc and manually trigger garbage collection every N iterations + It is super useful for large scale training to reduce gpu sync time! + Can reach 50% speedup. + + It is important to note that this callback only disables gc in main process and have auto gc enabled in subprocesses. + + We start disable gc after warm_up iterations to avoid disabling gc in subprocesses, such as dataloader, which can cause OOM + """ + + def __init__(self, *args, warm_up: int = 5, gc_level: int = 1, **kwargs): + kwargs["barrier_after_run"] = False + super().__init__(*args, **kwargs) + + self.counter = 0 + self.warm = warm_up + self.gc_level = gc_level + + def every_n_impl(self, trainer, model, data_batch, output_batch, loss, iteration): + del trainer, model, data_batch, output_batch, loss + self.counter += 1 + if self.counter < self.warm: + return + if self.counter == self.warm: + gc.disable() + log.critical("Garbage collection disabled") + + gc.collect(self.gc_level) diff --git a/cosmos_framework/callbacks/mfu.py b/cosmos_framework/callbacks/mfu.py new file mode 100644 index 0000000..3035437 --- /dev/null +++ b/cosmos_framework/callbacks/mfu.py @@ -0,0 +1,315 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""MFU (Model FLOPs Utilization) callback for OmniMoT training. + +Computes and logs MFU metrics for specified hardware targets (e.g. H100, GB200) +by calculating the actual training FLOPs per step and comparing against +theoretical peak throughput. +""" + +from __future__ import annotations + +import time +from dataclasses import dataclass +from decimal import Decimal + +import torch +import wandb + +from cosmos_framework.model.attention.utils import is_blackwell_dc +from cosmos_framework.callbacks.every_n import EveryN +from cosmos_framework.tools.flops import ( + OmniMoTModelDescriptor, + compute_omni_mot_flops_per_batch, + compute_wan_vae_encoder_flops, + get_omni_mot_model_descriptor, +) +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.trainer import ImaginaireTrainer +from cosmos_framework.utils import log +from cosmos_framework.utils.distributed import rank0_only + + +@dataclass +class HardwareTarget: + """Specification of a hardware target for MFU computation. + + Attributes: + name: Human-readable name (used as W&B tag, e.g. "H100"). + peak_tflops: Theoretical peak throughput in TFLOPS (e.g. 989 for H100 BF16). + """ + + name: str + peak_tflops: float + + +# Pre-defined hardware targets +H100 = HardwareTarget(name="H100", peak_tflops=989.0) +GB200 = HardwareTarget(name="GB200", peak_tflops=2250.0) + + +class MFUCallback(EveryN): + """Callback that computes and logs Model FLOPs Utilization (MFU) to W&B. + + MFU is defined as: + MFU = achieved_tflops_per_gpu / peak_tflops_per_gpu + + where achieved_tflops_per_gpu is computed from the model's theoretical + training FLOPs (forward + backward) divided by the measured wall-clock + time per step. + + This callback accumulates per-step FLOPs between logging intervals and + reports the average MFU over that window. + + Args: + backwardpass_ratio: Ratio of backward-to-forward FLOPs (default 2.0). + hit_thres: Number of warm-up iterations before logging begins. + include_vae_encoder: If True (default), include the Wan 2.2 VAE encoder + forward-pass FLOPs in the per-step total. The VAE is frozen during + training so only forward FLOPs are counted. + include_padding: If True, include FLOPs spent on padding tokens (the + causal split appended by sequence-packing finalize()). Gives a + ``total GPU FLOPs`` view instead of ``useful FLOPs`` only. + grad_accum_iter: Number of gradient accumulation steps per optimizer + update (default 1). When > 1, ``on_training_step_end`` is called + once per optimizer step but the wall-clock time covers all + micro-batches, so per-step FLOPs are multiplied by this count. + """ + + def __init__( + self, + *args, + backwardpass_ratio: float = 2.0, + hit_thres: int = 5, + include_vae_encoder: bool = True, + include_padding: bool = True, + grad_accum_iter: int = 1, + **kwargs, + ) -> None: + super().__init__(*args, **kwargs) + self.hardware_target = GB200 if is_blackwell_dc() else H100 + self.backwardpass_ratio = backwardpass_ratio + self.hit_thres = hit_thres + self.include_vae_encoder = include_vae_encoder + self.include_padding = include_padding + self.grad_accum_iter = grad_accum_iter + + # Lazily initialised from model config on first call + self._model_descriptor: OmniMoTModelDescriptor | None = None + self._freeze_und: bool = False + self._vision_gen: bool = True + self._action_gen: bool = False + self._sound_gen: bool = False + self._world_size: int = 1 + self._use_activation_checkpointing: bool = False + + # Accumulation state between every_n windows + self._accumulated_flops = Decimal(0) + self._accumulated_flops_vae = Decimal(0) + self._steps_in_window: int = 0 + self._window_start_time: float | None = None + + # Warm-up counter + self._hit_counter: int = 0 + + # ------------------------------------------------------------------ # + # Lazy initialisation from model + # ------------------------------------------------------------------ # + + def _ensure_initialised(self, model: ImaginaireModel) -> None: + """Build the ``OmniMoTModelDescriptor`` from the live model config.""" + if self._model_descriptor is not None: + return + + # Access VLM config from the language model inside the network + vlm_cfg = model.net.language_model.config # type: ignore[attr-defined] + net_cfg = model.net.config # type: ignore[attr-defined] + + self._freeze_und = getattr(vlm_cfg, "freeze_und", False) + self._vision_gen = getattr(net_cfg, "vision_gen", True) + self._action_gen = getattr(net_cfg, "action_gen", False) + self._sound_gen = getattr(net_cfg, "sound_gen", False) + + # Read activation checkpointing mode from the model config. + # Any non-"none" mode (i.e. "full" or "selective") triggers forward + # recomputation during backward, which adds ~1x layer-forward FLOPs. + model_cfg = getattr(model, "config", None) + ac_cfg = getattr(model_cfg, "activation_checkpointing", None) + ac_mode = getattr(ac_cfg, "mode", "none") + + + # Some activations don't need to be recomputed under selective AC, so + # we need to remove them from the FLOP computation. + self._use_activation_checkpointing = ac_mode != "none" + + # MoE fields (may not exist for dense-only configs) + text_config = vlm_cfg.text_config if hasattr(vlm_cfg, "text_config") else vlm_cfg + + num_experts = getattr(text_config, "num_experts", 0) + num_experts_per_tok = getattr(text_config, "num_experts_per_tok", 0) + moe_intermediate_size = getattr(text_config, "moe_intermediate_size", 0) + use_moe = num_experts > 0 + decoder_sparse_step = getattr(text_config, "decoder_sparse_step", 1) + mlp_only_layers = list(getattr(text_config, "mlp_only_layers", [])) + + self._model_descriptor = get_omni_mot_model_descriptor( + hidden_size=text_config.hidden_size, + num_hidden_layers=text_config.num_hidden_layers, + num_attention_heads=text_config.num_attention_heads, + num_key_value_heads=text_config.num_key_value_heads, + head_dim=getattr(text_config, "head_dim", None), + intermediate_size=text_config.intermediate_size, + vocab_size=text_config.vocab_size, + use_moe=use_moe, + num_experts=num_experts, + num_experts_per_tok=num_experts_per_tok, + moe_intermediate_size=moe_intermediate_size, + decoder_sparse_step=decoder_sparse_step, + mlp_only_layers=mlp_only_layers, + latent_patch_size=getattr(net_cfg, "latent_patch_size", 2), + latent_channel_size=getattr(net_cfg, "latent_channel_size", 48), + action_dim=getattr(net_cfg, "action_dim", 32), + sound_dim=getattr(net_cfg, "sound_dim", 64), + frequency_embedding_size=getattr(net_cfg, "frequency_embedding_size", 256), + predict_text_tokens=getattr(net_cfg, "predict_text_tokens", False), + ) + + self._world_size = torch.distributed.get_world_size() if torch.distributed.is_initialized() else 1 + + # ------------------------------------------------------------------ # + # Per-step accumulation + # ------------------------------------------------------------------ # + + def on_training_step_end( + self, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int = 0, + ) -> None: + # Warm-up: skip first few iterations (compilation, allocation, etc.) + if self._hit_counter < self.hit_thres: + self._hit_counter += 1 + return + + self._ensure_initialised(model) + + # Start the timing window on the first post-warmup step + if self._window_start_time is None: + self._window_start_time = time.monotonic() + + # Extract per-modality token counts from output_batch + und_token_length = output_batch.get("und_token_length") + if und_token_length is None: + return + + und_tokens = int(und_token_length) + vision_tokens = int(output_batch.get("vision_token_length", 0)) + action_tokens = int(output_batch.get("action_token_length", 0)) + sound_tokens = int(output_batch.get("sound_token_length", 0)) + + # Per-split attention metadata for packed sequences + split_lens: list[int] | None = output_batch.get("split_lens") + attn_modes_list: list[str] | None = output_batch.get("attn_modes") + + # Compute FLOPs for this per-device micro-batch. + # B = 1 because token counts are already summed across all samples in + # the packed sequence on this device. + assert self._model_descriptor is not None + step_flops = compute_omni_mot_flops_per_batch( + cfg=self._model_descriptor, + B=1, + text_tokens=und_tokens, + vision_tokens=vision_tokens, + action_tokens=action_tokens, + sound_tokens=sound_tokens, + freeze_und=self._freeze_und, + vision_gen=self._vision_gen, + action_gen=self._action_gen, + sound_gen=self._sound_gen, + backwardpass_ratio=self.backwardpass_ratio, + split_lens=split_lens, + attn_modes=attn_modes_list, + include_padding=self.include_padding, + use_activation_checkpointing=self._use_activation_checkpointing, + ) + + # VAE encoder forward-pass FLOPs (frozen, no backward). + if self.include_vae_encoder: + vae_pixel_shapes = output_batch.get("vae_pixel_shapes") + if vae_pixel_shapes: + for pT, pH, pW in vae_pixel_shapes: + vae_flops = compute_wan_vae_encoder_flops(B=1, T=pT, H=pH, W=pW) + self._accumulated_flops_vae += vae_flops + step_flops += vae_flops + + # When gradient accumulation is used, on_training_step_end is called + # once per optimizer step (not per micro-batch). Multiply by the + # accumulation count so the FLOPs cover all micro-batches in the step. + # For VAE with gradient accumulation we assume all micro-batches have the same FLOP count + if self.grad_accum_iter > 1: + step_flops *= self.grad_accum_iter + + self._accumulated_flops += step_flops + self._steps_in_window += 1 + + # Delegate to EveryN for the periodic reporting logic + super().on_training_step_end(model, data_batch, output_batch, loss, iteration) + + # ------------------------------------------------------------------ # + # Periodic reporting + # ------------------------------------------------------------------ # + + @rank0_only + def every_n_impl( + self, + trainer: ImaginaireTrainer, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int, + ) -> None: + if self._window_start_time is None or self._steps_in_window == 0: + return + + elapsed = time.monotonic() - self._window_start_time + if elapsed <= 0: + return + + if self._accumulated_flops <= 0: + log.warning( + f"Number of calculated FLOPs must be more than 0, got {self._accumulated_flops} at iteration {iteration} for {self._steps_in_window} steps." + ) + + # Achieved TFLOPS *per GPU* over the window + # accumulated_flops is the total per-device FLOPs over all steps in window + achieved_tflops_per_gpu = float(self._accumulated_flops) / elapsed / 1e12 + + avg_flops_per_step = float(self._accumulated_flops) / self._steps_in_window + avg_time_per_step = elapsed / self._steps_in_window + + log_info: dict[str, float] = { + "mfu/achieved_tflops_per_gpu": achieved_tflops_per_gpu, + "mfu/avg_flops_per_step": avg_flops_per_step, + "mfu/avg_time_per_step_s": avg_time_per_step, + "mfu/steps_in_window": float(self._steps_in_window), + "mfu/vae_flops_percentage": float(self._accumulated_flops_vae / self._accumulated_flops) * 100.0, + } + + mfu = ( + achieved_tflops_per_gpu / self.hardware_target.peak_tflops if self.hardware_target.peak_tflops > 0 else 0.0 + ) + log_info[f"mfu/{self.hardware_target.name}"] = mfu + + # W&B log + if wandb.run is not None: + wandb.log(log_info, step=iteration) + + # Reset accumulation window + self._accumulated_flops = Decimal(0) + self._accumulated_flops_vae = Decimal(0) + self._steps_in_window = 0 + self._window_start_time = time.monotonic() diff --git a/cosmos_framework/callbacks/moe_specialization_callback.py b/cosmos_framework/callbacks/moe_specialization_callback.py new file mode 100644 index 0000000..5f1f32c --- /dev/null +++ b/cosmos_framework/callbacks/moe_specialization_callback.py @@ -0,0 +1,192 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +""" +MoE Specialization Callback +============================ +Monitors whether MoE experts are developing distinct, stable roles over training. +A well-trained MoE should have experts that specialize — each processing a different +kind of input — rather than a few generalist experts doing everything while the rest +idle. + + Expert Co-activation Rate + ------------------------- + If two experts frequently fire together on the same token (both in the top-K + selected), they are likely learning redundant representations. Ideally experts + specialize on non-overlapping token types, so co-activation should stay close + to the chance baseline of K/N (e.g. 8/128 ≈ 0.0625 for the 235B model). + + For each layer and each unique expert pair (i, j), we compute: + CoAct(i, j) = N_{i,j} / N_i + where N_{i,j} = number of tokens where both i and j were selected, and N_i = + total tokens routed to expert i. We then summarize across all pairs as max and + mean. A rising mean_coact, especially well above the chance baseline, signals + that the router is collapsing onto a small correlated cluster of experts. + +Buffer ownership +---------------- + coactivation_counts is reset here (in compute_moe_coactivation_metrics). + Per-expert token counts are derived from coactivation_counts itself + (row_sum + col_sum) / (K-1), so this callback is fully independent of + ExpertHeatmap's reset cycle for total_tokens_per_expert. +""" + +import torch +import wandb +from torch.distributed.tensor import DTensor, Partial + +from cosmos_framework.callbacks.every_n import EveryN +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.trainer import ImaginaireTrainer +from cosmos_framework.utils import distributed +from cosmos_framework.model.vfm.vlm.qwen3_vl_moe.qwen3_vl_moe import Qwen3VLMoeTextSparseMoeBlock + + +def _get_device_mesh(vfm: torch.nn.Module): + weight = vfm.language_model.model.layers[0].self_attn.q_proj.weight + return weight.device_mesh if isinstance(weight, DTensor) else None + + +def _allreduce_dtensor(t: torch.Tensor, device_mesh) -> torch.Tensor: + """Sum-reduce a local tensor across all FSDP ranks and return the global tensor.""" + return DTensor.from_local( + t, + device_mesh=device_mesh, + placements=[Partial()] * device_mesh.ndim, + ).full_tensor() + + +def compute_moe_coactivation_metrics(vfm: torch.nn.Module) -> dict[str, dict]: + """ + Compute per-layer Expert Co-activation metrics for both towers. + + For each unique expert pair (i < j) in the upper triangle of the N×N + coactivation matrix, computes: + CoAct(i, j) = N_{i,j} / N_i + where N_{i,j} is the count of tokens where both i and j were in the top-K, + and N_i is the total token count for expert i (the row expert, i.e. the + lower-indexed expert in the pair). + + N_i is derived directly from the co-activation matrix rather than from + the shared total_tokens_per_expert buffer, so this metric is independent + of ExpertHeatmap's reset cycle. Each token routed to expert i contributes + to (K-1) co-activation pairs, so N_i = (row_sum_i + col_sum_i) / (K-1). + + High co-activation relative to the chance baseline (K/N) indicates that + certain expert pairs are systematically selected together — a sign of + redundancy rather than specialization. + + Returns a dict: tower -> { + "layer_indices": list[int] — actual model layer positions + "max_coact": Tensor[num_moe_layers] — worst pair per layer + "mean_coact": Tensor[num_moe_layers] — average over all pairs + "chance_baseline": float — K/N, same for all layers (reference) + } + """ + with torch.no_grad(): + device_mesh = _get_device_mesh(vfm) + if device_mesh is None: + return {} + + results: dict[str, dict] = {} + for tower in ["und", "gen"]: + layer_indices, max_coacts, mean_coacts, chance_baselines = [], [], [], [] + + num_layers = len(vfm.language_model.model.layers) + for layer_idx in range(num_layers): + layer = vfm.language_model.model.layers[layer_idx] + mlp = layer.mlp if tower == "und" else getattr(layer, "mlp_moe_gen", None) + if not isinstance(mlp, Qwen3VLMoeTextSparseMoeBlock): + continue + + coact_counts = _allreduce_dtensor(mlp.get_coactivation_counts(reset=True), device_mesh) # [N, N] + + n = mlp.num_experts + k = mlp.top_k + + # Derive per-expert token counts directly from the co-activation + # matrix so we don't depend on ExpertHeatmap's reset cycle. + # Each token that routes to expert i contributes (K-1) entries + # across row i and column i of the upper-triangle matrix. + tokens_per_expert = (coact_counts.sum(dim=1) + coact_counts.sum(dim=0)).float() / (k - 1) + + mask = torch.triu(torch.ones(n, n, dtype=torch.bool, device=coact_counts.device), diagonal=1) + # CoAct(i, j) = N_{i,j} / N_i — normalise by how often expert i fires overall. + denom = tokens_per_expert.unsqueeze(1).clamp(min=1) # [N, 1] + coact_rates = (coact_counts.float() / denom)[mask] # [N*(N-1)/2] + + layer_indices.append(layer_idx) + max_coacts.append(coact_rates.max()) + mean_coacts.append(coact_rates.mean()) + # Chance baseline = probability two randomly-chosen top-K slots land on the + # same pair under uniform routing = K/N. Constant across layers and steps, + # logged once per tower as a reference line. + chance_baselines.append(k / n) + + if layer_indices: + results[tower] = { + "layer_indices": layer_indices, + "max_coact": torch.stack(max_coacts), + "mean_coact": torch.stack(mean_coacts), + "chance_baseline": chance_baselines[0], # same value for all layers + } + + return results + + +class MoESpecializationCallback(EveryN): + """ + Logs per-layer MoE specialization metrics to W&B every N training steps. + + What it captures + ---------------- + Whether MoE experts are developing distinct routing identities: + + Expert Co-activation (logged every N steps) + - mean_coact / max_coact per layer: how often expert pairs fire together + relative to the chance_baseline (K/N). Values well above the baseline + suggest the router is selecting a redundant cluster of experts rather + than a diverse set. + + W&B layout + ---------- + moe_specialization/coact_chance_baseline/ — flat reference (K/N) + moe_specialization/max_coact//layer_NNN|mean|max + moe_specialization/mean_coact//layer_NNN|mean|max + + Args: + every_n (int): Logging interval in training steps. + """ + + def __init__(self, every_n: int = 100): + super().__init__(every_n=every_n) + + def every_n_impl( + self, + trainer: ImaginaireTrainer, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int, + ) -> None: + vfm = model.net + + coact_results = compute_moe_coactivation_metrics(vfm) + + if not (distributed.is_rank0() and wandb.run): + return + + log_dict: dict[str, float] = {} + + for tower, tower_metrics in coact_results.items(): + layer_indices = tower_metrics.pop("layer_indices") + chance_baseline = tower_metrics.pop("chance_baseline") + log_dict[f"moe_specialization/coact_chance_baseline/{tower}"] = chance_baseline + for metric_name, values in tower_metrics.items(): + for layer_idx, val in zip(layer_indices, values): + log_dict[f"moe_specialization/{metric_name}/{tower}/layer_{layer_idx:03d}"] = val.item() + log_dict[f"moe_specialization/{metric_name}/{tower}/mean"] = values.mean().item() + log_dict[f"moe_specialization/{metric_name}/{tower}/max"] = values.max().item() + + wandb.log(log_dict, step=iteration) diff --git a/cosmos_framework/callbacks/moe_stability_callback.py b/cosmos_framework/callbacks/moe_stability_callback.py new file mode 100644 index 0000000..e223e95 --- /dev/null +++ b/cosmos_framework/callbacks/moe_stability_callback.py @@ -0,0 +1,312 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +""" +MoE Stability Callback +====================== +Monitors whether the MoE router is staying healthy over the course of training. +A healthy router distributes tokens reasonably evenly, keeps all experts alive, +and remains uncertain enough (high entropy) that it is still learning to route. + +Five metrics are tracked per layer, per tower (und / gen): + + Dead Expert Rate + ---------------- + Fraction of experts receiving fewer than 10% of their fair-share of tokens + (i.e. load fraction f_i < 0.1 / N). A dead expert has been effectively shut + out by the router — it gets no gradient signal and its capacity is wasted. + Ideal = 0. A rising dead-expert rate in the gen tower during early training + is a common failure mode. + + Load Imbalance Factor (LIF) + --------------------------- + N * max(f_i), where f_i is the fraction of tokens routed to expert i. + Measures how much the busiest expert is overloaded relative to uniform. + LIF = 1.0 is perfect balance; <= 1.3 is healthy; > 3.0 indicates severe + collapse onto a small set of experts. This is the same quantity watched by + the load-balancing loss, but measured empirically rather than from the loss + objective. + + Router Entropy (normalized) + --------------------------- + Mean per-token Shannon entropy of the full routing distribution, divided by + log(N) to put it on a [0, 1] scale. H = 1 means the router is maximally + uncertain (uniform over all experts); H = 0 means it always picks the same + expert. Early in training entropy is high; we want it to stay reasonably + high (> ~0.7) so the router continues to explore. A sudden drop signals + routing collapse. + + Soft-vs-Hard Effective Experts (normalized) + ------------------------------------------- + Soft and hard effective experts separate what the router *considers* (full + probability distribution, before dispatch) from what top-k dispatch *actually + uses* (empirical token-to-expert assignment, after dispatch). Both are + expressed as a fraction of N, so they sit on the same axis as + router_entropy_normalized. Their lower bounds differ slightly: + soft_eff_normalized is bounded in [1/N, 1]. + hard_eff_normalized is bounded in [K/N, 1] — top-K dispatch always engages + at least K experts in aggregate (the floor case is when every token + picks the same K-expert subset). + + soft_eff_normalized = mean_t exp(H(p_t)) / N + Average per-token router perplexity, divided by N. Asks: what fraction + of experts is the router *considering* on a typical token? Computed + as sum_per_token_soft_eff / total_tokens / N. Note: the unnormalized + numerator is NOT exp of the mean entropy — by Jensen, + mean_t exp(H_t) >= exp(mean_t H_t), and the gap matters when + per-token entropies are heterogeneous. + + hard_eff_normalized = exp(H(f)) / N + where f_i is the empirical fraction of *expert assignments* (not + tokens) that went to expert i: f_i = tokens_per_expert_i / (T * K). + Perplexity of the buffer-wide dispatch distribution, divided by N. + Asks: what fraction of experts is top-k *actually* engaging across the + buffer? A smoother sibling of LIF: where LIF watches the busiest + expert, hard_eff watches the spread of the whole load distribution. + + Interpretation (high/low refer to values close to 1 vs close to 1/N): + + high soft_eff, high hard_eff + Router considers many experts; top-k dispatch also uses many experts. + Broadly healthy routing. + low soft_eff, low hard_eff + Router is confident or collapsed in probability space; dispatch is + also concentrated. Entropy, LIF, and hard usage all agree that + routing is narrow. + high soft_eff, low hard_eff + Router distribution is broad, but top-k dispatch is concentrated — + the "hidden top-k concentration" case where entropy can look healthy + while LIF and co-activation are high. + low soft_eff, high hard_eff + Less common: each token has a sharp router distribution, but + different tokens choose different experts. Per-token confidence with + buffer-wide diversity. + +Buffer ownership +---------------- + This callback is fully self-contained: it reads and resets its own dedicated + buffers (stability_tokens_per_expert, stability_total_tokens, sum_token_entropy, + sum_per_token_soft_eff). It does not depend on ExpertHeatmap's reset cycle. +""" + +import math + +# Fraction of uniform fair-share below which an expert is considered "dead" (e.g. 0.1 → < 10% of K/N). +DEAD_EXPERT_THRESHOLD_MULTIPLIER = 0.1 + +# Smoothing added inside log() to avoid log(0) for experts that received zero +# tokens in the current buffer window. Matches the constant used inside the +# MoE block when accumulating router entropy. +ENTROPY_EPSILON = 1e-9 + +import torch +import wandb +from torch.distributed.tensor import DTensor, Partial + +from cosmos_framework.callbacks.every_n import EveryN +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.trainer import ImaginaireTrainer +from cosmos_framework.utils import distributed +from cosmos_framework.model.vfm.vlm.qwen3_vl_moe.qwen3_vl_moe import Qwen3VLMoeTextSparseMoeBlock + + +def _effective_experts( + sum_per_token_soft_eff: torch.Tensor, + total_tokens: torch.Tensor, + tokens_per_expert: torch.Tensor, +) -> tuple[torch.Tensor, torch.Tensor]: + """Compute (soft_eff, hard_eff) from already-reduced stability buffers. + + Extracted as a pure-tensor function so it can be unit-tested without + instantiating any MoE module or distributed state. + + Args: + sum_per_token_soft_eff: 0-d or [1] tensor holding sum_t exp(H(p_t)) + accumulated across the buffer window. + total_tokens: 0-d or [1] tensor holding the number of tokens seen + since the last reset. + tokens_per_expert: [N] tensor of per-expert token counts over the + same buffer window. + + Returns: + soft_eff: scalar tensor, mean_t exp(H(p_t)) in [1, N]. + hard_eff: scalar tensor, exp(H(f)) over the empirical dispatch + distribution f_i = tokens_per_expert_i / sum_i tokens_per_expert_i. + Bounded in [K, N] (not [1, N]) because top-K dispatch always + engages at least K experts in aggregate. + + Note on hard_eff normalization: + tokens_per_expert is a histogram over the K top-k slots per token, so + it sums to T * K rather than T. We must divide by its own sum (== T*K) + to get a true probability distribution before taking entropy. + Dividing by total_tokens (== T) instead would give a vector summing to + K, producing exp(H) values up to (N/K)^K — orders of magnitude beyond + the intended [K, N] range. + """ + total = total_tokens.float().clamp(min=1) + soft_eff = (sum_per_token_soft_eff.float() / total).squeeze() + + total_assignments = tokens_per_expert.sum().float().clamp(min=1) + f_i = (tokens_per_expert.float() / total_assignments).clamp(min=ENTROPY_EPSILON) + hard_entropy = -(f_i * f_i.log()).sum() + hard_eff = hard_entropy.exp() + + return soft_eff, hard_eff + + +def compute_moe_stability_metrics(vfm: torch.nn.Module) -> dict[str, dict]: + """ + Compute per-layer MoE stability metrics for both towers. + + Iterates over all model layers, skipping any that do not use + Qwen3VLMoeTextSparseMoeBlock (e.g. dense layers when decoder_sparse_step > 1). + Actual model layer indices are preserved so W&B keys (layer_000, layer_042, ...) + always refer to the correct transformer layer regardless of MoE sparsity pattern. + + Returns a dict: tower -> { + "layer_indices": list[int] — actual model layer positions + "dead_expert_rate": Tensor[num_moe_layers] + "lif": Tensor[num_moe_layers] + "router_entropy_normalized": Tensor[num_moe_layers] + "soft_eff_normalized": Tensor[num_moe_layers] — mean_t exp(H(p_t)) / N, in [1/N, 1] + "hard_eff_normalized": Tensor[num_moe_layers] — exp(H(f)) / N, in [1/N, 1] + } + """ + with torch.no_grad(): + num_layers = len(vfm.language_model.model.layers) + + example_weight = vfm.language_model.model.layers[0].self_attn.q_proj.weight + device_mesh = example_weight.device_mesh if isinstance(example_weight, DTensor) else None + + if device_mesh is None: + return {} + + def _allreduce(t: torch.Tensor) -> torch.Tensor: + return DTensor.from_local( + t, + device_mesh=device_mesh, + placements=[Partial()] * device_mesh.ndim, + ).full_tensor() + + results: dict[str, dict] = {} + for tower in ["und", "gen"]: + layer_indices: list[int] = [] + dead_rates: list[torch.Tensor] = [] + lifs: list[torch.Tensor] = [] + entropies: list[torch.Tensor] = [] + soft_effs_norm: list[torch.Tensor] = [] + hard_effs_norm: list[torch.Tensor] = [] + + for layer_idx in range(num_layers): + layer_module = vfm.language_model.model.layers[layer_idx] + # "und" tower uses layer.mlp; "gen" tower uses layer.mlp_moe_gen. + # Both attributes exist on every layer (set in unified_mot.py), but only + # layers where (layer_idx+1) % decoder_sparse_step == 0 are MoE blocks. + mlp_module = layer_module.mlp if tower == "und" else getattr(layer_module, "mlp_moe_gen", None) + if not isinstance(mlp_module, Qwen3VLMoeTextSparseMoeBlock): + continue + + total_tokens_per_expert = _allreduce(mlp_module.get_stability_tokens_per_expert(reset=True)) + total_tokens = _allreduce(mlp_module.get_stability_total_tokens(reset=True)) + sum_token_entropy = _allreduce(mlp_module.get_sum_token_entropy(reset=True)) + sum_per_token_soft_eff = _allreduce(mlp_module.get_sum_per_token_soft_eff(reset=True)) + + n = mlp_module.num_experts + total = total_tokens.float().clamp(min=1) + f_i = total_tokens_per_expert.float() / total # [N] load fraction per expert + + k = mlp_module.top_k + + layer_indices.append(layer_idx) + # Uniform fair share per expert is K/N. "Dead" = below 10% of that. + dead_rates.append((f_i < DEAD_EXPERT_THRESHOLD_MULTIPLIER * k / n).float().mean()) + # LIF = max(f_i) * N / K. Interpretation: + # 1.0 = perfectly balanced (every expert gets its fair share) + # 2.0 = busiest expert handles 2x its fair share + # >3.0 = severe imbalance, consider tuning load-balancing loss + lifs.append(f_i.max() * n / k) + # Mean per-token entropy, normalized to [0, 1] by log(N). + # squeeze() collapses the [1] buffer shape to a 0-d scalar. + entropies.append((sum_token_entropy.float() / total / math.log(n)).squeeze()) + + soft_eff, hard_eff = _effective_experts( + sum_per_token_soft_eff=sum_per_token_soft_eff, + total_tokens=total_tokens, + tokens_per_expert=total_tokens_per_expert, + ) + soft_effs_norm.append(soft_eff / n) + hard_effs_norm.append(hard_eff / n) + + if layer_indices: + results[tower] = { + "layer_indices": layer_indices, + "dead_expert_rate": torch.stack(dead_rates), + "lif": torch.stack(lifs), + "router_entropy_normalized": torch.stack(entropies), + "soft_eff_normalized": torch.stack(soft_effs_norm), + "hard_eff_normalized": torch.stack(hard_effs_norm), + } + + return results + + +class MoEStabilityCallback(EveryN): + """ + Logs per-layer MoE stability metrics to W&B every N training steps. + + What it captures + ---------------- + Whether the MoE router remains in a healthy, balanced state over training. + The metrics collectively answer: are all experts still being used + (dead_expert_rate), is load spread evenly (lif), is the router still + making uncertain, exploratory decisions (router_entropy_normalized), and + do the experts the router considers (soft_eff) match the experts top-k + dispatch actually engages (hard_eff)? + + W&B layout + ---------- + For each metric and each tower, two kinds of series are logged: + - moe_stability///layer_NNN — per model layer time series + - moe_stability///mean|max — summary across all MoE layers + + Metrics logged: dead_expert_rate, lif, router_entropy_normalized, + soft_eff_normalized, hard_eff_normalized. + + Typical healthy ranges: + dead_expert_rate → 0 (any sustained non-zero value is a concern) + lif → <= 1.3 (alarm at > 3.0) + router_entropy_normalized → > 0.7 (collapse if it drops sharply) + soft_eff_normalized, hard_eff_normalized → high; a large gap between + them (e.g. soft high, hard low) indicates hidden top-k concentration + + Args: + every_n (int): Logging interval in training steps. + """ + + def __init__(self, every_n: int = 100): + super().__init__(every_n=every_n) + + def every_n_impl( + self, + trainer: ImaginaireTrainer, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int, + ) -> None: + metrics = compute_moe_stability_metrics(model.net) + + if not (distributed.is_rank0() and wandb.run): + return + + log_dict: dict[str, float] = {} + for tower, tower_metrics in metrics.items(): + layer_indices = tower_metrics.pop("layer_indices") + for metric_name, values in tower_metrics.items(): + for layer_idx, val in zip(layer_indices, values): + log_dict[f"moe_stability/{metric_name}/{tower}/layer_{layer_idx:03d}"] = val.item() + log_dict[f"moe_stability/{metric_name}/{tower}/mean"] = values.mean().item() + log_dict[f"moe_stability/{metric_name}/{tower}/max"] = values.max().item() + + wandb.log(log_dict, step=iteration) diff --git a/cosmos_framework/callbacks/norm_monitor.py b/cosmos_framework/callbacks/norm_monitor.py new file mode 100644 index 0000000..46f51cb --- /dev/null +++ b/cosmos_framework/callbacks/norm_monitor.py @@ -0,0 +1,328 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import os +from typing import Optional + +import torch +import torch.distributed as dist +import wandb +from torch import nn +from torch.distributed.tensor import DTensor + +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.utils import distributed, log, misc +from cosmos_framework.utils.callback import Callback +from cosmos_framework.utils.easy_io import easy_io +from cosmos_framework.data.vfm.sequence_packing import get_gen_seq + +try: + from apex.contrib.layer_norm import FastLayerNorm +except ImportError: + FastLayerNorm = None + + +class NormMonitor(Callback): + def __init__( + self, + every_n: Optional[int] = None, + step_size: int = 1, + layer_norm_only: bool = False, + model_key: Optional[str] = None, + log_stat_wandb: bool = False, + save_s3: bool = False, + track_activations: bool = False, + ): + """Monitor and log parameter/gradient/activation norms during training. + + Args: + every_n: Log statistics every N global steps. If None, logging is disabled. + step_size: Number of micro-steps per global step (for gradient accumulation). + layer_norm_only: If True, only track LayerNorm and Embedding parameters. + If False, track all parameters. + model_key: Attribute name to access the model (e.g., "diffusion_model"). + If None, use the model directly. + log_stat_wandb: If True, log per-parameter statistics to wandb. + If False, only log aggregate norms. + save_s3: If True, save statistics to S3 bucket. + track_activations: If True, track activation norms + and gradients of activations at each transformer block. If set to False, only + weight norms and weight gradient norms will be tracked. + """ + self.every_n = every_n + self.step_size = step_size + self.model_key = model_key + self.layer_norm_only = layer_norm_only + self.log_stat_wandb = log_stat_wandb + self.save_s3 = save_s3 + self.track_activations = track_activations + self.name = self.__class__.__name__ + + # Storage for activation statistics (populated by hooks) + self._activation_stats: dict[str, dict[str, torch.Tensor]] = {} + self._activation_grad_stats: dict[str, dict[str, torch.Tensor]] = {} + self._hooks: list[torch.utils.hooks.RemovableHandle] = [] + self._should_record = False + + def on_train_start(self, model: ImaginaireModel, iteration: int = 0) -> None: + config_job = self.config.job + self.local_dir = f"{config_job.path_local}/norm_monitor" + if distributed.get_rank() == 0: + os.makedirs(self.local_dir, exist_ok=True) + log.info(f"{self.__class__.__name__} callback: local_dir: {self.local_dir}") + + # Register activation hooks if enabled + if self.track_activations: + self._register_activation_hooks(model) + + def _register_activation_hooks(self, model: ImaginaireModel) -> None: + """Register forward and backward hooks on transformer blocks to capture activation statistics. + + Hooks are registered at the block level (on model.model.layers children) rather than + on individual modules inside blocks. This is compatible with torch.compile since + compile is applied per-block, and hooks on the outer block fire outside the compiled graph. + """ + if self.model_key is not None: + model = getattr(model, self.model_key) + + # Get the transformer layers - hooks are registered on each block + if not hasattr(model.net.language_model.model, "layers"): + log.warning( + f"{self.__class__.__name__}: Could not find model.net.language_model.model.layers. " + "Activation tracking requires model structure with model.net.language_model.model.layers." + ) + return + + layers = model.net.language_model.model.layers + + for layer_id, block in layers.named_children(): + block_name = f"blocks.{layer_id}" + + # Forward hook to capture activation norms (block output) + # Also registers a tensor hook for gradient tracking + def make_forward_hook(name: str): + def forward_hook( + mod: nn.Module, inp: tuple[torch.Tensor, ...], out: torch.Tensor | tuple[torch.Tensor, ...] + ) -> None: + if not self._should_record: + return + # We track activation norms of only generation sequences. + activation = get_gen_seq(out[0]) + + # Certain algorithms do more than one pass through the model. + # (E.g. teacher forcing). We merge stats in that case. + new_stats = self._compute_l2_stats(activation) + existing = self._activation_stats.get(name) + if existing is not None: + existing["sq_sum"] += new_stats["sq_sum"] + existing["max"] = torch.max(existing["max"], new_stats["max"]) + else: + self._activation_stats[name] = new_stats + + # Register tensor hook for gradient tracking. + # This works with activation checkpointing (unlike module backward hooks). + def make_tensor_grad_hook(hook_name: str): + def tensor_grad_hook(grad: torch.Tensor | None) -> None: + # The block may get gradients internally via attention, + # even if the output is unused. + if grad is None: + return + + # If there is more than one pass through the model + # (e.g. teacher forcing), then merge the stats. + new_stats = self._compute_l2_stats(grad) + existing = self._activation_grad_stats.get(hook_name) + if existing is not None: + existing["sq_sum"] += new_stats["sq_sum"] + existing["max"] = torch.max(existing["max"], new_stats["max"]) + else: + self._activation_grad_stats[hook_name] = new_stats + + return tensor_grad_hook + + if activation.requires_grad: + activation.register_hook(make_tensor_grad_hook(name)) + + return forward_hook + + forward_handle = block.register_forward_hook(make_forward_hook(block_name)) + self._hooks.append(forward_handle) + + if distributed.is_rank0(): + num_blocks = len(list(layers.named_children())) + log.info(f"{self.__class__.__name__}: Registered activation hooks on {num_blocks} transformer blocks") + + def on_train_end(self, model: ImaginaireModel, iteration: int = 0) -> None: + """Clean up hooks when training ends.""" + for hook in self._hooks: + hook.remove() + self._hooks.clear() + + def on_before_forward( + self, + iteration: int = 0, + ) -> None: + """Enable activation recording before forward pass if this iteration should be logged.""" + if not self.track_activations: + return + global_step = iteration // self.step_size + should_run = global_step % self.every_n == 0 + self._should_record = should_run + if should_run: + # Clear previous activation stats + self._activation_stats.clear() + self._activation_grad_stats.clear() + + def on_before_optimizer_step( + self, + model: ImaginaireModel, + optimizer: torch.optim.Optimizer, + scheduler: torch.optim.lr_scheduler.LRScheduler, + grad_scaler: torch.amp.GradScaler, + iteration: int = 0, + ) -> None: + global_step = iteration // self.step_size + should_run = global_step % self.every_n == 0 + if not should_run: + return + + if self.model_key is not None: + model = getattr(model, self.model_key) + + self._compute_and_log_stats(model, iteration) + + # Disable recording after logging + self._should_record = False + + def _get_named_parameters(self, model: nn.Module) -> dict[str, nn.Parameter]: + """Get named parameters, optionally filtered to layer norm only.""" + named_parameters = {} + if self.layer_norm_only: + ln_modules = (nn.LayerNorm, nn.Embedding) + if FastLayerNorm is not None: + ln_modules += (FastLayerNorm,) + for mn, m in model.named_modules(): + if isinstance(m, ln_modules): + for pn, p in m.named_parameters(): + fpn = f"{mn}.{pn}" if mn else pn + named_parameters[fpn] = p + else: + named_parameters = dict(model.named_parameters()) + return named_parameters + + def _should_track_param(self, param_name: str) -> bool: + """Check if parameter should be tracked based on naming conventions.""" + # Track only generation tower params, exclude EMA params + return "moe_gen" in param_name and "net_ema" not in param_name + + def _compute_l2_stats(self, tensor: torch.Tensor, detach: bool = True) -> dict[str, torch.Tensor]: + """Compute statistics (squared sum and max) for a tensor. + + Args: + tensor: Input tensor to compute statistics for. + detach: If True, detach the tensor before computing stats. + + Returns: + Dictionary with "sq_sum" (squared sum for L2 norm) and "max" (absolute max). + """ + data = tensor.detach() if detach else tensor + if isinstance(data, DTensor): + data = data.to_local() + + return { + "sq_sum": (data.float() ** 2).sum(), + "max": data.abs().max(), + } + + @misc.timer("norm_monitor") + def _compute_and_log_stats(self, model: nn.Module, iteration: int = 0) -> None: + """FSDP-efficient implementation using local shards + all_reduce. + + Instead of gathering full parameters with summon_full_params (expensive), + we compute local statistics on each rank's shard and use all_reduce to + aggregate them across all ranks. + """ + named_parameters = self._get_named_parameters(model) + + # Accumulators for local shard statistics (squared sum for L2 norm) + local_param_sq_sum = torch.tensor(0.0, device="cuda", dtype=torch.float32) + local_grad_sq_sum = torch.tensor(0.0, device="cuda", dtype=torch.float32) + + # Per-parameter stats: {param_name: [local_sq_sum, local_max]} + per_param_stats: dict[str, dict[str, torch.Tensor]] = {} + per_grad_stats: dict[str, dict[str, torch.Tensor]] = {} + + for param_name, param in named_parameters.items(): + if not self._should_track_param(param_name): + continue + + # Compute local statistics on this rank's shard + per_param_stats[param_name] = self._compute_l2_stats(param) + local_param_sq_sum += per_param_stats[param_name]["sq_sum"] + + if param.grad is not None: + per_grad_stats[param_name] = self._compute_l2_stats(param.grad, detach=False) + local_grad_sq_sum += per_grad_stats[param_name]["sq_sum"] + + # All-reduce to aggregate statistics across all FSDP ranks + dist.all_reduce(local_param_sq_sum, op=dist.ReduceOp.SUM) + dist.all_reduce(local_grad_sq_sum, op=dist.ReduceOp.SUM) + + # All-reduce per-parameter stats + for param_name, stats_dict in per_param_stats.items(): + dist.all_reduce(stats_dict["sq_sum"], op=dist.ReduceOp.SUM) + dist.all_reduce(stats_dict["max"], op=dist.ReduceOp.MAX) + + for param_name, stats_dict in per_grad_stats.items(): + dist.all_reduce(stats_dict["sq_sum"], op=dist.ReduceOp.SUM) + dist.all_reduce(stats_dict["max"], op=dist.ReduceOp.MAX) + + # All-reduce activation stats (activations are replicated, so reduce across all ranks for consistency) + for module_name, stats_dict in self._activation_stats.items(): + dist.all_reduce(stats_dict["sq_sum"], op=dist.ReduceOp.SUM) + dist.all_reduce(stats_dict["max"], op=dist.ReduceOp.MAX) + + for module_name, stats_dict in self._activation_grad_stats.items(): + dist.all_reduce(stats_dict["sq_sum"], op=dist.ReduceOp.SUM) + dist.all_reduce(stats_dict["max"], op=dist.ReduceOp.MAX) + + # Only rank 0 logs the results + if distributed.is_rank0(): + important_info = { + "trainer/global_step": iteration, + "sample_counter": getattr(self.trainer, "sample_counter", iteration), + "total_param_l2_norm": local_param_sq_sum.sqrt().item(), + } + if local_grad_sq_sum > 0: + important_info["total_grad_l2_norm"] = local_grad_sq_sum.sqrt().item() + + stats = {} + for param_name, stats_dict in per_param_stats.items(): + l2_norm = stats_dict["sq_sum"].sqrt() + stats[f"stats/weight_norm/{param_name}"] = l2_norm.item() + stats[f"stats/weight_max/{param_name}"] = stats_dict["max"].item() + + for param_name, stats_dict in per_grad_stats.items(): + l2_norm = stats_dict["sq_sum"].sqrt() + stats[f"stats/grad_norm/{param_name}"] = l2_norm.item() + stats[f"stats/grad_max/{param_name}"] = stats_dict["max"].item() + + # Add activation stats + for module_name, stats_dict in self._activation_stats.items(): + l2_norm = stats_dict["sq_sum"].sqrt() + stats[f"stats/act_norm/{module_name}"] = l2_norm.item() + stats[f"stats/act_max/{module_name}"] = stats_dict["max"].item() + + for module_name, stats_dict in self._activation_grad_stats.items(): + l2_norm = stats_dict["sq_sum"].sqrt() + stats[f"stats/act_grad_norm/{module_name}"] = l2_norm.item() + stats[f"stats/act_grad_max/{module_name}"] = stats_dict["max"].item() + + if wandb.run is not None: + if self.log_stat_wandb: + wandb.log({**stats, **important_info}, step=iteration) + else: + wandb.log(important_info, step=iteration) + + if self.save_s3: + easy_io.dump({**stats, **important_info}, f"s3://rundir/{self.name}/stats_{iteration:09d}.pt") diff --git a/cosmos_framework/callbacks/ofu.py b/cosmos_framework/callbacks/ofu.py new file mode 100644 index 0000000..8fd6113 --- /dev/null +++ b/cosmos_framework/callbacks/ofu.py @@ -0,0 +1,281 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""OFU (Operational FLOPs Utilization) callback for OmniMoT training. + +Computes and logs OFU metrics by launching ``nvidia-smi dmon`` as a background +subprocess and parsing the Tensor Core activity (mmaact) and processor clock +(pclk) columns. OFU is defined as:: + + OFU = mmaact * (pclk / max_pclk) + +where ``max_pclk`` is the max boost clock for the detected hardware (e.g. +1980 MHz for H100, 2062 MHz for GB200). The result is in the 0-100 range. +""" + +from __future__ import annotations + +import subprocess +import threading +from collections import defaultdict +from dataclasses import dataclass + +import torch +import wandb + +from cosmos_framework.model.attention.utils import is_blackwell_dc +from cosmos_framework.callbacks.every_n import EveryN +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.trainer import ImaginaireTrainer +from cosmos_framework.utils import log +from cosmos_framework.utils.distributed import is_rank0, rank0_only + + +@dataclass +class HardwareTarget: + """Hardware-specific constants for OFU normalisation. + + Attributes: + name: Human-readable name (used as W&B tag, e.g. "H100"). + max_pclk_mhz: Max boost SM clock in MHz used to normalise OFU. + """ + + name: str + max_pclk_mhz: float + + +# Pre-defined hardware targets +H100 = HardwareTarget(name="H100", max_pclk_mhz=1980.0) +GB200 = HardwareTarget(name="GB200", max_pclk_mhz=2062.0) + + +class OFUCallback(EveryN): + """Callback that computes and logs Operational FLOPs Utilization (OFU) to W&B. + + OFU = mmaact * (pclk / max_pclk), where mmaact is the MMA activity + percentage and pclk is the current processor clock from ``nvidia-smi dmon``. + ``max_pclk`` is determined from the detected hardware (H100 or GB200). + The result is in the 0-100 range. + + The callback launches ``nvidia-smi dmon`` as a background subprocess on + ``on_train_start`` and a daemon thread continuously reads its output. + At every logging interval, accumulated samples are consumed, averaged per GPU + and overall, and logged to W&B under ``ofu/{hardware_name}``. + + Args: + hit_thres: Number of warm-up training iterations to skip before logging. + """ + + def __init__( + self, + *args, + hit_thres: int = 5, + **kwargs, + ) -> None: + super().__init__(*args, **kwargs) + self.hardware_target = GB200 if is_blackwell_dc() else H100 + self.hit_thres = hit_thres + + # Subprocess state + self._process: subprocess.Popen | None = None + self._reader_thread: threading.Thread | None = None + self._stop_event = threading.Event() + + # Buffered samples protected by a lock: list of (gpu_idx, mmaact, pclk) + self._lock = threading.Lock() + self._samples: list[tuple[int, float, float]] = [] + + # Column indices parsed from the header (set by _reader_loop) + self._col_gpu: int | None = None + self._col_mmaact: int | None = None + self._col_pclk: int | None = None + + # Warm-up counter + self._hit_counter: int = 0 + + # ------------------------------------------------------------------ # + # Background reader + # ------------------------------------------------------------------ # + + def _parse_header(self, line: str) -> bool: + """Parse a dmon header line to locate column indices. + + Called on every ``#`` line because nvidia-smi dmon reprints the header + every few seconds. Returns True if ``gpu``, ``mmaact``, and ``pclk`` + columns are all found; warns only when the column-names line (identified + by the presence of ``gpu``) lacks a required column. Silently ignores + the units line (``# Idx W C ...``) which does not contain ``gpu``. + """ + cols = line.lstrip("#").strip().split() + col_map = {name.lower(): idx for idx, name in enumerate(cols)} + gpu_idx = col_map.get("gpu") + mmaact_idx = col_map.get("mmaact") + pclk_idx = col_map.get("pclk") + if gpu_idx is not None and mmaact_idx is not None and pclk_idx is not None: + if self._col_mmaact is None: + log.info(f"OFUCallback: found mmaact at column {mmaact_idx}, pclk at column {pclk_idx}") + self._col_gpu = gpu_idx + self._col_mmaact = mmaact_idx + self._col_pclk = pclk_idx + return True + if gpu_idx is not None: + missing = [name for name, idx in [("mmaact", mmaact_idx), ("pclk", pclk_idx)] if idx is None] + log.warning( + f"OFUCallback: column(s) {missing} not found in nvidia-smi dmon header: {cols}. " + "OFU metrics will not be available." + ) + return False + + def _reader_loop(self) -> None: + """Background thread that reads nvidia-smi dmon output line-by-line.""" + assert self._process is not None and self._process.stdout is not None + + for line in self._process.stdout: + if self._stop_event.is_set(): + break + line = line.strip() + if not line: + continue + + # Header lines repeat every few seconds — always re-parse so that a + # missed or failed first parse is recovered on the next occurrence. + if line.startswith("#"): + self._parse_header(line) + continue + + # Skip data lines until we have column indices + if self._col_gpu is None or self._col_mmaact is None or self._col_pclk is None: + continue + + parts = line.split() + try: + gpu_idx = int(parts[self._col_gpu]) + mmaact = float(parts[self._col_mmaact]) + pclk = float(parts[self._col_pclk]) + except (ValueError, IndexError): + continue + + with self._lock: + self._samples.append((gpu_idx, mmaact, pclk)) + + # ------------------------------------------------------------------ # + # Lifecycle hooks + # ------------------------------------------------------------------ # + + def on_train_start(self, model: ImaginaireModel, iteration: int = 0) -> None: + if not is_rank0(): + return + + try: + # --gpm-metrics 5 means that we access Tensor Activity under mmaact column. + # -d 5 means that we sample the data every 5 seconds. + cmd = ["nvidia-smi", "dmon", "--gpm-metrics", "5", "-d", "5"] + self._process = subprocess.Popen( + cmd, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + bufsize=1, # line-buffered + ) + self._reader_thread = threading.Thread(target=self._reader_loop, daemon=True) + self._reader_thread.start() + log.info(f"OFUCallback: launched nvidia-smi dmon --gpm-metrics 5") + except FileNotFoundError: + log.warning("OFUCallback: nvidia-smi not found, OFU metrics will not be available") + except Exception as e: + log.warning(f"OFUCallback: failed to launch nvidia-smi dmon: {e}") + + def on_train_end(self, model: ImaginaireModel, iteration: int = 0) -> None: + if not is_rank0(): + return + self._stop_event.set() + if self._process is not None: + try: + self._process.terminate() + self._process.wait(timeout=5) + except ProcessLookupError: + pass # already exited + except subprocess.TimeoutExpired: + self._process.kill() + self._process = None + if self._reader_thread is not None: + self._reader_thread.join(timeout=5) + self._reader_thread = None + + # ------------------------------------------------------------------ # + # Per-step gating + # ------------------------------------------------------------------ # + + def on_training_step_end( + self, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int = 0, + ) -> None: + # All ranks must enter super().on_training_step_end() so they reach the + # distributed barrier inside EveryN. Only rank 0 has samples to clear. + if self._hit_counter < self.hit_thres: + self._hit_counter += 1 + if self._hit_counter == self.hit_thres: + # Discard samples collected during warm-up (compilation, allocation, etc.) + with self._lock: + self._samples.clear() + return + # Delegate to EveryN for the periodic reporting logic + super().on_training_step_end(model, data_batch, output_batch, loss, iteration) + + # ------------------------------------------------------------------ # + # Periodic reporting + # ------------------------------------------------------------------ # + + @rank0_only + def every_n_impl( + self, + trainer: ImaginaireTrainer, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int, + ) -> None: + if self._process is None: + return + + # Drain buffered samples + with self._lock: + samples = list(self._samples) + self._samples.clear() + + if not samples: + log.warning( + f"OFUCallback: no nvidia-smi samples collected at iteration {iteration}. " + "Check that the dmon subprocess launched and that the mmaact column is present." + ) + return + + # Compute per-GPU OFU: mmaact * (pclk / max_pclk) + max_pclk = self.hardware_target.max_pclk_mhz + gpu_ofu: dict[int, list[float]] = defaultdict(list) + gpu_mmaact: dict[int, list[float]] = defaultdict(list) + gpu_pclk: dict[int, list[float]] = defaultdict(list) + for gpu_idx, mmaact, pclk in samples: + gpu_ofu[gpu_idx].append(mmaact * (pclk / max_pclk)) + gpu_mmaact[gpu_idx].append(mmaact) + gpu_pclk[gpu_idx].append(pclk) + + # Overall averages across all GPUs and samples + all_ofu = [v for vals in gpu_ofu.values() for v in vals] + all_mmaact = [v for vals in gpu_mmaact.values() for v in vals] + all_pclk = [v for vals in gpu_pclk.values() for v in vals] + + log_info: dict[str, float] = { + f"ofu/{self.hardware_target.name}": sum(all_ofu) / len(all_ofu), + "ofu/mmaact": sum(all_mmaact) / len(all_mmaact), + "ofu/avg_pclk_mhz": sum(all_pclk) / len(all_pclk), + "ofu/num_samples": float(len(samples)), + } + + if wandb.run is not None: + wandb.log(log_info, step=iteration) diff --git a/cosmos_framework/callbacks/param_count.py b/cosmos_framework/callbacks/param_count.py new file mode 100644 index 0000000..6557c38 --- /dev/null +++ b/cosmos_framework/callbacks/param_count.py @@ -0,0 +1,38 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Union + +import torch.nn as nn + +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.utils import distributed, log +from cosmos_framework.utils.callback import Callback +from cosmos_framework.utils.count_params import count_params +from cosmos_framework.utils.distributed import rank0_only +from cosmos_framework.utils.easy_io import easy_io + + +class ParamCount(Callback): + def __init__( + self, + save_s3: bool = False, + ): + self.save_s3 = save_s3 + self.name = self.__class__.__name__ + + @rank0_only + def on_train_start(self, model: Union[ImaginaireModel, list[nn.Module]], iteration: int = 0) -> None: + if isinstance(model, list): + num_param = sum([count_params(m) for m in model]) + else: + num_param = count_params(model) + + log.info(f"Total number of parameters on current rank: {num_param}", rank0_only=False) + info = { + "num_parameters": num_param, + } + + if self.save_s3: + rank = distributed.get_rank() + easy_io.dump(info, f"s3://rundir/{self.name}_{rank}.yaml") diff --git a/cosmos_framework/callbacks/sequence_packing_padding.py b/cosmos_framework/callbacks/sequence_packing_padding.py new file mode 100644 index 0000000..c4ab4b8 --- /dev/null +++ b/cosmos_framework/callbacks/sequence_packing_padding.py @@ -0,0 +1,58 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import torch +import wandb + +import cosmos_framework.data.vfm.sequence_packing as sequence_packing +from cosmos_framework.callbacks.every_n import EveryN +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.trainer import ImaginaireTrainer + + +class SequencePackingPadding(EveryN): + """ + Callback that saves lengths to which und and gen sequences are padded. This information will be used + to compute FLOPs done during training. + + Args: + every_n (int): Frequency with which callback is run during training. + """ + + def __init__(self, every_n: int = 500): + super().__init__(every_n=every_n, step_size=1, barrier_after_run=False, run_at_start=True) + + def every_n_impl( + self, + trainer: ImaginaireTrainer, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int, + ) -> None: + if wandb.run: + log_dict = { + "SequencePackingPadding/max_causal_len_image_batch": sequence_packing.MAX_CAUSAL_LEN_IMAGE_BATCH, + "SequencePackingPadding/max_full_len_image_batch": sequence_packing.MAX_FULL_LEN_IMAGE_BATCH, + "SequencePackingPadding/max_causal_len_video_batch": sequence_packing.MAX_CAUSAL_LEN_VIDEO_BATCH, + "SequencePackingPadding/max_full_len_video_batch": sequence_packing.MAX_FULL_LEN_VIDEO_BATCH, + } + modality = "video" + if "is_image_batch" in output_batch: + modality = "image" if output_batch["is_image_batch"] else "video" + if "und_token_length" in output_batch: + log_dict[f"SequencePackingPadding/und_token_length_{modality}"] = output_batch["und_token_length"] + if "gen_token_length" in output_batch: + log_dict[f"SequencePackingPadding/gen_token_length_{modality}"] = output_batch["gen_token_length"] + if "action_token_length" in output_batch: + log_dict[f"SequencePackingPadding/action_token_length"] = output_batch["action_token_length"] + if "sound_token_length" in output_batch: + log_dict[f"SequencePackingPadding/sound_token_length"] = output_batch["sound_token_length"] + if "vision_token_length" in output_batch: + log_dict[f"SequencePackingPadding/vision_token_length"] = output_batch["vision_token_length"] + + wandb.log( + log_dict, + step=iteration, + ) diff --git a/cosmos_framework/callbacks/sigma_loss_analysis.py b/cosmos_framework/callbacks/sigma_loss_analysis.py new file mode 100644 index 0000000..3a7f18d --- /dev/null +++ b/cosmos_framework/callbacks/sigma_loss_analysis.py @@ -0,0 +1,344 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from dataclasses import dataclass +from typing import Optional + +import matplotlib +import matplotlib.pyplot as plt +import numpy as np +import torch +import torch.distributed as dist +import wandb + +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.utils import distributed, misc +from cosmos_framework.utils.callback import Callback +from cosmos_framework.utils.easy_io import easy_io + + +def _get_quantile_bins(n=10) -> np.ndarray: + """Get predefined bins based on logarithmically spaced values""" + points = torch.linspace(0, 1, n + 1) + return points.numpy() + + +@dataclass +class _SigmaLossCache: + """A fixed-size queue for caching sigma and loss tensors. + + Stores sigma/loss pairs on CPU. + When the total number of elements exceeds queue_size, the oldest entries + are automatically removed to maintain the size limit. + + Args: + queue_size: Maximum number of elements to store in the cache. + """ + + def __init__(self, queue_size: int = 2000): + self.queue_size = queue_size + self.reset() + + def reset(self): + self.sigma_list: list[torch.Tensor] = [] + self.loss_list: list[torch.Tensor] = [] + self._total_elements: int = 0 + + def add(self, sigma: torch.Tensor, loss: torch.Tensor): + # Convert to bf16 and store on CPU + sigma_cpu = sigma.detach().cpu().to(torch.bfloat16) + loss_cpu = loss.detach().cpu().to(torch.bfloat16) + + self.sigma_list.append(sigma_cpu) + self.loss_list.append(loss_cpu) + self._total_elements += sigma_cpu.numel() + + # Remove oldest elements if queue exceeds max size + while self._total_elements > self.queue_size and len(self.sigma_list) > 1: + removed_sigma = self.sigma_list.pop(0) + self.loss_list.pop(0) + self._total_elements -= removed_sigma.numel() + + def get_arrays(self) -> tuple[torch.Tensor, torch.Tensor]: + if not self.sigma_list: + return torch.tensor([], dtype=torch.bfloat16), torch.tensor([], dtype=torch.bfloat16) + + sigma_arr = torch.cat(self.sigma_list, dim=0) # [N_total] (concatenated across cached batches) + loss_arr = torch.cat(self.loss_list, dim=0) # [N_total] + + return sigma_arr, loss_arr + + +class SigmaLossAnalysis(Callback): + """Analyze the relationship between sigma (noise level) and flow matching loss. + + This callback tracks per-instance flow matching losses at different sigma values + during training. It maintains separate caches for image and video batches, + periodically aggregates statistics across all distributed ranks, and logs + the results to wandb. + + The analysis helps understand how well the model learns to denoise at different + noise levels, which is useful for diagnosing training dynamics in flow matching + models. + + Args: + every_n: Log statistics every N iterations. + every_n_viz: Create visualization plots every N iterations (must be multiple of every_n). + save_s3: If True, save raw data to S3 for offline analysis. + """ + + def __init__( + self, + every_n: int = 1, + every_n_viz: int = 1, + save_s3: bool = False, + ) -> None: + super().__init__() + self.save_s3 = save_s3 + self.every_n = every_n + assert every_n_viz % every_n == 0, "every_n_viz must be a multiple of every_n in sigma_loss_analysis callback" + self.every_n_viz = every_n_viz + self.name = self.__class__.__name__ + + self.image_cache = _SigmaLossCache(queue_size=2000) + self.video_cache = _SigmaLossCache(queue_size=2000) + + def _create_analysis_plots( + self, + sigma_arr: torch.Tensor, + loss_arr: torch.Tensor, # [N] # [N] + ) -> Optional[wandb.Image]: + if len(sigma_arr) == 0: + return None + + # Convert to numpy for plotting + sigma_np = sigma_arr.cpu().float().numpy() + loss_np = loss_arr.cpu().float().numpy() + + fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5)) + + # Get predefined bins based on logarithmically spaced values + sigma_bins = _get_quantile_bins(10) + + # y_tick_min, y_tick_max = 0, 1.0 + y_tick_min, y_tick_max = 0, 1.0 + # 2D histogram with exponential sigma bins and fixed [0,1] loss range + loss_bins = np.linspace(y_tick_min, y_tick_max, 20) + + counts, xedges, yedges = np.histogram2d(sigma_np, loss_np, bins=(sigma_bins, loss_bins)) + if counts.max() < 0.1: + return None + + # Plot heatmap with exponential scale colormap + im = ax1.imshow( + counts.T, + origin="lower", + aspect="auto", + extent=[sigma_bins[0], sigma_bins[-1], y_tick_min, y_tick_max], + norm=matplotlib.colors.LogNorm(vmin=1, vmax=counts.max()), + ) + plt.colorbar(im, ax=ax1) + + # Set fixed loss ticks from 0 to 1 + yticks = np.linspace(y_tick_min, y_tick_max, 6) + ax1.set_yticks(yticks) + ax1.set_yticklabels([f"{y:.1f}" for y in yticks]) + + ax1.set_xlabel("Sigma") + ax1.set_ylabel("Loss") + title = "Sigma vs Loss Distribution" + ax1.set_title(title) + + # Sigma histogram with loss statistics per bin + hist_counts, _ = np.histogram(sigma_np, bins=sigma_bins) + bin_indices = np.digitize(sigma_np, sigma_bins) - 1 + + # Calculate statistics per bin + n_bins = len(sigma_bins) - 1 + means = np.zeros(n_bins) + stds = np.zeros(n_bins) + for i in range(n_bins): + bin_mask = bin_indices == i + if bin_mask.any(): + means[i] = loss_np[bin_mask].mean() + stds[i] = loss_np[bin_mask].std() + else: + means[i] = np.nan + stds[i] = np.nan + + # Plot histogram + bin_centers = (sigma_bins[:-1] + sigma_bins[1:]) / 2 + ax2.bar(bin_centers, hist_counts, width=np.diff(sigma_bins), alpha=0.3, align="center") + + # Plot loss statistics on twin axis + ax2_twin = ax2.twinx() + valid_mask = ~np.isnan(means) + ax2_twin.errorbar( + bin_centers[valid_mask], means[valid_mask], yerr=stds[valid_mask], color="red", fmt="o-", alpha=0.5 + ) + + ax2.set_xlabel("Sigma (Log Scale)") + ax2.set_ylabel("Count") + ax2_twin.set_ylabel("Loss (mean ± std)") + title = "Sigma Distribution with Loss Statistics" + ax2.set_title(title) + + # Add grid for better readability + ax1.grid(True, alpha=0.3) + ax2.grid(True, alpha=0.3) + + # Create log-scale labels + sigma_labels = [f"{val:.1e}" for val in sigma_bins] + ax1.set_xticks(sigma_bins[1:-1]) # Skip boundary bins + ax1.set_xticklabels(sigma_labels[1:-1], rotation=45) + ax1.set_xscale("linear") + ax2.set_xticks(sigma_bins[1:-1]) + ax2.set_xticklabels(sigma_labels[1:-1], rotation=45) + ax2.set_xscale("linear") + + plt.tight_layout() + fig_img = wandb.Image(fig) + plt.close(fig) + + return fig_img + + def _process_stats(self, sigma: torch.Tensor, loss: torch.Tensor) -> dict: + """Calculate summary statistics for sigma and loss distributions. + + Args: + sigma: Tensor of sigma (noise level) values. + loss: Tensor of corresponding loss values. + + Returns: + Dictionary containing: + - sigma_log_mean: Mean of log(sigma). Log-space is used since sigma spans + multiple orders of magnitude, a standard practice on flow matching / EDM models. + - sigma_log_std: Standard deviation of log(sigma). + - loss_mean: Average loss across all samples. + - loss_std: Standard deviation of loss, measuring spread. + - loss_min: Minimum loss value observed. + - loss_max: Maximum loss value observed. + - loss_median: Median (50th percentile) loss, robust to outliers. + - loss_q1: First quartile (25th percentile) of loss. + - loss_q3: Third quartile (75th percentile) of loss. + """ + return { + "sigma_log_mean": float(sigma.log().mean()), + "sigma_log_std": float(sigma.log().std()), + "loss_mean": float(loss.mean()), + "loss_std": float(loss.std()), + "loss_min": float(loss.min()), + "loss_max": float(loss.max()), + "loss_median": float(loss.median()), + "loss_q1": float(torch.quantile(loss.float(), 0.25)), + "loss_q3": float(torch.quantile(loss.float(), 0.75)), + } + + def _gather_and_save(self, cache: _SigmaLossCache, iteration: int, prefix: str, log_viz: bool = True) -> dict: + info = {} + + # Gather data from all ranks + local_sigma, local_loss = cache.get_arrays() + world_size = dist.get_world_size() + + if world_size > 1: + # Gather sizes first + local_size = torch.tensor([len(local_sigma)], dtype=torch.long, device="cuda") # [1] + sizes = [torch.zeros_like(local_size) for _ in range(world_size)] + dist.all_gather(sizes, local_size) + sizes = [s.item() for s in sizes] + + # Gather data + max_size = max(sizes) + if max_size > 0: + # Move to GPU for gathering + padded_sigma = torch.zeros(max_size, dtype=torch.bfloat16, device="cuda") # [max_size] + padded_loss = torch.zeros(max_size, dtype=torch.bfloat16, device="cuda") # [max_size] + + if len(local_sigma) > 0: + padded_sigma[: len(local_sigma)] = local_sigma.cuda() + padded_loss[: len(local_loss)] = local_loss.cuda() + + all_sigma = [torch.zeros_like(padded_sigma) for _ in range(world_size)] + all_loss = [torch.zeros_like(padded_loss) for _ in range(world_size)] + + dist.all_gather(all_sigma, padded_sigma) + dist.all_gather(all_loss, padded_loss) + + if distributed.is_rank0(): + # Combine data from all ranks + valid_sigma = [] + valid_loss = [] + for sigma, loss, size in zip(all_sigma, all_loss, sizes): + if size > 0: + valid_sigma.append(sigma[:size]) + valid_loss.append(loss[:size]) + + if valid_sigma: + sigma_arr = torch.cat(valid_sigma) # [N_total] (across all ranks) + loss_arr = torch.cat(valid_loss) # [N_total] + + # Overall statistics + info[f"{prefix}/total_samples"] = sigma_arr.shape[0] + + # Calculate statistics + stats = self._process_stats(sigma_arr, loss_arr) + info.update({f"{prefix}/{k}": v for k, v in stats.items()}) + + # Create visualization + if log_viz: + fig_img = self._create_analysis_plots(sigma_arr, loss_arr) + print(fig_img) + if fig_img is not None: + info[f"{prefix}/distribution_plot"] = fig_img + + if self.save_s3: + save_data = { + "sigma": sigma_arr.cpu(), + "loss": loss_arr.cpu(), + "stats": {k: v for k, v in info.items() if not isinstance(v, wandb.Image)}, + } + easy_io.dump( + save_data, + f"s3://rundir/{self.name}/{prefix}_Iter{iteration:09d}.pkl", + ) + + cache.reset() + return info + + def on_training_step_end( + self, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int = 0, + ): + sigma = output_batch["sigma"] + fm_loss_vision_per_instance = output_batch["flow_matching_loss_vision_per_instance"] + + # sigma is [B] (base), [B,1] (TF), or [B,T_max] (DF); reduce to [B] for logging + assert sigma.ndim <= 2, f"Sigma should be [B] or [B,T_max], got shape {sigma.shape}" + if sigma.ndim == 2: + sigma = sigma.mean(dim=-1) # [B] (reduced from [B,T_max] or [B,1]) + + if model.is_image_batch(data_batch): + self.image_cache.add(sigma, fm_loss_vision_per_instance) + else: + self.video_cache.add(sigma, fm_loss_vision_per_instance) + + if iteration % self.every_n == 0: + info = {} + + with misc.timer("sigma_loss_analysis"): + log_viz = iteration % self.every_n_viz == 0 + # Process image data + if len(self.image_cache.sigma_list) > 0: + info.update(self._gather_and_save(self.image_cache, iteration, "sigma_loss_image", log_viz=log_viz)) + + # Process video data + if len(self.video_cache.sigma_list) > 0: + info.update(self._gather_and_save(self.video_cache, iteration, "sigma_loss_video", log_viz=log_viz)) + + if distributed.is_rank0() and info and wandb.run: + wandb.log(info, step=iteration) diff --git a/cosmos_framework/callbacks/skip_nan_step.py b/cosmos_framework/callbacks/skip_nan_step.py new file mode 100644 index 0000000..c0c567a --- /dev/null +++ b/cosmos_framework/callbacks/skip_nan_step.py @@ -0,0 +1,81 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from __future__ import annotations + +import torch +import torch.distributed as dist + +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.utils import log +from cosmos_framework.utils.callback import Callback + + +class SkipNaNStep(Callback): + """Skip the optimizer step only when ALL ranks produce NaN/Inf loss. + + When only some ranks produce NaN, the existing GradClip callback's + nan_to_num handling is sufficient (NaN gradients become 0, valid + gradients from clean ranks are still used). This callback only + intervenes when every rank has NaN, meaning no useful gradient + signal exists. + + The all-reduce ensures all ranks agree on skip/no-skip, preventing + NCCL desync. + + Args: + max_consecutive_nan: Abort training after this many consecutive + all-rank-NaN optimizer steps. Set to 0 to disable the limit. + """ + + def __init__(self, max_consecutive_nan: int = 100) -> None: + super().__init__() + self.max_consecutive_nan = max_consecutive_nan + self._nan_detected = False + self._consecutive_nan_count = 0 + + def on_before_backward( + self, + model: ImaginaireModel, + loss: torch.Tensor, + iteration: int = 0, + ) -> None: + if torch.isnan(loss).any() or torch.isinf(loss).any(): + self._nan_detected = True + + def on_before_optimizer_step( + self, + model: ImaginaireModel, + optimizer: torch.optim.Optimizer, + scheduler: torch.optim.lr_scheduler.LRScheduler, + grad_scaler: torch.amp.GradScaler, + iteration: int = 0, + ) -> None: + nan_flag = torch.tensor([1.0 if self._nan_detected else 0.0], device="cuda") + dist.all_reduce(nan_flag, op=dist.ReduceOp.SUM) + nan_rank_count = int(nan_flag.item()) + world_size = dist.get_world_size() + + if nan_rank_count > 0 and nan_rank_count < world_size: + self._consecutive_nan_count = 0 + + elif nan_rank_count == world_size: + for param in model.parameters(): + if param.grad is not None: + param.grad.zero_() + + self._consecutive_nan_count += 1 + log.warning( + f"ALL ranks NaN/Inf at iteration {iteration}, skipping optimizer step " + f"(consecutive: {self._consecutive_nan_count})", + ) + + if self.max_consecutive_nan > 0 and self._consecutive_nan_count >= self.max_consecutive_nan: + raise RuntimeError( + f"Training unstable: all-rank NaN/Inf loss for {self._consecutive_nan_count} " + f"consecutive optimizer steps at iteration {iteration}. Aborting.", + ) + else: + self._consecutive_nan_count = 0 + + self._nan_detected = False diff --git a/cosmos_framework/callbacks/termination_signal_checkpoint.py b/cosmos_framework/callbacks/termination_signal_checkpoint.py new file mode 100644 index 0000000..04ca3d9 --- /dev/null +++ b/cosmos_framework/callbacks/termination_signal_checkpoint.py @@ -0,0 +1,167 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Callback that saves an emergency checkpoint before a Slurm job is killed. + +Slurm signal timeline +--------------------- + +**Timeout** (job hits ``--time`` limit):: + + T SIGUSR1 → batch shell (N is 300 via ``--signal=B:SIGUSR1@300``) + T+N sec SIGTERM → all processes + T+N+KillWait SIGKILL → anything still alive (KillWait is 30s) + +**Preemption** (higher-priority job needs the nodes):: + + T SIGUSR1 → batch shell + T+Grace SIGTERM + SIGKILL (GraceTime is 300s in GCP-IAD) + +**User cancel** (``scancel ``):: + + T SIGTERM → all processes (no SIGUSR1) + T+KillWait SIGKILL → anything still alive (KillWait is 30s) + +Implementation +-------------- + +* **How the SIGUSR1 signal is handled:** + + - The Slurm batch script responds to SIGUSR1 by creating a sentinel file + (``$SLURM_LOG_DIR/SIGUSR1_RECEIVED``) on the shared filesystem. + - This callback polls for the presence of this sentinel file at the end of + each training step. + - When detected, it triggers an emergency checkpoint save. + +* **Why the Python process can't receive SIGUSR1 directly:** + + - Pyxis/Enroot containers do not receive SIGUSR1 signals from Slurm (the + signal is sent to the batch shell, not the container). + - Attempted to forward SIGUSR1 with both ``srun`` and + ``scancel --signal`` in batch scripts, proven not working. + +* **Why no SIGTERM handler is needed:** + + - The SIGUSR1 signal is the trigger of preemption and timeout, it + is already able to distinguish them from user cancel. + - Before SIGTERM arrives there are at least 300 s (``GraceTime=300`` for + preemption, ``--signal=B:SIGUSR1@N`` for timeout), which is sufficent + for the poll to detect the sentinel and save a checkpoint. + - SIGTERM and the subsequent SIGKILL are left to terminate the process + naturally after the checkpoint has been saved. + - SIGTERM and SIGUSR1 are logged so we can observe signal delivery into + Pyxis/Enroot containers for debugging purposes. + +To avoid redundant checkpoints, a save is only performed if at least +``save_iter * min_save_fraction`` iterations have elapsed since the last +checkpoint. +""" + +from __future__ import annotations + +import os +import signal +import sys + +import torch + +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.utils import log +from cosmos_framework.utils.callback import Callback + + +class TerminationSignalCheckpoint(Callback): + """Save a checkpoint in response to SIGUSR1 (preemption or timeout). + + Args: + min_save_fraction: Fraction of the regular checkpoint interval (between 0 + and 1) that must have elapsed since the last checkpoint before an + emergency save is allowed. Defaults to 1/3. + """ + + def __init__(self, min_save_fraction: float = 1 / 3): + super().__init__() + self._min_save_fraction = min_save_fraction + self._current_iteration: int = 0 + self._last_checkpoint_iteration: int = 0 + # Captured from on_before_optimizer_step so we can call checkpointer.save(). + self._optimizer: torch.optim.Optimizer | None = None + self._scheduler: torch.optim.lr_scheduler.LRScheduler | None = None + self._grad_scaler: torch.amp.GradScaler | None = None + # Sentinel file created by the batch-shell trap when SIGUSR1 arrives. + # This is the sole detection mechanism because srun/Pyxis does not + # relay SIGUSR1 into the container. + slurm_log_dir = os.environ.get("SLURM_LOG_DIR", "") + self._sigusr1_sentinel = os.path.join(slurm_log_dir, "SIGUSR1_RECEIVED") if slurm_log_dir else "" + + # ------------------------------------------------------------------ + # Lifecycle hooks + # ------------------------------------------------------------------ + + def on_train_start(self, model: ImaginaireModel, iteration: int = 0) -> None: + self._current_iteration = iteration + self._last_checkpoint_iteration = iteration + self._install_termination_signal_handlers() + + def on_before_optimizer_step( + self, + model: ImaginaireModel, + optimizer: torch.optim.Optimizer, + scheduler: torch.optim.lr_scheduler.LRScheduler, + grad_scaler: torch.amp.GradScaler, + iteration: int = 0, + ) -> None: + del model + self._optimizer = optimizer + self._scheduler = scheduler + self._grad_scaler = grad_scaler + + def on_save_checkpoint_success(self, iteration: int = 0, elapsed_time: float = 0) -> None: + self._last_checkpoint_iteration = iteration + + def on_training_step_end( + self, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int = 0, + ) -> None: + self._current_iteration = iteration + + if not self._sigusr1_sentinel or not os.path.exists(self._sigusr1_sentinel): + return + + log.info("[TerminationSignalCheckpoint] Detected SIGUSR1 sentinel file. Will save checkpoint.") + + # Check if the minimum progress has been reached since the last checkpoint. + min_progress = int(self.config.checkpoint.save_iter * self._min_save_fraction) + if (iteration - self._last_checkpoint_iteration) < min_progress: + log.info( + f"[TerminationSignalCheckpoint] Only {iteration - self._last_checkpoint_iteration} iterations " + f"since last checkpoint (threshold {min_progress}). Skipping checkpoint save." + ) + sys.exit(0) + + assert self._optimizer is not None, ( + "[TerminationSignalCheckpoint] Optimizer reference not set — on_before_optimizer_step was never called" + ) + + log.info(f"[TerminationSignalCheckpoint] Saving checkpoint at iteration {iteration}.") + self.trainer.checkpointer.save(model, self._optimizer, self._scheduler, self._grad_scaler, iteration=iteration) + # Async DCP checkpointing queues the write to a background process. + # We must wait for it to finish before exiting. + self.trainer.checkpointer.finalize() + log.info(f"[TerminationSignalCheckpoint] Checkpoint saved at iteration {iteration}.") + sys.exit(0) + + # ------------------------------------------------------------------ + # Termination signal handlers + # ------------------------------------------------------------------ + + def _install_termination_signal_handlers(self) -> None: + signal.signal(signal.SIGTERM, self._log_sigterm) + log.info("[TerminationSignalCheckpoint] Installed SIGTERM handler.") + + def _log_sigterm(self, signum: int, frame: object) -> None: + log.info(f"[TerminationSignalCheckpoint] Received SIGTERM at iteration {self._current_iteration}.") diff --git a/cosmos_framework/callbacks/training_stats.py b/cosmos_framework/callbacks/training_stats.py new file mode 100644 index 0000000..9eb0212 --- /dev/null +++ b/cosmos_framework/callbacks/training_stats.py @@ -0,0 +1,295 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from __future__ import annotations + +import torch +import torch.distributed as dist +import wandb + +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.utils import distributed +from cosmos_framework.utils.callback import Callback +from cosmos_framework.callbacks.wandb_log import _LossRecord +from cosmos_framework.data.vfm.action.domain_utils import EMBODIMENT_TO_DOMAIN_ID + +# Build inverse mapping: domain_id -> embodiment_type. First occurrence wins when multiple embodiment names share the +# same domain id. +DOMAIN_ID_TO_EMBODIMENT: dict[int, str] = {} +for _k, _v in EMBODIMENT_TO_DOMAIN_ID.items(): + DOMAIN_ID_TO_EMBODIMENT.setdefault(_v, _k) + + +class TrainingStatsCallback(Callback): + """Callback for tracking and logging training mode and embodiment statistics to wandb.""" + + def __init__(self, log_freq: int = 100): + super().__init__() + self.log_freq = log_freq + self._mode_counts: dict[str, int] = {} + self._mode_total_count: int = 0 + self._embodiment_counts: dict[str, int] = {} + self._embodiment_total_count: int = 0 + self._per_embodiment_loss: dict[str, _LossRecord] = {} + self._per_embodiment_sub_loss: dict[str, dict[str, _LossRecord]] = {} + + def _accumulate_mode_counts(self, data_batch: dict[str, torch.Tensor]) -> None: + modes = data_batch.get("mode", None) + if modes is None: + return + + if isinstance(modes, str): + modes_list = [modes] + elif isinstance(modes, (list, tuple)): + modes_list = [str(m) for m in modes] + elif isinstance(modes, torch.Tensor): + # Defensive: support cases where mode might be encoded numerically. + modes_list = [str(m) for m in modes.detach().cpu().tolist()] + else: + modes_list = [str(modes)] + + for mode in modes_list: + self._mode_total_count += 1 + self._mode_counts[mode] = self._mode_counts.get(mode, 0) + 1 + + def _accumulate_embodiment_counts(self, data_batch: dict[str, torch.Tensor]) -> None: + domain_ids = data_batch.get("domain_id", None) + if domain_ids is None: + return + + if isinstance(domain_ids, int): + domain_id_list = [domain_ids] + elif isinstance(domain_ids, (list, tuple)): + domain_id_list = [int(d) for d in domain_ids if d is not None] + elif isinstance(domain_ids, torch.Tensor): + # Flatten to handle any shape (scalar, 1D, or 2D with trailing dim) + domain_id_list = [int(d) for d in domain_ids.detach().cpu().flatten().tolist()] + else: + domain_id_list = [int(domain_ids)] + + for domain_id in domain_id_list: + embodiment = DOMAIN_ID_TO_EMBODIMENT.get(domain_id, f"unknown_{domain_id}") + self._embodiment_total_count += 1 + self._embodiment_counts[embodiment] = self._embodiment_counts.get(embodiment, 0) + 1 + + def _gather_global_mode_counts(self) -> tuple[int, dict[str, int]]: + """ + Returns (global_total, global_mode_counts) aggregated across all ranks. + """ + local: dict[str, int] = dict(self._mode_counts) + local["__total__"] = int(self._mode_total_count) + + if dist.is_available() and dist.is_initialized(): + world_size = int(dist.get_world_size()) + gathered: list[dict[str, int] | None] = [None for _ in range(world_size)] + dist.all_gather_object(gathered, local) + else: + gathered = [local] + + global_total = 0 + global_counts: dict[str, int] = {} + for item in gathered: + if not item: + continue + global_total += int(item.get("__total__", 0)) + for k, v in item.items(): + if k == "__total__": + continue + global_counts[k] = global_counts.get(k, 0) + int(v) + return global_total, global_counts + + def _gather_global_embodiment_counts(self) -> tuple[int, dict[str, int]]: + """ + Returns (global_total, global_embodiment_counts) aggregated across all ranks. + """ + local: dict[str, int] = dict(self._embodiment_counts) + local["__total__"] = int(self._embodiment_total_count) + + if dist.is_available() and dist.is_initialized(): + world_size = int(dist.get_world_size()) + gathered: list[dict[str, int] | None] = [None for _ in range(world_size)] + dist.all_gather_object(gathered, local) + else: + gathered = [local] + + global_total = 0 + global_counts: dict[str, int] = {} + for item in gathered: + if not item: + continue + global_total += int(item.get("__total__", 0)) + for k, v in item.items(): + if k == "__total__": + continue + global_counts[k] = global_counts.get(k, 0) + int(v) + return global_total, global_counts + + def _build_mode_log_dict( + self, *, log_prefix: str, global_total: int, global_counts: dict[str, int] + ) -> dict[str, float]: + info: dict[str, float] = {} + + denom = float(global_total) if global_total > 0 else 0.0 + for mode in sorted(global_counts.keys()): + count = float(global_counts.get(mode, 0)) + pct = (100.0 * count / denom) if denom > 0 else 0.0 + info[f"{log_prefix}_stats_mode/{mode}"] = pct + + return info + + def _build_embodiment_log_dict( + self, *, log_prefix: str, global_total: int, global_counts: dict[str, int] + ) -> dict[str, float]: + info: dict[str, float] = {} + + denom = float(global_total) if global_total > 0 else 0.0 + for embodiment in sorted(global_counts.keys()): + count = float(global_counts.get(embodiment, 0)) + pct = (100.0 * count / denom) if denom > 0 else 0.0 + info[f"{log_prefix}_stats_embodiment/{embodiment}"] = pct + + return info + + def _get_batch_embodiment(self, data_batch: dict[str, torch.Tensor]) -> str | None: + """Extract the embodiment name from the first non-None sample's domain_id.""" + domain_ids = data_batch.get("domain_id", None) + if domain_ids is None: + return None + if isinstance(domain_ids, torch.Tensor): + if domain_ids.numel() == 0: + return None + domain_id = int(domain_ids.flatten()[0].item()) + elif isinstance(domain_ids, (list, tuple)): + first = next((d for d in domain_ids if d is not None), None) + if first is None: + return None + domain_id = int(first) + else: + domain_id = int(domain_ids) + return DOMAIN_ID_TO_EMBODIMENT.get(domain_id, f"unknown_{domain_id}") + + def _accumulate_per_embodiment_loss( + self, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + ) -> None: + embodiment = self._get_batch_embodiment(data_batch) + if embodiment is None: + return + + if embodiment not in self._per_embodiment_loss: + self._per_embodiment_loss[embodiment] = _LossRecord() + self._per_embodiment_loss[embodiment].loss += loss.detach().float() + self._per_embodiment_loss[embodiment].iter_count += 1 + + if embodiment not in self._per_embodiment_sub_loss: + self._per_embodiment_sub_loss[embodiment] = {} + for key in output_batch: + if "loss" in key and "per_instance" not in key: + if key not in self._per_embodiment_sub_loss[embodiment]: + self._per_embodiment_sub_loss[embodiment][key] = _LossRecord() + self._per_embodiment_sub_loss[embodiment][key].loss += output_batch[key].detach().float() + self._per_embodiment_sub_loss[embodiment][key].iter_count += 1 + + def _compute_per_embodiment_loss_stats(self, log_prefix: str) -> dict[str, float]: + """Compute per-embodiment loss averages across all ranks. + + All ranks must call this method (contains collective operations). + Returns the log dict (only meaningful on rank 0). + """ + dist_available = dist.is_available() and dist.is_initialized() + world_size = int(dist.get_world_size()) if dist_available else 1 + + # Step 1: gather union of embodiment names across ranks + local_embodiments = sorted(self._per_embodiment_loss.keys()) + if dist_available: + all_embodiments: list[list[str] | None] = [None for _ in range(world_size)] + dist.all_gather_object(all_embodiments, local_embodiments) + else: + all_embodiments = [local_embodiments] + union_embodiments = sorted({e for el in all_embodiments for e in el}) + + # Step 2: gather union of sub-loss keys across ranks + local_sub_keys = sorted({k for d in self._per_embodiment_sub_loss.values() for k in d}) + if dist_available: + all_sub_keys: list[list[str] | None] = [None for _ in range(world_size)] + dist.all_gather_object(all_sub_keys, local_sub_keys) + else: + all_sub_keys = [local_sub_keys] + union_sub_keys = sorted({k for kl in all_sub_keys for k in kl}) + + # Step 3: insert NaN dummy _LossRecord for missing embodiment/key combos + for emb in union_embodiments: + if emb not in self._per_embodiment_loss: + dummy = _LossRecord() + dummy.loss += torch.tensor([float("nan")], device="cuda") + dummy.iter_count += 1 + self._per_embodiment_loss[emb] = dummy + if emb not in self._per_embodiment_sub_loss: + self._per_embodiment_sub_loss[emb] = {} + for key in union_sub_keys: + if key not in self._per_embodiment_sub_loss[emb]: + dummy = _LossRecord() + dummy.loss += torch.tensor([float("nan")], device="cuda") + dummy.iter_count += 1 + self._per_embodiment_sub_loss[emb][key] = dummy + + # Step 4: compute distributed averages (all ranks participate in all_reduce) + log_dict: dict[str, float] = {} + for emb in union_embodiments: + avg, valid = self._per_embodiment_loss[emb].get_stat(return_valid_mask_sum=True) + if valid > 0: + log_dict[f"{log_prefix}_stats_loss/{emb}"] = avg + + for emb in union_embodiments: + for key in union_sub_keys: + avg, valid = self._per_embodiment_sub_loss[emb][key].get_stat(return_valid_mask_sum=True) + if valid > 0: + log_dict[f"{log_prefix}_stats_loss_detail/{emb}_{key}"] = avg + + # Step 5: reset accumulators + self._per_embodiment_loss = {} + self._per_embodiment_sub_loss = {} + + return log_dict + + @torch.no_grad() + def on_training_step_end( + self, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int = 0, + ) -> None: + self._accumulate_mode_counts(data_batch) + self._accumulate_embodiment_counts(data_batch) + self._accumulate_per_embodiment_loss(data_batch, output_batch, loss) + + if iteration % self.log_freq != 0: + return + + # All ranks must participate in collective operations below. + mode_total, mode_counts = self._gather_global_mode_counts() + embodiment_total, embodiment_counts = self._gather_global_embodiment_counts() + per_embodiment_loss_dict = self._compute_per_embodiment_loss_stats(log_prefix="train") + + if not distributed.is_rank0(): + return + + if wandb.run is None: + return + + log_dict: dict[str, float] = {} + log_dict.update( + self._build_mode_log_dict(log_prefix="train", global_total=mode_total, global_counts=mode_counts) + ) + log_dict.update( + self._build_embodiment_log_dict( + log_prefix="train", global_total=embodiment_total, global_counts=embodiment_counts + ) + ) + log_dict.update(per_embodiment_loss_dict) + + wandb.log({k: float(v) for k, v in log_dict.items()}, step=iteration) diff --git a/cosmos_framework/callbacks/wandb_log.py b/cosmos_framework/callbacks/wandb_log.py new file mode 100644 index 0000000..04d0799 --- /dev/null +++ b/cosmos_framework/callbacks/wandb_log.py @@ -0,0 +1,230 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Tuple + +import torch +import torch.distributed as dist +import torch.utils.data +import wandb + +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.utils import distributed, log +from cosmos_framework.utils.callback import Callback +from cosmos_framework.utils.easy_io import easy_io + + +@dataclass +class _LossRecord: + loss: torch.Tensor | float = 0 + iter_count: int = 0 + name: str | None = None + + def reset(self) -> None: + self.loss = 0 + self.iter_count = 0 + + def get_stat(self, return_valid_mask_sum: bool = False) -> Tuple[float, float]: + if self.iter_count == 0: + self.loss = torch.tensor([float("nan")], device="cuda") + self.iter_count = 1 + self.loss = self.loss.mean() + msg_str = f"{self.name}: sum_loss={self.loss.item()}/iter_count={self.iter_count}=" + avg_loss_tensor = self.loss / self.iter_count + # Create a mask (1 if valid, 0 if NaN or Inf) + valid_mask = torch.tensor([torch.isfinite(avg_loss_tensor).float()], device="cuda") + msg_str += f"avg_loss={avg_loss_tensor.item()}, valid_mask={valid_mask.item()}, " + + # Replace NaN/Inf with 0 to avoid affecting sum + avg_loss_tensor = torch.where( + torch.isfinite(avg_loss_tensor), avg_loss_tensor, torch.tensor([0.0], device="cuda") + ) + + # Reduce across all ranks + dist.all_reduce(avg_loss_tensor, op=dist.ReduceOp.SUM) # Sum of valid losses + dist.all_reduce(valid_mask, op=dist.ReduceOp.SUM) # Count of valid losses + msg_str += f" | all_reduce: avg_loss={avg_loss_tensor.item()}, valid_mask={valid_mask.item()}" + # Compute final average, avoiding division by zero + if valid_mask.item() > 0: + final_avg_loss = (avg_loss_tensor / valid_mask).item() + valid_mask_sum = valid_mask.item() + else: + final_avg_loss = 0.0 # Default to zero if all values were invalid + valid_mask_sum = 0 + + avg_loss = final_avg_loss + msg_str += f" | final: avg_loss={final_avg_loss}" + if self.name is not None: + log.debug(msg_str, rank0_only=False) + self.reset() + if return_valid_mask_sum: + return avg_loss, valid_mask_sum + else: + return avg_loss + + +class WandbCallback(Callback): + def __init__( + self, + logging_iter_multipler: int = 1, + save_logging_iter_multipler: int = 1, + save_s3: bool = False, + ) -> None: + super().__init__() + self.final_loss_log = _LossRecord() + self.final_loss_log_per_dataset = {} + self.final_all_loss_log = {} + self.logging_iter_multipler = logging_iter_multipler + self.save_logging_iter_multipler = save_logging_iter_multipler + assert self.logging_iter_multipler > 0, "logging_iter_multipler should be greater than 0" + self.save_s3 = save_s3 + self.wandb_extra_tag = f"@{logging_iter_multipler}" if logging_iter_multipler > 1 else "" + self.name = "wandb_loss_log" + self.wandb_extra_tag + self.unstable_count = torch.zeros(1, device="cuda") + + def on_training_step_end( + self, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int = 0, + ) -> None: + if torch.isnan(loss) or torch.isinf(loss): + log.critical( + f"Unstable loss {loss} at iteration {iteration}", + rank0_only=False, + ) + self.unstable_count += 1 + + dataset_name = data_batch.get("dataset_name", "default") + + # Handle case where dataset_name gets batched into a list + if isinstance(dataset_name, list): + # For reasoner, dataset_name will be a list of different datasets + # For generator, dataset_name is a list of size larger than one, + # assume they are all the same + # Dedup using set to extract identical dataset names + dataset_name = list(set(dataset_name)) + + if dataset_name == "default" and "__url__" in data_batch: + # try to get the name from url + dataset_name = ["/".join(data_batch["__url__"][0].split("/")[:-1])] + + for single_dataset_name in dataset_name: + if single_dataset_name not in self.final_loss_log_per_dataset: + self.final_loss_log_per_dataset[single_dataset_name] = _LossRecord() + self.final_loss_log_per_dataset[single_dataset_name].name = single_dataset_name + self.final_loss_log_per_dataset[single_dataset_name].loss += loss.detach().float() + self.final_loss_log_per_dataset[single_dataset_name].iter_count += 1 + + # VLM: per-sequence loss normalization using token counts when available + if "avg_num_assistant_tokens" in output_batch: + per_seq_loss = ( + loss + * output_batch["avg_num_assistant_tokens"] + * output_batch["batch_size_local"] + / output_batch["current_num_assistant_tokens"] + ) + per_seq_key = f"per_seq/{dataset_name}" + if per_seq_key not in self.final_loss_log_per_dataset: + self.final_loss_log_per_dataset[per_seq_key] = _LossRecord() + self.final_loss_log_per_dataset[per_seq_key].name = per_seq_key + self.final_loss_log_per_dataset[per_seq_key].loss += per_seq_loss + self.final_loss_log_per_dataset[per_seq_key].iter_count += 1 + + self.final_loss_log.loss += loss.detach().float() + self.final_loss_log.iter_count += 1 + + for key in output_batch.keys(): + # Curve can be plotted only on aggregated loss, not per-instance loss + if "loss" in key and "per_instance" not in key: + if key not in self.final_all_loss_log: + self.final_all_loss_log[key] = _LossRecord() + self.final_all_loss_log[key].loss += output_batch[key].detach().float() + self.final_all_loss_log[key].iter_count += 1 + + if iteration % (self.config.trainer.logging_iter * self.logging_iter_multipler) == 0: + avg_final_loss = self.final_loss_log.get_stat() + + avg_final_all_loss = {} + for key in self.final_all_loss_log.keys(): + avg_final_all_loss[key] = self.final_all_loss_log[key].get_stat() + + # Step 1: Gather all dataset names across ranks + local_dataset_names = list(self.final_loss_log_per_dataset.keys()) + all_dataset_names = [None for _ in range(dist.get_world_size())] + dist.all_gather_object(all_dataset_names, local_dataset_names) + + # Step 2: Create the union of all dataset names + union_dataset_names = set() + for names in all_dataset_names: + union_dataset_names.update(names) + # Step 3: For any missing dataset name, add dummy _LossRecord with NaN loss + union_dataset_names = sorted(list(union_dataset_names)) # This is very important! + for dataset_name in union_dataset_names: + if dataset_name not in self.final_loss_log_per_dataset: + dummy = _LossRecord() + dummy.loss += torch.tensor([float("nan")], device="cuda") # Will be masked out + dummy.iter_count += 1 + self.final_loss_log_per_dataset[dataset_name] = dummy + + avg_final_loss_per_dataset = {} + for dataset_name in union_dataset_names: + avg_loss, valid_mask_sum = self.final_loss_log_per_dataset[dataset_name].get_stat( + return_valid_mask_sum=True + ) + if valid_mask_sum > 0: + avg_final_loss_per_dataset[dataset_name] = avg_loss + + dist.all_reduce(self.unstable_count, op=dist.ReduceOp.SUM) + + if distributed.is_rank0() and wandb.run is not None: + info = {} + info.update( + { + f"train{self.wandb_extra_tag}/loss": avg_final_loss, + f"train{self.wandb_extra_tag}/unstable_count": self.unstable_count.item(), + "iteration": iteration, + } + ) + for key, loss in avg_final_all_loss.items(): + info.update( + { + f"train{self.wandb_extra_tag}_detail/{key}": loss, + } + ) + for dataset_name, loss in avg_final_loss_per_dataset.items(): + tag = "" + if "per_seq" in dataset_name: + tag = "_per_seq" + dataset_name = dataset_name.replace("per_seq/", "") + info.update( + { + f"train{self.wandb_extra_tag}_per_data{tag}/{dataset_name}": loss, + } + ) + if self.save_s3: + if ( + iteration + % ( + self.config.trainer.logging_iter + * self.logging_iter_multipler + * self.save_logging_iter_multipler + ) + == 0 + ): + easy_io.dump( + info, + f"s3://rundir/{self.name}/Train_Iter{iteration:09d}.json", + ) + + if wandb: + wandb.log(info, step=iteration) + + # reset unstable count + self.unstable_count.zero_() + self.final_loss_log_per_dataset = {} diff --git a/cosmos_framework/callbacks/wandb_log_eval.py b/cosmos_framework/callbacks/wandb_log_eval.py new file mode 100644 index 0000000..ac6911f --- /dev/null +++ b/cosmos_framework/callbacks/wandb_log_eval.py @@ -0,0 +1,141 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Tuple + +import torch +import torch.distributed as dist +import torch.utils.data +import wandb + +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.utils import distributed, log +from cosmos_framework.utils.callback import Callback +from cosmos_framework.utils.easy_io import easy_io + + +@dataclass +class _LossRecord: + loss: float = 0 + iter_count: int = 0 + + def reset(self) -> None: + self.loss = 0 + self.iter_count = 0 + + def get_stat(self) -> Tuple[float, float]: + if self.iter_count > 0: + avg_loss_tensor = self.loss / self.iter_count + # Create a mask (1 if valid, 0 if NaN or Inf) + valid_mask = torch.tensor([torch.isfinite(avg_loss_tensor).float()], device="cuda") + + # Replace NaN/Inf with 0 to avoid affecting sum + avg_loss_tensor = torch.where( + torch.isfinite(avg_loss_tensor), avg_loss_tensor, torch.tensor([0.0], device="cuda") + ) + + # Reduce across all ranks + dist.all_reduce(avg_loss_tensor, op=dist.ReduceOp.SUM) # Sum of valid losses + dist.all_reduce(valid_mask, op=dist.ReduceOp.SUM) # Count of valid losses + + # Compute final average, avoiding division by zero + if valid_mask.item() > 0: + final_avg_loss = (avg_loss_tensor / valid_mask).item() + else: + final_avg_loss = 0.0 # Default to zero if all values were invalid + + avg_loss = final_avg_loss + else: + avg_loss = 0 + self.reset() + return avg_loss + + +class WandbCallback(Callback): + def __init__( + self, + save_s3: bool = False, + ) -> None: + super().__init__() + self.final_loss_log = _LossRecord() + self.final_loss_log_per_dataset = {} + + self.save_s3 = save_s3 + self.wandb_extra_tag = "" + self.name = "wandb_loss_val_log" + self.unstable_count = torch.zeros(1, device="cuda") + self.url_key_list = [] + + def on_validation_step_end( + self, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int = 0, + ) -> None: + if torch.isnan(loss) or torch.isinf(loss): + log.critical( + f"Unstable val loss {loss} at iteration {iteration}", + rank0_only=False, + ) + self.unstable_count += 1 + + dataset_name = data_batch.get("dataset_name", "default") + + # Handle case where dataset_name gets batched into a list + if isinstance(dataset_name, list): + + assert len(dataset_name) == 1, "dataset_name should be a list of 1" + dataset_name = dataset_name[0] + + if dataset_name not in self.final_loss_log_per_dataset: + self.final_loss_log_per_dataset[dataset_name] = _LossRecord() + + self.final_loss_log_per_dataset[dataset_name].loss += loss.detach().float() + self.final_loss_log_per_dataset[dataset_name].iter_count += 1 + self.final_loss_log.loss += loss.detach().float() + self.final_loss_log.iter_count += 1 + + self.url_key_list.append(f"{data_batch.get('__url__', [''])[0]}, {data_batch.get('__key__', [''])[0]}") + + def on_validation_end(self, model: ImaginaireModel, iteration: int = 0) -> None: + avg_final_loss = self.final_loss_log.get_stat() + + log.info(f"avg_final_loss: {avg_final_loss}") + dist.all_reduce(self.unstable_count, op=dist.ReduceOp.SUM) + # gather url and key list from all ranks + url_key_list = [None] * dist.get_world_size() + dist.all_gather_object(url_key_list, self.url_key_list) + url_key_list = [item for sublist in url_key_list for item in sublist] + + unique_url_key_list = list(set(url_key_list)) + if distributed.is_rank0(): + info = {} + log.info( + f"[val] number of unique url and key: {len(unique_url_key_list)} / {len(url_key_list)}; avg_final_loss: {avg_final_loss}" + ) + info.update( + { + f"val{self.wandb_extra_tag}/loss": avg_final_loss, + f"val{self.wandb_extra_tag}/unstable_count": self.unstable_count.item(), + "iteration": iteration, + f"val{self.wandb_extra_tag}/num_unique_url_key": len(unique_url_key_list), + f"val{self.wandb_extra_tag}/total_url_key": len(url_key_list), + } + ) + if self.save_s3: + easy_io.dump( + info, + f"s3://rundir/{self.name}/Val_Iter{iteration:09d}.json", + ) + + if wandb.run is not None: + wandb.log(info, step=iteration) + + # reset unstable count + self.unstable_count.zero_() + self.url_key_list = [] diff --git a/cosmos_framework/callbacks/wandb_log_simple.py b/cosmos_framework/callbacks/wandb_log_simple.py new file mode 100644 index 0000000..6087a33 --- /dev/null +++ b/cosmos_framework/callbacks/wandb_log_simple.py @@ -0,0 +1,155 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Tuple + +import torch +import torch.distributed as dist +import torch.utils.data +import wandb + +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.utils import distributed, log +from cosmos_framework.utils.callback import Callback +from cosmos_framework.utils.easy_io import easy_io + + +@dataclass +class _LossRecord: + loss: float = 0 + iter_count: int = 0 + name: str = None + + def reset(self) -> None: + self.loss = 0 + self.iter_count = 0 + + def get_stat(self, return_valid_mask_sum: bool = False) -> Tuple[float, float]: + if self.iter_count == 0: + self.loss = torch.tensor([float("nan")], device="cuda") # [1] + self.iter_count = 1 + msg_str = f"{self.name}: sum_loss={self.loss.item()}/iter_count={self.iter_count}=" + avg_loss_tensor = self.loss / self.iter_count + # Create a mask (1 if valid, 0 if NaN or Inf) + valid_mask = torch.tensor([torch.isfinite(avg_loss_tensor).float()], device="cuda") # [1] + msg_str += f"avg_loss={avg_loss_tensor.item()}, valid_mask={valid_mask.item()}, " + + # Replace NaN/Inf with 0 to avoid affecting sum + avg_loss_tensor = torch.where( + torch.isfinite(avg_loss_tensor), + avg_loss_tensor, + torch.tensor([0.0], device="cuda"), # [1] + ) + + # Reduce across all ranks + dist.all_reduce(avg_loss_tensor, op=dist.ReduceOp.SUM) # Sum of valid losses + dist.all_reduce(valid_mask, op=dist.ReduceOp.SUM) # Count of valid losses + msg_str += f" | all_reduce: avg_loss={avg_loss_tensor.item()}, valid_mask={valid_mask.item()}" + # Compute final average, avoiding division by zero + if valid_mask.item() > 0: + final_avg_loss = (avg_loss_tensor / valid_mask).item() + valid_mask_sum = valid_mask.item() + else: + final_avg_loss = 0.0 # Default to zero if all values were invalid + valid_mask_sum = 0 + + avg_loss = final_avg_loss + msg_str += f" | final: avg_loss={final_avg_loss}" + if self.name is not None: + log.debug(msg_str, rank0_only=False) + self.reset() + if return_valid_mask_sum: + return avg_loss, valid_mask_sum + else: + return avg_loss + + +class WandbCallback(Callback): + def __init__( + self, + logging_iter_multipler: int = 1, + save_logging_iter_multipler: int = 1, + save_s3: bool = False, + ) -> None: + super().__init__() + self.final_loss_log = _LossRecord() + self.final_all_loss_log = {} + self.logging_iter_multipler = logging_iter_multipler + self.save_logging_iter_multipler = save_logging_iter_multipler + assert self.logging_iter_multipler > 0, "logging_iter_multipler should be greater than 0" + self.save_s3 = save_s3 + self.wandb_extra_tag = f"@{logging_iter_multipler}" if logging_iter_multipler > 1 else "" + self.name = "wandb_loss_log" + self.wandb_extra_tag + self.unstable_count = torch.zeros(1, device="cuda") # [1] + + def on_training_step_end( + self, + model: ImaginaireModel, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int = 0, + ) -> None: + if torch.isnan(loss) or torch.isinf(loss): + log.critical( + f"Unstable loss {loss} at iteration {iteration}", + rank0_only=False, + ) + self.unstable_count += 1 + + self.final_loss_log.loss += loss.detach().float() + self.final_loss_log.iter_count += 1 + + for key in output_batch.keys(): + if "loss" in key: + if key not in self.final_all_loss_log: + self.final_all_loss_log[key] = _LossRecord() + self.final_all_loss_log[key].loss += output_batch[key].detach().float() + self.final_all_loss_log[key].iter_count += 1 + + if iteration % (self.config.trainer.logging_iter * self.logging_iter_multipler) == 0: + avg_final_loss = self.final_loss_log.get_stat() + + avg_final_all_loss = {} + for key in self.final_all_loss_log.keys(): + avg_final_all_loss[key] = self.final_all_loss_log[key].get_stat() + + dist.all_reduce(self.unstable_count, op=dist.ReduceOp.SUM) + + if distributed.is_rank0() and wandb.run is not None: + info = {} + info.update( + { + f"train{self.wandb_extra_tag}/loss": avg_final_loss, + f"train{self.wandb_extra_tag}/unstable_count": self.unstable_count.item(), + "iteration": iteration, + } + ) + for key, loss in avg_final_all_loss.items(): + info.update( + { + f"train{self.wandb_extra_tag}_detail/{key}": loss, + } + ) + if self.save_s3: + if ( + iteration + % ( + self.config.trainer.logging_iter + * self.logging_iter_multipler + * self.save_logging_iter_multipler + ) + == 0 + ): + easy_io.dump( + info, + f"s3://rundir/{self.name}/Train_Iter{iteration:09d}.json", + ) + + if wandb: + wandb.log(info, step=iteration) + # reset unstable count + self.unstable_count.zero_() diff --git a/cosmos_framework/callbacks/wandb_vis.py b/cosmos_framework/callbacks/wandb_vis.py new file mode 100644 index 0000000..2b5b67b --- /dev/null +++ b/cosmos_framework/callbacks/wandb_vis.py @@ -0,0 +1,94 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from __future__ import annotations + +import html +import json +import tempfile +from typing import Literal + +import torch +import wandb +from einops import rearrange + +from cosmos_framework.utils.config import JobConfig +from cosmos_framework.utils import callback, distributed +from cosmos_framework.utils.easy_io import easy_io + + +class VisualizationLoggingCallback(callback.WandBCallback): + def __init__( + self, + every_n: int = 1, + input_normalization: str | None = None, + job: JobConfig | None = None, + config: callback.Config | None = None, + trainer: callback.ImaginaireTrainer | None = None, + ): + super().__init__(config, trainer) + self.every_n = every_n + self.input_normalization = input_normalization + self.job = job + + def on_training_step_end( + self, + model, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int = 0, + ) -> None: + if iteration % self.every_n == 0 and distributed.is_rank0() and wandb.run is not None: + # log images to wandb for rank0 + self.log_videos(log_type="train", data=data_batch, output=output_batch, iteration=iteration) + + def on_validation_step_end( + self, + model, + data_batch: dict[str, torch.Tensor], + output_batch: dict[str, torch.Tensor], + loss: torch.Tensor, + iteration: int = 0, + ) -> None: # Collect the validation batch and aggregate the overall loss. + super().on_validation_step_end(model, data_batch, output_batch, loss, iteration) + if iteration % self.every_n == 0 and distributed.is_rank0() and wandb.run is not None: + # log images to wandb for rank0 + self.log_videos(log_type="val", data=data_batch, output=output_batch, iteration=iteration) + + @torch.no_grad() + def log_videos( + self, + log_type: Literal["train", "val"], + data: dict[str, torch.Tensor], + output: dict[str, torch.Tensor], + iteration: int = 0, + ): + if "raw_image" in data: + video = data["raw_image"][0].cpu() # [3,T,H,W], range [0, 255], uint8 + elif "raw_video" in data: + video = data["raw_video"][0].cpu() # [3,T,H,W], range [0, 255], uint8 + video = video.permute(1, 0, 2, 3) # [T,3,H,W] + video = video.numpy() + wandb_video = rearrange(video, "t c h w -> t h w c") # [T,H,W,3] + # create temp file + temp_file = tempfile.NamedTemporaryFile(delete=False, suffix=".mp4") + temp_file_path = temp_file.name + easy_io.dump(wandb_video, temp_file_path, file_format="mp4", format="mp4", fps=10, quality=5) + wandb.log({f"{log_type}/video": wandb.Video(temp_file_path)}, step=iteration) + + # Convert dialog string to JSON and format it for HTML display + try: + dialog_json = json.loads(data["dialog_str"][0], indent=4) + formatted_html = f""" + 
{dialog_json}
+ """ + wandb.log({f"{log_type}/prompt": wandb.Html(formatted_html)}, step=iteration) + except Exception as e: + # Fallback to original format if JSON conversion fails + # Escape HTML tags in the dialog string to display them properly + escaped_dialog = html.escape(data["dialog_str"][0]) + wandb.log( + {f"{log_type}/prompt": wandb.Html(f"
{escaped_dialog}
")}, + step=iteration, + ) diff --git a/cosmos_framework/checkpoint/__init__.py b/cosmos_framework/checkpoint/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/checkpoint/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/checkpoint/base.py b/cosmos_framework/checkpoint/base.py new file mode 100644 index 0000000..0c11601 --- /dev/null +++ b/cosmos_framework/checkpoint/base.py @@ -0,0 +1,177 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import os +from abc import ABC, abstractmethod +from typing import Optional + +import torch + +from cosmos_framework.utils.config import CheckpointConfig, JobConfig +from cosmos_framework.utils.flags import INTERNAL +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.utils import callback +from cosmos_framework.utils.easy_io import easy_io + + +class AbstractCheckpointer(ABC): + """The checkpointer class. Supports checkpoint saving/loading to both local disk or object store.""" + + def __init__( + self, + config_checkpoint: CheckpointConfig, + config_job: JobConfig, + callbacks: Optional[callback.CallBackGroup] = None, + ): + """Constructor of the checkpointer. + + Args: + config_checkpoint (CheckpointConfig): The config object for the checkpointer. + """ + self.config_checkpoint = config_checkpoint + # Set the callback functions. + self.callbacks = callbacks + self.save_to_object_store = config_checkpoint.save_to_object_store.enabled + self.load_from_object_store = config_checkpoint.load_from_object_store.enabled + + # Set checkpoint directories for local and object store paths + self._local_dirname = os.path.join(config_job.path_local, "checkpoints") + self._object_store_dirname = os.path.join(config_job.path, "checkpoints") + + self.strict_resume = config_checkpoint.strict_resume + load_path = config_checkpoint.load_path or None + if not INTERNAL: + from cosmos_framework.utils.checkpoint_db import download_checkpoint_v2 + + if load_path: + load_path = download_checkpoint_v2(load_path) + self.load_path = load_path + self.load_training_state = config_checkpoint.load_training_state + self.only_load_scheduler_state = config_checkpoint.only_load_scheduler_state + self.save_thread = None + self.verbose = config_checkpoint.verbose + self.keys_not_to_resume = config_checkpoint.keys_not_to_resume + self.keys_to_skip_loading = getattr(config_checkpoint, "keys_to_skip_loading", []) + self.broadcast_via_filesystem = config_checkpoint.broadcast_via_filesystem + # Create the object store client interface. + if config_checkpoint.load_from_object_store.enabled: + self.load_s3_backend_key = "_ckpt_s3_loader" + easy_io.set_s3_backend( + key="_ckpt_s3_loader", + backend_args={ + "backend": "s3", + "path_mapping": { + "s3://ckpt/": f"s3://{config_checkpoint.load_from_object_store.bucket}/", + }, + "s3_credential_path": config_checkpoint.load_from_object_store.credentials, + }, + ) + else: + self.load_s3_backend_key = None + + if config_checkpoint.save_to_object_store.enabled: + self.save_s3_backend_key = "_ckpt_s3_saver" + easy_io.set_s3_backend( + key="_ckpt_s3_saver", + backend_args={ + "backend": "s3", + "path_mapping": { + "s3://ckpt/": f"s3://{config_checkpoint.save_to_object_store.bucket}/", + }, + "s3_credential_path": config_checkpoint.save_to_object_store.credentials, + }, + ) + else: + self.save_s3_backend_key = None + + @abstractmethod + def save( + self, + model: ImaginaireModel, + optimizer: torch.optim.Optimizer, + scheduler: torch.optim.lr_scheduler.LRScheduler, + grad_scaler: torch.amp.GradScaler, + iteration: int, + ) -> None: + pass + + @abstractmethod + def load( + self, + model: ImaginaireModel, + optimizer: Optional[torch.optim.Optimizer] = None, + scheduler: Optional[torch.optim.lr_scheduler.LRScheduler] = None, + grad_scaler: Optional[torch.amp.GradScaler] = None, + ) -> int: + pass + + @property + def save_bucket(self): + """Get the bucket name for saving checkpoints.""" + return self.config_checkpoint.save_to_object_store.bucket if self.save_to_object_store else None + + @property + def load_bucket(self): + """Get the bucket name for loading checkpoints.""" + return self.config_checkpoint.load_from_object_store.bucket if self.load_from_object_store else None + + @property + def save_dirname(self): + return ( + f"s3://{self.save_bucket}/{self._object_store_dirname}" + if self.save_to_object_store + else self._local_dirname + ) + + @property + def load_dirname(self): + return ( + f"s3://{self.load_bucket}/{self._object_store_dirname}" + if self.load_from_object_store + else self._local_dirname + ) + + def finalize(self) -> None: + """Finalize the checkpointer.""" + if self.save_thread: + self.save_thread.join() + + def _read_latest_checkpoint_file(self) -> str | None: + """Get the file name of the latest saved checkpoint. If it doesn't exist, return None. + + Returns: + checkpoint_file (str | None): file name of the latest saved checkpoint. + """ + checkpoint_file = None + checkpoint_path = os.path.join(self.load_dirname, "latest_checkpoint.txt") + if easy_io.exists(f"{checkpoint_path}", backend_key=self.load_s3_backend_key): + checkpoint_file = easy_io.load(f"{checkpoint_path}", backend_key=self.load_s3_backend_key).strip() + + return checkpoint_file + + def has_resumable_checkpoint(self) -> bool: + """True iff a ``latest_checkpoint.txt`` exists in the load directory.""" + return self._read_latest_checkpoint_file() is not None + + def _write_latest_checkpoint_file(self, checkpoint_file: str) -> None: + """Track the file name of the latest saved checkpoint. + + Args: + checkpoint_file (str): file name of the latest saved checkpoint. + """ + content = f"{checkpoint_file}\n" + checkpoint_path = os.path.join(self.save_dirname, "latest_checkpoint.txt") + easy_io.dump( + content, + checkpoint_path, + backend_key=self.save_s3_backend_key, + ) + + def _check_checkpoint_exists(self, checkpoint_path: str) -> None: + """If the file checkpoint_path does not exist, raise an error. + + Args: + checkpoint_path (str): full path to the checkpoint. + """ + if not easy_io.exists(f"{checkpoint_path}", backend_key=self.load_s3_backend_key): + raise FileNotFoundError(f"File not found (object store): {checkpoint_path}") diff --git a/cosmos_framework/checkpoint/dcp.py b/cosmos_framework/checkpoint/dcp.py new file mode 100644 index 0000000..9fa64bb --- /dev/null +++ b/cosmos_framework/checkpoint/dcp.py @@ -0,0 +1,837 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +""" +Distributed checkpoint (DCP) directory structure and storage backends. + +The checkpointer saves model state in a sharded format across multiple processes: + +self.save_dirname/ +├── iter_000000005/ # Checkpoint at iteration 5 +│ ├── model/ # Model state shards +│ │ ├── __0_0.distcp # Shard 0 from rank 0 +│ │ └── __1_0.distcp # Shard 1 from rank 1 +│ ├── optim/ # Optimizer state shards +│ │ ├── __0_0.distcp # Shard 0 from rank 0 +│ │ └── __1_0.distcp # Shard 1 from rank 1 +│ ├── scheduler/ # Learning rate scheduler state +│ │ ├── __0_0.distcp # Shard 0 from rank 0 +│ │ └── __1_0.distcp # Shard 1 from rank 1 +│ └── trainer/ # Additional training state +│ ├── __0_0.distcp # Shard 0 from rank 0 +│ └── __1_0.distcp # Shard 1 from rank 1 +│ └── dataloader/ # Optional per-rank dataloader state +│ ├── rank_0.pkl +│ └── rank_1.pkl +└── latest_checkpoint.txt # Points to most recent checkpoint folder, e.g. iter_000000005 + +Storage backends: +- Local filesystem: + self.save_dirname = "{config_job.path_local}/checkpoints" + +- S3 object store: + self.save_dirname = "s3://{bucket}/{config_job.path}/checkpoints" + where bucket = self.config_checkpoint.save_to_object_store.bucket + +The sharded format enables efficient distributed saving/loading by: +1. Parallelizing I/O across processes +2. Reducing memory usage per process +3. Supporting both local and cloud storage backends +""" + +import enum +import multiprocessing +import os +import re +import time +from multiprocessing import get_context +from typing import Any, Dict, List, Optional, Protocol, Tuple, Union, runtime_checkable + +import torch +import torch.distributed as dist +import torch.distributed.checkpoint as dcp +from torch import nn +from torch.distributed.checkpoint.filesystem import FileSystemReader, FileSystemWriter +from torch.distributed.checkpoint.metadata import ( + STATE_DICT_TYPE, + Metadata, + StorageMeta, +) +from torch.distributed.checkpoint.state_dict import ( + StateDictOptions, + get_model_state_dict, + set_model_state_dict, +) +from torch.distributed.checkpoint.stateful import Stateful + +from cosmos_framework.checkpoint.base import AbstractCheckpointer +from cosmos_framework.checkpoint.s3_filesystem import S3StorageReader, S3StorageWriter +from cosmos_framework.utils.config import CheckpointConfig, JobConfig +from cosmos_framework.model._base import ImaginaireModel +from cosmos_framework.utils import callback, distributed, log, misc +from cosmos_framework.utils.easy_io import easy_io +from cosmos_framework.utils.vfm.rand_state import get_rand_state_dict, set_rand_state_dict + + +class ModelWrapper(Stateful): + """ + Wrapper for model state dict handling. Strips away the _orig_mod. prefix + among other things from the state dict keys. + """ + + def __init__(self, model: nn.Module) -> None: + self.model = model + + def state_dict(self) -> dict[str, Any]: + return get_model_state_dict(self.model) + + def load_state_dict(self, state_dict: dict[str, Any]) -> None: + set_model_state_dict( + self.model, + model_state_dict=state_dict, + options=StateDictOptions(strict=True), + ) + + +@runtime_checkable +class _DataloaderStateHandler(Protocol): + """Structural contract for callbacks that participate in dataloader-state checkpointing.""" + + checkpoint_component: str + + def has_checkpoint_state(self) -> bool: ... + def state_dict(self) -> dict[Any, Any]: ... + def load_state_dict(self, state_dict: dict[Any, Any]) -> None: ... + + +class _DataloaderWrapper: + """Adapter that surfaces a dataloader-state callback's checkpoint API. + + Walks the registered callbacks at construction time and binds to the + first callback that: + + 1. Declares ``checkpoint_component == "dataloader"``, AND + 2. Returns ``True`` from ``has_checkpoint_state()``. + + The bound callback's ``state_dict`` / ``load_state_dict`` methods are + re-exposed via :meth:`state_dict` / :meth:`load_state_dict`. Callers + must gate those on :meth:`has_state` — invoking them when nothing was + bound raises :class:`RuntimeError`. + + Note: only the first callback tagged ``checkpoint_component=="dataloader"`` + is considered; if it does not currently want its state checkpointed, + no further callbacks are searched. In practice there is at most one + such callback (see ``DataLoaderStateCallback``). + """ + + def __init__(self, callbacks: callback.CallBackGroup | None) -> None: + self._callback: _DataloaderStateHandler | None = None + if callbacks is None: + return + for current_callback in callbacks._callbacks: + if getattr(current_callback, "checkpoint_component", None) != "dataloader": + continue + if current_callback.has_checkpoint_state(): + self._callback = current_callback + return + + def has_state(self) -> bool: + return self._callback is not None + + def state_dict(self) -> dict[Any, Any]: + if self._callback is None: + raise RuntimeError("No dataloader state handler is registered, cannot save dataloader state.") + return self._callback.state_dict() + + def load_state_dict(self, state_dict: dict[Any, Any]) -> None: + if self._callback is None: + raise RuntimeError("No dataloader state handler is registered, cannot load dataloader state.") + self._callback.load_state_dict(state_dict) + + +class AsyncMode(str, enum.Enum): + DISABLED = "disabled" + ASYNC_WITH_PINNED_MEM = "async_with_pinned_mem" + + +class Terminate: + pass + + +class SaveDone: + def __init__(self, iteration: int, elapsed_time: float, succeeded: bool): + self.iteration = iteration + self.elapsed_time = elapsed_time + self.succeeded = succeeded + + def __str__(self): + return f"SaveDone(iteration={self.iteration}, elapsed_time={self.elapsed_time}, succeeded={self.succeeded})" + + +def save_checkpoint_in_background( + receiver_queue: multiprocessing.Queue, + sender_queue: multiprocessing.Queue, + config_checkpoint: CheckpointConfig, + config_job: JobConfig, +) -> None: + """ + Handles model checkpoint saving in a separate background process using PyTorch's distributed functionality. + This function runs in a dedicated process to avoid blocking the main training loop. + + Args: + receiver_queue: Queue to receive state dictionaries and commands from the main process + sender_queue: Queue to send completion signals back to the main process + config_checkpoint: Configuration settings for checkpoint saving behavior + config_job: Configuration settings for the training job + + Flow: + 1. Initializes distributed processing environment + 2. Continuously waits for state dictionaries to save + 3. Saves checkpoints asynchronously + 4. Signals completion back to main process + 5. Terminates when receiving a Terminate signal + + Raises: + AssertionError: If received object is neither Terminate signal nor valid state dict tuple + + Note: + - Uses a different port than the main process to avoid conflicts + - Disables TorchElastic agent store for checkpoint operations + - Automatically cleans up distributed process group on exit + """ + # Configure distributed environment + os.environ["MASTER_PORT"] = str(int(os.environ["MASTER_PORT"]) + 2) + os.environ["TORCHELASTIC_USE_AGENT_STORE"] = "False" + + # Set up GPU device and distributed processing + torch.cuda.set_device(int(os.environ["LOCAL_RANK"])) + if dist.is_initialized(): + dist.destroy_process_group() + dist.init_process_group(backend="gloo") + + # Initialize checkpointing mechanism + checkpoint_handler = DistributedCheckpointer( + config_checkpoint=config_checkpoint, + config_job=config_job, + callbacks=None, + disable_async=True, + ) + + while True: + log.info(f"Checkpoint background process is ready for next task, waiting for new state_dict") + received_data = receiver_queue.get() + log.info(f"Checkpoint background process received new state_dict") + + if isinstance(received_data, Terminate): + log.info(f"Checkpoint background process received termination signal, closing sender queue") + break + + assert isinstance(received_data, tuple), "Received data must be a tuple of (state_dict, checkpoint_path)" + state_dict, checkpoint_path = received_data + + # Save checkpoint and measure time taken. + start_time = time.monotonic() + iteration = state_dict["trainer"][0]["iteration"] + succeeded = False + + try: + log.info(f"Saving checkpoint to {checkpoint_path}") + checkpoint_handler.save_state_dict_worker(state_dict, checkpoint_path) + succeeded = True + except Exception as e: + log.error(f"Error saving checkpoint to {checkpoint_path}: {e}") + # continue because if the thread exits, the main thread keeps on adding to the queue + finally: + elapsed_time = time.monotonic() - start_time + log.info( + f"Checkpoint save completed in background process. " + f"Time taken: {elapsed_time:.2f} seconds, iteration: {iteration}, " + f"status: {'SUCCESS' if succeeded else 'FAILURE'}" + ) + sender_queue.put(SaveDone(iteration, elapsed_time, succeeded)) + + log.info("Cleaning up: destroying distributed process group") + dist.destroy_process_group() + + +def _replace_keys_with_ema_keys(state_dict: STATE_DICT_TYPE) -> STATE_DICT_TYPE: + """ + Renames model parameters from "net." to "net_ema.". + """ + if not all(k.startswith("net.") for k in state_dict.keys()): + raise ValueError("State dict must start with net. keys when load_ema_to_reg is True") + return {k.replace("net.", "net_ema."): v for k, v in state_dict.items()} + + +class CustomLoadPlanner(dcp.DefaultLoadPlanner): + """ + CustomLoadPlanner that supports ignoring keys during checkpoint load. + This is useful when the checkpoint is saved with a different component + architecture, e.g. different RoPE embeddings than the current model. + """ + + def __init__( + self, + flatten_state_dict: bool = True, + flatten_sharded_tensors: bool = True, + allow_partial_load: bool = False, + keys_to_skip_loading: List[str] = [], + load_ema_to_reg: bool = False, + ) -> None: + super().__init__( + flatten_state_dict=flatten_state_dict, + flatten_sharded_tensors=flatten_sharded_tensors, + allow_partial_load=allow_partial_load, + ) + self.keys_to_skip_loading = keys_to_skip_loading + self.load_ema_to_reg = load_ema_to_reg + if len(keys_to_skip_loading) > 0: + log.info(f"Skipping loading of keys that match the following patterns: {keys_to_skip_loading}") + + def set_up_planner( + self, + state_dict: STATE_DICT_TYPE, + metadata: Metadata | None = None, + is_coordinator: bool = False, + ) -> None: + state_dict = self._skip_keys_if_found(state_dict) + + if self.load_ema_to_reg: + state_dict = _replace_keys_with_ema_keys(state_dict) + + super().set_up_planner( + state_dict=state_dict, + metadata=metadata, + is_coordinator=is_coordinator, + ) + + def _skip_keys_if_found( + self, + state_dict: STATE_DICT_TYPE, + ) -> Dict[str, Any]: + """ + While loading the checkpoint, skip the weight loading for the keys + that contain any element of `self.keys_to_skip_loading` as a substring. + """ + if len(self.keys_to_skip_loading) == 0: + return state_dict + + new_state_dict = {} + for fqn, obj in state_dict.items(): + if any(skip_key in fqn for skip_key in self.keys_to_skip_loading): + log.warning(f"Skipping loading of key: {fqn}") + continue + new_state_dict[fqn] = obj + return new_state_dict + + +class CustomSavePlanner(dcp.DefaultSavePlanner): + """ + Custom save planner that enables an override for cache_plans_key when + caching of save plans is enabled. Caching of save plans reduces checkpointing + time by reusing the same save plan across checkpoints. This reduces the + checkpointing time by ~60% (benchmarked using the 235B-A22B Qwen3-VL model + on 64 GB200 nodes). + """ + + def __init__( + self, + flatten_state_dict: bool = True, + flatten_sharded_tensors: bool = True, + dedup_save_to_lowest_rank: bool = False, + save_reg_to_ema: bool = False, + enable_plan_caching: bool = False, + cache_plans_key: str | None = None, + ) -> None: + super().__init__( + flatten_state_dict=flatten_state_dict, + flatten_sharded_tensors=flatten_sharded_tensors, + dedup_save_to_lowest_rank=dedup_save_to_lowest_rank, + enable_plan_caching=enable_plan_caching, + ) + if cache_plans_key is not None: + self._cached_plans_key = cache_plans_key + + self.save_reg_to_ema = save_reg_to_ema + + def set_up_planner( + self, + state_dict: STATE_DICT_TYPE, + storage_meta: StorageMeta | None = None, + is_coordinator: bool = False, + ) -> None: + if self.save_reg_to_ema: + state_dict = _replace_keys_with_ema_keys(state_dict) + + super().set_up_planner( + state_dict=state_dict, + storage_meta=storage_meta, + is_coordinator=is_coordinator, + ) + + +class DistributedCheckpointer(AbstractCheckpointer): + CHECKPOINT_KEYS = ["model", "optim", "scheduler", "trainer", "dataloader"] + + def __init__( + self, + config_checkpoint: CheckpointConfig, + config_job: JobConfig, + callbacks: Optional[callback.CallBackGroup] = None, + disable_async: bool = False, + ): + super().__init__(config_checkpoint, config_job, callbacks) + self.config_checkpoint = config_checkpoint + if config_checkpoint.dcp_async_mode_enabled and not disable_async: + self.async_mode = AsyncMode.ASYNC_WITH_PINNED_MEM + else: + self.async_mode = AsyncMode.DISABLED + + if self.async_mode == AsyncMode.ASYNC_WITH_PINNED_MEM: + ctx = get_context("spawn") + self.mp_queue_send = ctx.Queue() + self.mp_queue_recv = ctx.Queue() + self.mp = ctx.Process( + target=save_checkpoint_in_background, + args=( + self.mp_queue_send, + self.mp_queue_recv, + config_checkpoint, + config_job, + ), + daemon=True, + ) + self.mp.start() + self.cpu_offload_state_dict = None + self.staging_ckpt_file = None + self.staging_stream = torch.cuda.Stream() + self.checkpoint_in_progress = False + + def keys_to_resume_during_load(self) -> tuple[set[str], str | None, bool | None]: + """ + Determines the keys to resume from the checkpoint and the checkpoint path. + If the checkpoint is the latest checkpoint of the same model, then it is a + normal resume. If the checkpoint is a different model's checkpoint, then it is + a warm start. + + Args: + None + + Returns: + resume_keys: The keys to resume from the checkpoint. + checkpoint_path: The path to the checkpoint. If the checkpoint is a different + warm_start: Whether to warm start the training from a different model's checkpoint. + If the checkpoint is a different model's checkpoint, then this is True. + If the checkpoint is the latest checkpoint of the same model, then this is False. + """ + latest_checkpoint_file = self._read_latest_checkpoint_file() + + resume_keys = [] + warm_start = None + + if latest_checkpoint_file is not None: + # 1. Resume training from the latest checkpoint of the same model. + warm_start = False + checkpoint_path = os.path.join(self.load_dirname, latest_checkpoint_file) + resume_keys.extend(self.CHECKPOINT_KEYS) + + else: + if self.load_path and not str(self.load_path).endswith(".pt"): + # 2. Warm Start: Resume training from a different model's checkpoint + # specified by `load_path`. + warm_start = True + checkpoint_path = self.load_path + + if self.load_s3_backend_key: + checkpoint_path = f"s3://{self.config_checkpoint.load_from_object_store.bucket}/{checkpoint_path}" + + # If the path doesn't end with specific checkpoint, read the latest + # checkpoint file to determine the most recent checkpoint iteration. + if not re.search(r"/checkpoints/iter_\d{9}/?$", checkpoint_path): + old_ckpt_path = checkpoint_path + latest_ckpt_path = os.path.join(checkpoint_path, "checkpoints/latest_checkpoint.txt") + + # If the latest checkpoint file exists, use it to determine the + # checkpoint path. Otherwise, use the original path. + if easy_io.exists(latest_ckpt_path, backend_key=self.load_s3_backend_key): + checkpoint_file = easy_io.load( + latest_ckpt_path, backend_key=self.load_s3_backend_key + ).strip() + checkpoint_path = f"{checkpoint_path}/checkpoints/{checkpoint_file}" + else: + log.warning( + f"Latest checkpoint file {latest_ckpt_path} not found, load from {old_ckpt_path}" + ) + checkpoint_path = old_ckpt_path + + if self.load_training_state: + resume_keys.extend(self.CHECKPOINT_KEYS) + else: + resume_keys.append("model") + if self.only_load_scheduler_state: + resume_keys.append("scheduler") + else: + checkpoint_path = None + + if len(self.keys_not_to_resume) > 0: + for key in self.keys_not_to_resume: + assert key in self.CHECKPOINT_KEYS, f"Invalid key to resume: {key} not in {self.CHECKPOINT_KEYS}" + resume_keys = [key for key in resume_keys if key not in self.keys_not_to_resume] + + return set(resume_keys), checkpoint_path, warm_start + + @misc.timer("checkpoint loading") + def load( + self, + model: ImaginaireModel, + optimizer: torch.optim.Optimizer | None = None, + scheduler: torch.optim.lr_scheduler.LRScheduler | None = None, + grad_scaler: torch.amp.GradScaler | None = None, + ) -> int: + if self.callbacks is not None: + self.callbacks.on_load_checkpoint_start(model) + + resume_keys, checkpoint_path, warm_start = self.keys_to_resume_during_load() + resume_keys = sorted(resume_keys) + log.critical(f"Resuming ckpt {checkpoint_path} with keys: {resume_keys}") + + iteration = 0 + + if checkpoint_path is not None: + self._check_checkpoint_exists(checkpoint_path) + + for key in resume_keys: + dist.barrier() + + cur_key_ckpt_full_path = os.path.join(checkpoint_path, key) + log.critical(f"Start loading checkpoint from {cur_key_ckpt_full_path}") + + storage_reader = self.get_storage_reader(cur_key_ckpt_full_path) + strict_resume = self.config_checkpoint.strict_resume + + # Note that we only allow skipping loading of keys during warm start. If the checkpoint is + # the latest checkpoint of the same model, then we don't need to skip any keys. + keys_to_skip_loading = self.config_checkpoint.keys_to_skip_loading if warm_start else [] + + load_planner = CustomLoadPlanner( + allow_partial_load=not strict_resume, + keys_to_skip_loading=keys_to_skip_loading, + ) + + if key == "model": + log.info("- Loading the model...") + _model_wrapper = ModelWrapper(model) + _state_dict = _model_wrapper.state_dict() + dcp.load( + _state_dict, + storage_reader=storage_reader, + planner=load_planner, + ) + if self.config_checkpoint.load_ema_to_reg: + # The model has both net.* and net_ema.* submodules, so _state_dict + # contains both sets of keys after dcp.load(). Copy EMA weights into + # regular model weights so we can resume from EMA and reset EMA. + for sd_key in list(_state_dict.keys()): + if sd_key.startswith("net."): + key_ema = "net_ema." + sd_key.removeprefix("net.") + assert key_ema in _state_dict, ( + f"EMA key {key_ema} not found in state_dict. " + "Ensure the model has net_ema submodule." + ) + _state_dict[sd_key] = _state_dict[key_ema] + results = _model_wrapper.load_state_dict(_state_dict) + if results is not None: + if len(results.missing_keys) > 0: + raise ValueError(f"Missing keys (not found in checkpoint): {results.missing_keys}") + if len(results.unexpected_keys) > 0: + raise ValueError( + f"Unexpected keys (found in checkpoint but not in model): {results.unexpected_keys}" + ) + + elif key == "optim": + log.info("- Loading the optimizer...") + _state_dict = optimizer.state_dict() + dcp.load( + _state_dict, + storage_reader=storage_reader, + planner=load_planner, + ) + optimizer.load_state_dict(_state_dict) + + elif key == "scheduler": + log.info("- Loading the scheduler...") + _state_dict = scheduler.state_dict() + dcp.load( + _state_dict, + storage_reader=storage_reader, + planner=load_planner, + ) + scheduler.load_state_dict(_state_dict) + + elif key == "trainer": + log.info("- Loading the trainer...") + + # Use rank-specific key for RNG state to support correct per-rank restoration + rng_key = f"rng_state_{dist.get_rank()}" + current_rng_state = get_rand_state_dict() + _state_dict = { + "grad_scaler": grad_scaler.state_dict(), + "iteration": iteration, + } + # Check if rng_key exists in checkpoint metadata to avoid failure with strict_resume=True + metadata = storage_reader.read_metadata() + rng_key_exists = any( + k.startswith(f"{rng_key}.") or k == rng_key for k in metadata.state_dict_metadata.keys() + ) + if rng_key_exists: + _state_dict[rng_key] = current_rng_state + + dcp.load( + _state_dict, + storage_reader=storage_reader, + planner=load_planner, + ) + grad_scaler.load_state_dict(_state_dict["grad_scaler"]) + iteration = _state_dict["iteration"] + set_rand_state_dict(_state_dict.get(rng_key, current_rng_state)) + + elif key == "dataloader": + if not easy_io.exists(cur_key_ckpt_full_path, backend_key=self.load_s3_backend_key): + log.info( + f"Checkpoint {cur_key_ckpt_full_path} does not exist, skip loading dataloader.", + rank0_only=False, + ) + continue + + rank = dist.get_rank() + dataloader_pkl_path = os.path.join(cur_key_ckpt_full_path, f"rank_{rank}.pkl") + if not easy_io.exists(dataloader_pkl_path, backend_key=self.load_s3_backend_key): + log.info(f"No dataloader checkpoint found at {dataloader_pkl_path}", rank0_only=False) + continue + + log.info(f"- Loading the dataloader {cur_key_ckpt_full_path}...", rank0_only=False) + _state_dict = easy_io.load( + dataloader_pkl_path, + file_format="pkl", + backend_key=self.load_s3_backend_key, + ) + dataloader_wrapper = _DataloaderWrapper(self.callbacks) + if dataloader_wrapper.has_state(): + dataloader_wrapper.load_state_dict(_state_dict) + + else: + raise ValueError(f"Invalid key: {key}. not support to resume.") + + if self.callbacks is not None and resume_keys: + # Note that this callback is never used in the codebase. + self.callbacks.on_load_checkpoint(model, state_dict={}) + log.info(f"Loaded checkpoint from {checkpoint_path} in iteration {iteration}") + + else: + log.info("Training from scratch.") + + torch.cuda.empty_cache() + + if self.callbacks is not None: + self.callbacks.on_load_checkpoint_end(model, iteration=iteration, checkpoint_path=checkpoint_path) + return iteration + + def _checkpoint_async_with_pinned_memory( + self, checkpoint_file: str, state_dict: Dict[str, Tuple[Any, str]] + ) -> None: + assert self.async_mode == AsyncMode.ASYNC_WITH_PINNED_MEM, "Async mode must be AsyncMode.ASYNC_WITH_PINNED_MEM" + + from torch.distributed._state_dict_utils import _copy_state_dict, _create_cpu_state_dict + + if self.cpu_offload_state_dict is None: + log.info(f"Preparing the CPU memory for staging") + self.cpu_offload_state_dict = _create_cpu_state_dict(state_dict, pin_memory=True, share_memory=True) + + log.info(f"Staging the state_dict in CPU memory") + with torch.cuda.stream(self.staging_stream): + self.cpu_offload_state_dict = _copy_state_dict( + state_dict, + self.cpu_offload_state_dict, + non_blocking=True, + ) + self.staging_ckpt_file = checkpoint_file + + self.staging_stream.synchronize() + log.info(f"Staging the state_dict in CPU memory completed") + + self.mp_queue_send.put_nowait((self.cpu_offload_state_dict, self.staging_ckpt_file)) + self.checkpoint_in_progress = True + log.info(f"Submitted checkpoint to background process") + + def _wait_for_previous_async_checkpoint(self) -> None: + """ + Gets the results of previously submitted checkpoints. + Pass them to callbacks if checkpoint succeeded. + """ + assert self.async_mode == AsyncMode.ASYNC_WITH_PINNED_MEM, "Async mode must be AsyncMode.ASYNC_WITH_PINNED_MEM" + + if not self.checkpoint_in_progress: + return + + success = False + try: + log.info(f"Waiting for checkpoint save result") + + # Note that we set a timeout of 1 hour to avoid blocking the main process + # indefinitely. Gloo and NCCL timeouts are ~30 minutes, so this timeout + # should typically be sufficient. + save_done: SaveDone = self.mp_queue_recv.get(timeout=3600) + + log.info(f"Received checkpoint save result: {save_done}") + + if self.callbacks is not None and save_done.succeeded: + self.callbacks.on_save_checkpoint_success( + iteration=save_done.iteration, elapsed_time=save_done.elapsed_time + ) + self.checkpoint_in_progress = False + success = save_done.succeeded + + except Exception as e: + log.error(f"Error waiting for checkpoint save result: {e}") + + if not success: + # Terminate training execution upon a failed checkpoint save attempt. + # A failure at this stage typically indicates a non-recoverable system error. + # Continuing execution would result in subsequent persistent failures and + # unnecessary waste of GPU resources. + raise RuntimeError("Previous checkpoint save failed. Exiting...") + + def get_storage_writer(self, checkpoint_path: str) -> Union[S3StorageWriter, FileSystemWriter]: + if self.save_to_object_store: + return S3StorageWriter( + credential_path=self.config_checkpoint.save_to_object_store.credentials, + path=checkpoint_path, + enable_gcs_patch_in_boto3=self.config_checkpoint.enable_gcs_patch_in_boto3, + ) + return FileSystemWriter(path=checkpoint_path) + + def get_storage_reader(self, checkpoint_path: str) -> Union[S3StorageReader, FileSystemReader]: + if self.load_from_object_store: + return S3StorageReader( + credential_path=self.config_checkpoint.load_from_object_store.credentials, + path=checkpoint_path, + enable_gcs_patch_in_boto3=self.config_checkpoint.enable_gcs_patch_in_boto3, + ) + return FileSystemReader(checkpoint_path) + + def _save_as_pkl(self, obj: Any, output_dir: str) -> None: + """Save per-rank Python checkpoint state such as no-replace dataloader progress.""" + rank = dist.get_rank() + path = os.path.join(output_dir, f"rank_{rank}.pkl") + easy_io.dump( + obj, + path, + file_format="pkl", + backend_key=self.save_s3_backend_key, + ) + log.info(f"Saved state to {path}") + + def save_state_dict_worker(self, to_save_dict: Dict[str, Tuple[Any, str]], checkpoint_file: str) -> None: + for key, (v, full_checkpoint_path) in to_save_dict.items(): + if key == "dataloader": + self._save_as_pkl(v, full_checkpoint_path) + else: + storage_writer = self.get_storage_writer(full_checkpoint_path) + # Note that it is ok to create a new CustomSavePlanner object + # for each checkpoint save since the save plans are cached in a + # class dictionary. + save_planner = CustomSavePlanner( + dedup_save_to_lowest_rank=True, + enable_plan_caching=True, + cache_plans_key=f"custom_planner_{key}", + ) + dcp.save( + v, + storage_writer=storage_writer, + planner=save_planner, + ) + + if distributed.is_rank0(): + log.info(f"Saving last checkpoint file {checkpoint_file}") + self._write_latest_checkpoint_file(checkpoint_file) + + log.info(f"Saved checkpoint to {os.path.join(self.save_dirname, checkpoint_file)}") + + def save( + self, + model: ImaginaireModel, + optimizer: torch.optim.Optimizer, + scheduler: torch.optim.lr_scheduler.LRScheduler, + grad_scaler: torch.amp.GradScaler, + iteration: int, + ) -> None: + """Save network weights, optimizer parameters, scheduler parameters to a checkpoint. + + Args: + model (ImaginaireModel): The PyTorch model. + optimizer (torch.optim.Optimizer): The model optimizer. + scheduler (torch.optim.lr_scheduler.LRScheduler): The optimization scheduler. + grad_scaler (torch.amp.GradScaler): The gradient scaler (for mixed precision training). + iteration (int): Current iteration number. + """ + if self.async_mode == AsyncMode.ASYNC_WITH_PINNED_MEM: + self._wait_for_previous_async_checkpoint() + + if self.callbacks is not None: + self.callbacks.on_save_checkpoint_start(model, iteration) + + checkpoint_file = f"iter_{iteration:09}" + + # Use rank-specific key for RNG state to ensure each rank saves its own state + rng_key = f"rng_state_{dist.get_rank()}" + + to_save_dict = { + "model": ModelWrapper(model).state_dict(), + "optim": optimizer.state_dict(), + "scheduler": scheduler.state_dict(), + "trainer": { + "grad_scaler": grad_scaler.state_dict(), + "iteration": iteration, + rng_key: get_rand_state_dict(), + }, + } + dataloader_wrapper = _DataloaderWrapper(self.callbacks) + if dataloader_wrapper.has_state(): + to_save_dict["dataloader"] = dataloader_wrapper.state_dict() + + if self.callbacks is not None: + self.callbacks.on_save_checkpoint(model, state_dict=to_save_dict) + + for k in to_save_dict.keys(): + output_dirname = os.path.join(self.save_dirname, f"iter_{iteration:09}/{k}") + to_save_dict[k] = (to_save_dict[k], output_dirname) + + if self.async_mode == AsyncMode.ASYNC_WITH_PINNED_MEM: + dataloader_entry = to_save_dict.pop("dataloader", None) + if dataloader_entry is not None: + dataloader_state, dataloader_save_dir = dataloader_entry + self._save_as_pkl(dataloader_state, dataloader_save_dir) + self._checkpoint_async_with_pinned_memory(checkpoint_file, to_save_dict) + else: + start_time = time.monotonic() + self.save_state_dict_worker(to_save_dict, checkpoint_file) + elapsed_time = time.monotonic() - start_time + log.info(f"Checkpoint save completed: Time taken: {elapsed_time:.2f} seconds") + + if self.callbacks is not None: + self.callbacks.on_save_checkpoint_success(iteration=iteration, elapsed_time=elapsed_time) + + # This measures exposed (synchronous) checkpoint time, on_save_checkpoint_success() + # is instead called to measure the entire duration for asynchronous checkpoint for the async case too. + if self.callbacks is not None: + self.callbacks.on_save_checkpoint_end(model=None, iteration=iteration) + + def finalize(self) -> None: + super().finalize() + if self.async_mode == AsyncMode.ASYNC_WITH_PINNED_MEM: + if self.mp and self.mp.is_alive(): + # Wait for the previous checkpoint to complete. + self._wait_for_previous_async_checkpoint() + + self.mp_queue_send.put(Terminate()) + self.mp.join() diff --git a/cosmos_framework/checkpoint/dummy.py b/cosmos_framework/checkpoint/dummy.py new file mode 100644 index 0000000..3cfa1ba --- /dev/null +++ b/cosmos_framework/checkpoint/dummy.py @@ -0,0 +1,35 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Optional + +import torch +import torch.distributed + +from cosmos_framework.checkpoint.base import AbstractCheckpointer +from cosmos_framework.model._base import ImaginaireModel + + +class Checkpointer(AbstractCheckpointer): + """ + A dummy checkpointer that does not save or load anything. This is useful for debugging jobs or share workload with collobrators. + """ + + def save( + self, + model: ImaginaireModel, + optimizer: torch.optim.Optimizer, + scheduler: torch.optim.lr_scheduler.LRScheduler, + grad_scaler: torch.amp.GradScaler, + iteration: int, + ) -> None: + pass + + def load( + self, + model: ImaginaireModel, + optimizer: Optional[torch.optim.Optimizer] = None, + scheduler: Optional[torch.optim.lr_scheduler.LRScheduler] = None, + grad_scaler: Optional[torch.amp.GradScaler] = None, + ) -> int: + return 0 diff --git a/cosmos_framework/checkpoint/s3_filesystem.py b/cosmos_framework/checkpoint/s3_filesystem.py new file mode 100644 index 0000000..e47219e --- /dev/null +++ b/cosmos_framework/checkpoint/s3_filesystem.py @@ -0,0 +1,318 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import io +import os +import time +from contextlib import contextmanager +from typing import Generator, Union +from urllib.parse import urlparse + +from botocore.exceptions import ClientError +from torch.distributed.checkpoint import FileSystemReader, FileSystemWriter +from torch.distributed.checkpoint.filesystem import FileSystemBase + +from cosmos_framework.utils import log +from cosmos_framework.utils.easy_io import easy_io + + +class S3Stream(io.BytesIO): + """ + Workaround for PyTorch manually closing the stream before we can upload it to S3. We override the close() as noop + and instead call our own _true_close() method to close the stream after we are done using it. + The commit at fault is https://github.com/pytorch/pytorch/commit/9c909bf3bb122db2cce95e2eb7459bbe50dfa15a + """ + + def close(self): + self.flush() + # No close + + def _true_close(self): + super().close() + + +class S3FileSystem(FileSystemBase): + """Implementation of FileSystemBase for AWS S3 storage.""" + + def __init__( + self, + credential_path: str, + max_attempts: int = 20, + initial_backoff: float = 1.0, + max_backoff: float = 30.0, + backoff_factor: float = 2.0, + enable_gcs_patch_in_boto3: bool = False, + ) -> None: + """ + Initialize S3FileSystem with retry configuration. + + Args: + credential_path: Path to AWS credentials JSON file + max_attempts: Maximum number of retry attempts + initial_backoff: Initial backoff time in seconds + max_backoff: Maximum backoff time in seconds + backoff_factor: Multiplicative factor for backoff time + enable_gcs_patch_in_boto3: Whether to enable GCS patch in boto3 + """ + self.easy_io_backend = easy_io.get_file_backend( + backend_args={ + "backend": "s3", + "s3_credential_path": credential_path, + "path_mapping": None, + } + ) + self.max_attempts = max_attempts + self.initial_backoff = initial_backoff + self.max_backoff = max_backoff + self.backoff_factor = backoff_factor + self.enable_gcs_patch_in_boto3 = enable_gcs_patch_in_boto3 + if enable_gcs_patch_in_boto3: + log.info("enable_gcs_patch_in_boto3: True") + + def _retry_with_backoff(self, operation_func, *args, **kwargs): + """ + Execute an operation with exponential backoff retry logic. + + Args: + operation_func: Function to execute + *args: Positional arguments for the function + **kwargs: Keyword arguments for the function + + Returns: + Result of the operation function + + Raises: + Exception: If all retry attempts fail + """ + last_exception = None + backoff = self.initial_backoff + + for attempt in range(self.max_attempts): + try: + return operation_func(*args, **kwargs) + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + log.info(f"S3 Filesystem: Received ClientError: {error_code}", rank0_only=False) + + # Handle specific error cases + if error_code in ["SlowDown", "ThrottlingException", "RequestLimitExceeded", "InternalError"]: + last_exception = e + if attempt < self.max_attempts - 1: # Don't sleep on last attempt + current_backoff = min(backoff, self.max_backoff) + log.info(f"S3 Filesystem: Retrying in {current_backoff} seconds", rank0_only=False) + time.sleep(current_backoff) + backoff *= self.backoff_factor + continue + # For other client errors, raise immediately + raise + except Exception as e: + log.info(f"S3 Filesystem: Received Exception: {str(e)}", rank0_only=False) + last_exception = e + if attempt < self.max_attempts - 1: + current_backoff = min(backoff, self.max_backoff) + log.info(f"S3 Filesystem: Retrying in {current_backoff} seconds", rank0_only=False) + time.sleep(current_backoff) + backoff *= self.backoff_factor + continue + + # pyrefly: ignore [bad-raise] + raise last_exception + + @contextmanager + def create_stream(self, path: Union[str, os.PathLike], mode: str) -> Generator[io.IOBase, None, None]: + """Create a stream for reading from or writing to S3 with retry logic.""" + path_str = str(path) + bucket, key = self._parse_s3_uri(path_str) + log.info(f"S3 Filesystem: Creating stream for {key} in bucket {bucket}", rank0_only=False) + + if mode == "rb": + stream = io.BytesIO() + try: + + def download_operation(): + stream.write(self.easy_io_backend.get(filepath=path_str)) + stream.seek(0) + + log.info(f"S3 Filesystem: Downloading {key} from bucket {bucket}", rank0_only=False) + self._retry_with_backoff(download_operation) + log.info("S3 Filesystem: Download complete", rank0_only=False) + yield stream + finally: + stream.close() + elif mode == "wb": + stream = S3Stream() + try: + yield stream + + def upload_operation(): + stream.seek(0) + self.easy_io_backend.put(obj=stream, filepath=path_str) + + log.info(f"S3 Filesystem: Uploading {key} to bucket {bucket}", rank0_only=False) + self._retry_with_backoff(upload_operation) + log.info("S3 Filesystem: Upload complete", rank0_only=False) + finally: + stream._true_close() + else: + raise ValueError(f"Unsupported mode: {mode}") + + def concat_path(self, path: Union[str, os.PathLike], suffix: str) -> Union[str, os.PathLike]: + """Concatenate S3 path with suffix.""" + path_str = str(path) + if path_str.endswith("/"): + return f"{path_str}{suffix}" + return f"{path_str}/{suffix}" + + def init_path(self, path: Union[str, os.PathLike]) -> Union[str, os.PathLike]: + """Initialize and validate S3 path.""" + path_str = str(path) + if not path_str.startswith("s3://"): + raise ValueError(f"Invalid S3 URI: {path_str}. Must start with 's3://'") + return path_str + + def rename(self, path: Union[str, os.PathLike], new_path: Union[str, os.PathLike]) -> None: + """Rename (move) an object in S3 with retry logic.""" + src_path = str(path) + dst_path = str(new_path) + + def copy_operation(): + self.easy_io_backend.copyfile(src=src_path, dst=dst_path) + + self._retry_with_backoff(copy_operation) + + def delete_operation(): + self.easy_io_backend.remove(filepath=src_path) + + self._retry_with_backoff(delete_operation) + + def mkdir(self, path: Union[str, os.PathLike]) -> None: + """ + Create a "directory" in S3. + + Note: S3 doesn't have real directories, but we can create an empty object + with a trailing slash to simulate a directory. + """ + # Creating same buckets from different ranks can cause rate limit issues in GCP. + # In object store, we don't need to create a directory. + pass + + def ls(self, path: Union[str, os.PathLike]) -> list[str]: + """List objects under the given S3 path (prefix) and return s3:// URIs.""" + path_str = str(path) + return [ + f"{path_str.removesuffix('/')}/{obj_suffix}" + for obj_suffix in self.easy_io_backend.list_dir_or_file(dir_path=path_str, list_dir=False, list_file=True) + ] + + @classmethod + def validate_checkpoint_id(cls, checkpoint_id: Union[str, os.PathLike]) -> bool: + """Validate if the checkpoint_id is a valid S3 URI.""" + checkpoint_id_str = str(checkpoint_id) + try: + if not checkpoint_id_str.startswith("s3://"): + return False + parsed = urlparse(checkpoint_id_str) + return bool(parsed.netloc and parsed.path) # Must have bucket and key + except Exception: + return False + + def exists(self, path: Union[str, os.PathLike]) -> bool: + """Check if an object exists in S3 with retry logic.""" + try: + + def head_operation() -> bool: + return self.easy_io_backend.exists(filepath=str(path)) + + return self._retry_with_backoff(head_operation) + except ClientError as e: + if e.response.get("Error", {}).get("Code", "") == "404": + return False + raise + + def rm_file(self, path: Union[str, os.PathLike]) -> None: + """Remove a file from S3 with retry logic.""" + + def delete_operation(): + self.easy_io_backend.remove(filepath=str(path)) + + self._retry_with_backoff(delete_operation) + + def _parse_s3_uri(self, uri: str) -> tuple[str, str]: + """ + Parse an S3 URI into bucket and key. + + Args: + uri: S3 URI in the format s3://bucket-name/key + + Returns: + Tuple of (bucket_name, key) + + Raises: + ValueError: If the URI is invalid + """ + uri = uri if isinstance(uri, str) else str(uri) + if not uri.startswith("s3://"): + raise ValueError(f"Invalid S3 URI: {uri}. Must start with 's3://'") + + parsed = urlparse(uri) + bucket = parsed.netloc + + # Remove leading slash from key + key = parsed.path.lstrip("/") + + if not bucket: + raise ValueError(f"Invalid S3 URI: {uri}. No bucket specified") + + return bucket, key + + +class S3StorageWriter(FileSystemWriter): + def __init__( + self, + credential_path: str, + path: str, + enable_gcs_patch_in_boto3: bool = False, + **kwargs, + ) -> None: + """ + Initialize an S3 writer for distributed checkpointing. + + Args: + region (str): The AWS region for S3. + path (str): The S3 URI to write checkpoints to. + kwargs (dict): Keyword arguments to pass to the parent :class:`FileSystemWriter`. + enable_gcs_patch_in_boto3 (bool): Whether to enable GCS patch in boto3 + """ + super().__init__( + path=path, + sync_files=False, + **kwargs, + ) + self.fs = S3FileSystem(credential_path, enable_gcs_patch_in_boto3=enable_gcs_patch_in_boto3) # type: ignore + self.path = self.fs.init_path(path) + + @classmethod + def validate_checkpoint_id(cls, checkpoint_id: Union[str, os.PathLike]) -> bool: + return S3FileSystem.validate_checkpoint_id(checkpoint_id) + + +class S3StorageReader(FileSystemReader): + def __init__( + self, credential_path: str, path: Union[str, os.PathLike], enable_gcs_patch_in_boto3: bool = False + ) -> None: + """ + Initialize an S3 reader for distributed checkpointing. + + Args: + region (str): The AWS region for S3. + path (Union[str, os.PathLike]): The S3 path to read checkpoints from. + enable_gcs_patch_in_boto3 (bool): Whether to enable GCS patch in boto3 + """ + super().__init__(path) + self.fs = S3FileSystem(credential_path, enable_gcs_patch_in_boto3=enable_gcs_patch_in_boto3) # type: ignore + self.path = self.fs.init_path(path) + self.sync_files = False + + @classmethod + def validate_checkpoint_id(cls, checkpoint_id: Union[str, os.PathLike]) -> bool: + return S3FileSystem.validate_checkpoint_id(checkpoint_id) diff --git a/cosmos_framework/configs/base/__init__.py b/cosmos_framework/configs/base/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/configs/base/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/configs/base/base_config_test.py b/cosmos_framework/configs/base/base_config_test.py new file mode 100644 index 0000000..093e17b --- /dev/null +++ b/cosmos_framework/configs/base/base_config_test.py @@ -0,0 +1,102 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +""" +This file is used to test the config of the cosmos3 vfm project. +It is used to verify the config is loadable. + +To run the test, you can use the following command: +pytest -s cosmos_framework/configs/base/base_config_test.py +""" + +import importlib +from unittest.mock import MagicMock, patch + +import pytest + +from cosmos_framework.utils.config_helper import get_config_module, override + + +@pytest.mark.L0 +@pytest.mark.parametrize( + "experiment_name", + [ + "t2i_mot_exp001_009_qwen3_vl_2b_256res_frozen_llm", + ], +) +def test_config_init_experiment_mot(experiment_name): + """ + Parameterized test to verify config initialization for multiple experiments. + PYTHONPATH=. torchrun --nproc_per_node=8 -m pytest -s cosmos_framework/configs/base/config_test_mot.py --L1 + """ + config_file = "configs/base/config.py" + config_module = get_config_module(config_file) + config = importlib.import_module(config_module).make_config() + config = override( + config, + [ + "--", + f"experiment={experiment_name}", + ], + ) + + +def _make_self_mock(*, pretrained_enabled: bool, load_weights_from_pretrained: bool) -> MagicMock: + """Mock the OmniMoTModel attributes that load_pretrained_model_if_needed reads.""" + self_mock = MagicMock() + self_mock.vlm_config.pretrained_weights.enabled = pretrained_enabled + self_mock.config.diffusion_expert_config.load_weights_from_pretrained = load_weights_from_pretrained + self_mock.config.ema.enabled = False + return self_mock + + +@pytest.mark.L0 +class TestLoadPretrainedGate: + """Decision matrix for ``OmniMoTModel.load_pretrained_model_if_needed``. + + Replaces the previous ``OmniMoTModelConfig.validate`` tests now that + LoadPretrained callback probes ``latest_checkpoint.txt`` / ``load_path`` at + ``on_train_start`` and forwards the two booleans, instead of mutating the + config during validation. + """ + + _LOADER_TARGET = "cosmos_framework.model.vfm.omni_mot_model.load_language_model_safetensors" + + def _call(self, self_mock: MagicMock, *, has_resumable_checkpoint: bool, has_load_path: bool) -> MagicMock: + from cosmos_framework.model.vfm.omni_mot_model import OmniMoTModel + + with patch(self._LOADER_TARGET) as loader: + OmniMoTModel.load_pretrained_model_if_needed( + self_mock, + has_resumable_checkpoint=has_resumable_checkpoint, + has_load_path=has_load_path, + ) + return loader + + def test_fresh_init_loads_and_copies(self): + """No checkpoint, no load_path: HF load AND understanding→generation copy.""" + self_mock = _make_self_mock(pretrained_enabled=True, load_weights_from_pretrained=True) + loader = self._call(self_mock, has_resumable_checkpoint=False, has_load_path=False) + loader.assert_called_once() + self_mock.net.language_model.init_moe.assert_called_once() + + def test_resume_skips_everything(self): + """Resumable checkpoint exists: neither HF load nor copy.""" + self_mock = _make_self_mock(pretrained_enabled=True, load_weights_from_pretrained=True) + loader = self._call(self_mock, has_resumable_checkpoint=True, has_load_path=False) + loader.assert_not_called() + self_mock.net.language_model.init_moe.assert_not_called() + + def test_warm_start_loads_but_skips_copy(self): + """load_path set, no checkpoint: HF load but skip understanding→generation copy.""" + self_mock = _make_self_mock(pretrained_enabled=True, load_weights_from_pretrained=True) + loader = self._call(self_mock, has_resumable_checkpoint=False, has_load_path=True) + loader.assert_called_once() + self_mock.net.language_model.init_moe.assert_not_called() + + def test_pretrained_disabled_short_circuits(self): + """pretrained_weights.enabled=False: early return regardless of other flags.""" + self_mock = _make_self_mock(pretrained_enabled=False, load_weights_from_pretrained=True) + loader = self._call(self_mock, has_resumable_checkpoint=False, has_load_path=False) + loader.assert_not_called() + self_mock.net.language_model.init_moe.assert_not_called() diff --git a/cosmos_framework/configs/base/config.py b/cosmos_framework/configs/base/config.py new file mode 100644 index 0000000..435ff02 --- /dev/null +++ b/cosmos_framework/configs/base/config.py @@ -0,0 +1,113 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Any, List + +import attrs + +from cosmos_framework.utils import config +from cosmos_framework.trainer import ImaginaireTrainer as Trainer +from cosmos_framework.utils.config_helper import import_all_modules_from_package + + +@attrs.define(slots=False) +class DataSetting: + """Configuration for data. + + Attributes: + qwen_max_video_token_length: Maximum video token length. + qwen_target_fps: Target fps for video sampling. + text_chat_order: Order of text items in user messages. + """ + + qwen_max_video_token_length: int = 8192 + + +@attrs.define(slots=False) +class Config(config.Config): + data_setting: DataSetting = attrs.field(factory=DataSetting) + defaults: List[Any] = attrs.field( + factory=lambda: [ + "_self_", + {"model": "mot_fsdp"}, + {"data_train": None}, + {"data_val": None}, + {"optimizer": "adamw"}, + {"scheduler": "warmup_cosine_lr"}, + {"checkpoint": "s3"}, + {"callbacks": ["basic", "optimization", "job_monitor", "generation"]}, + {"ema": "power"}, + {"tokenizer": "wan2pt2_tokenizer"}, + {"sound_tokenizer": None}, # Optional: for audio-video generation + {"cluster": "gcp_iad_gb200"}, + {"vlm_config": None}, + {"ckpt_type": "dcp"}, + {"experiment": None}, + ] + ) + + +def make_config() -> Config: + c = Config( + model=None, + optimizer=None, + scheduler=None, + dataloader_train=None, + dataloader_val=None, + ) + + # Specifying values through instances of attrs + c.job.project = "cosmos3_vfm" + c.job.group = "debug" + c.job.name = "delete_${now:%Y-%m-%d}_${now:%H-%M-%S}" + + c.trainer.type = Trainer + c.trainer.straggler_detection.enabled = False + c.trainer.max_iter = 400_000 + c.trainer.logging_iter = 20 + c.trainer.validation_iter = 100 + c.trainer.run_validation = False + c.trainer.callbacks = None + + c.upload_reproducible_setup = False + + from cosmos_framework.configs.base.defaults.callbacks import register_callbacks + from cosmos_framework.configs.base.defaults.checkpointer import register_checkpoint, register_ckpt_type + from cosmos_framework.configs.base.defaults.cluster import register_cluster + from cosmos_framework.configs.base.defaults.ema import register_ema + + # from cosmos_framework.configs.base.defaults.data import register_data + from cosmos_framework.configs.base.defaults.model import register_model + from cosmos_framework.configs.base.defaults.optimizer import register_optimizer, register_scheduler + from cosmos_framework.configs.base.defaults.tokenizer import register_sound_tokenizer, register_tokenizer + from cosmos_framework.configs.base.defaults.vlm import register_vlm + + # Call this function to register config groups for advanced overriding. the order follows the default config groups + # register_data() + register_model() + register_checkpoint() + register_ckpt_type() + register_optimizer() + register_scheduler() + register_callbacks() + register_tokenizer() + register_sound_tokenizer() + register_ema() + register_cluster() + register_vlm() + + # Register experiments explicitly. Auto-discover via + # import_all_modules_from_package("cosmos_framework.configs.base.experiment", ...) would walk + # the action posttrain_config subtree and try to import sibling files that + # reference unshipped modules (camera_dataset/av_dataset/imaginaire.modules), + # cascading into ModuleNotFoundError. Explicit imports avoid that. + import cosmos_framework.configs.base.experiment.action.posttrain_config.libero_policy_datapacker_experiment # noqa: F401 + import cosmos_framework.configs.base.experiment.sft.action_fdm_sft_nano # noqa: F401 + import cosmos_framework.configs.base.experiment.sft.action_fdm_sft_super # noqa: F401 + import cosmos_framework.configs.base.experiment.sft.action_policy_sft_nano # noqa: F401 + import cosmos_framework.configs.base.experiment.sft.action_policy_sft_super # noqa: F401 + import cosmos_framework.configs.base.experiment.sft.action_fdm_sft_nano_datapacker # noqa: F401 + import cosmos_framework.configs.base.experiment.sft.action_policy_sft_nano_datapacker # noqa: F401 + import cosmos_framework.configs.base.experiment.sft.vision_sft_nano # noqa: F401 + import cosmos_framework.configs.base.experiment.sft.vision_sft_super # noqa: F401 + return c diff --git a/cosmos_framework/configs/base/defaults/__init__.py b/cosmos_framework/configs/base/defaults/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/configs/base/defaults/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/configs/base/defaults/activation_checkpointing.py b/cosmos_framework/configs/base/defaults/activation_checkpointing.py new file mode 100644 index 0000000..543c029 --- /dev/null +++ b/cosmos_framework/configs/base/defaults/activation_checkpointing.py @@ -0,0 +1,69 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Shared activation checkpointing schema for the MoT and VLM paths. + +``ActivationCheckpointingConfig`` is referenced from both +``OmniMoTModelConfig.activation_checkpointing`` (MoT) and +``PolicyConfig.activation_checkpointing`` (VLM, in +vfm/configs/base/vlm/defaults/training.py), but the two read sites +consume different subsets of the schema: + +- MoT (vfm/models/mot/parallelize_unified_mot.py): every field is + consumed. +- VLM (vfm/models/vlm_model.py): only ``mode`` is consumed, and only + ``"full"`` actually enables checkpointing. The HF backbone exposes a + single binary ``gradient_checkpointing_enable`` API and has no + per-op SAC support, so ``"selective"`` is accepted at the type level + but degrades to no checkpointing on the VLM path; the SAC-specific + fields (``save_ops_regex``, ``preserve_rng_state``, + ``determinism_check``) are ignored entirely. +""" + +import attrs + + +@attrs.define(slots=False) +class ActivationCheckpointingConfig: + """Activation checkpointing (AC) policy shared by MoT and VLM training. + + Mirrors the torchtitan SAC design: a single ``mode`` knob switches between + full-block recompute, and per-op selective AC. The remaining fields are + knobs for the per-op selective policy or the underlying + ``torch.utils.checkpoint`` plumbing. + + Read sites: + + - MoT path consumes every field — see + cosmos_framework/model/vfm/mot/parallelize_unified_mot.py. + - VLM path consumes only ``mode`` (and only ``"full"`` enables + checkpointing) — see cosmos_framework/model/vfm/vlm_model.py. + """ + + # AC mode: + # - "selective": per-op SAC. Save expensive matmuls/attention + # ops, recompute the rest. MoT only — on the VLM + # path this degrades to no checkpointing because + # the HF backbone has no per-op SAC support. + # - "full": checkpoint each whole transformer block. + # - "none": no activation checkpointing. + mode: str = attrs.field( + default="full", + validator=attrs.validators.in_({"selective", "full", "none"}), + ) + + # Regex patterns for ops to save when using selective AC. Ignored if + # mode is "full" or "none". MoT only — unused on the VLM path. + save_ops_regex: list[str] = attrs.field( + factory=lambda: ["fmha"], + ) + + # Stash and restore RNG state across recompute boundaries. Required for + # deterministic output vs. non-checkpointed passes; slower otherwise. + # MoT only — unused on the VLM path. + preserve_rng_state: bool = True + + # Determinism check forwarded to ``ptd_checkpoint_wrapper`` / + # ``torch.utils.checkpoint.checkpoint``. MoT only — unused on the + # VLM path. + determinism_check: str = "default" diff --git a/cosmos_framework/configs/base/defaults/callbacks.py b/cosmos_framework/configs/base/defaults/callbacks.py new file mode 100644 index 0000000..46c85ed --- /dev/null +++ b/cosmos_framework/configs/base/defaults/callbacks.py @@ -0,0 +1,130 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Dataloader config options.""" + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.callbacks.manual_gc import ManualGarbageCollection +from cosmos_framework.utils.lazy_config import PLACEHOLDER +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils.callback import LowPrecisionCallback, WandBCallback +from cosmos_framework.callbacks.compile_tokenizer import CompileTokenizer + +from cosmos_framework.callbacks.device_monitor import DeviceMonitor +from cosmos_framework.callbacks.every_n_draw_sample import EveryNDrawSample +from cosmos_framework.callbacks.expert_heatmap import ExpertHeatmap +from cosmos_framework.callbacks.grad_clip import GradClip +from cosmos_framework.callbacks.heart_beat import HeartBeat +from cosmos_framework.callbacks.iter_speed import IterSpeed +from cosmos_framework.callbacks.load_pretrained import LoadPretrained +from cosmos_framework.callbacks.mfu import MFUCallback +from cosmos_framework.callbacks.moe_specialization_callback import MoESpecializationCallback +from cosmos_framework.callbacks.moe_stability_callback import MoEStabilityCallback +from cosmos_framework.callbacks.norm_monitor import NormMonitor +from cosmos_framework.callbacks.ofu import OFUCallback +from cosmos_framework.callbacks.param_count import ParamCount +from cosmos_framework.callbacks.sequence_packing_padding import SequencePackingPadding +from cosmos_framework.callbacks.sigma_loss_analysis import SigmaLossAnalysis +from cosmos_framework.callbacks.skip_nan_step import SkipNaNStep +from cosmos_framework.callbacks.termination_signal_checkpoint import TerminationSignalCheckpoint +from cosmos_framework.callbacks.training_stats import TrainingStatsCallback +from cosmos_framework.callbacks.wandb_log import WandbCallback as WandBCallbackMultiplier +from cosmos_framework.callbacks.wandb_log_eval import WandbCallback as WandBCallbackEval + +BASIC_CALLBACKS = dict( + iter_speed=L(IterSpeed)( # does not use model or optimizer + every_n="${trainer.logging_iter}", + save_s3="${upload_reproducible_setup}", + save_s3_every_log_n=500, + hit_thres=50, + ), + manual_gc=L(ManualGarbageCollection)(every_n=5), # does not use model or optimizer + wandb=L(WandBCallback)(), + wandb_2x=L(WandBCallbackMultiplier)( + logging_iter_multipler=2, + save_logging_iter_multipler=1, + save_s3="${upload_reproducible_setup}", + ), + param_count=L(ParamCount)( # use model + save_s3="${upload_reproducible_setup}", + ), + wandb_val=L(WandBCallbackEval)( + save_s3="${upload_reproducible_setup}", + ), + moe_stability=L(MoEStabilityCallback)(every_n=250), + moe_specialization=L(MoESpecializationCallback)(every_n=250), + expert_heatmap=L(ExpertHeatmap)(), + load_pretrained=L(LoadPretrained)(), + compile_tokenizer=L(CompileTokenizer)(enabled=False, compile_after_iterations=3), + norm_monitor=L(NormMonitor)( + every_n=5000, + log_stat_wandb=True, + save_s3="${upload_reproducible_setup}", + track_activations=True, + ), + sigma_loss_analysis=L(SigmaLossAnalysis)( + every_n=5000, + every_n_viz=5000, + save_s3="${upload_reproducible_setup}", + ), + sequence_packing_padding=L(SequencePackingPadding)(every_n="${trainer.logging_iter}"), + mfu=L(MFUCallback)(every_n="${trainer.logging_iter}", grad_accum_iter="${trainer.grad_accum_iter}"), + ofu=L(OFUCallback)(every_n="${trainer.logging_iter}"), +) + +JOB_MONITOR_CALLBACKS = dict( + heart_beat=L(HeartBeat)( + every_n=200, + update_interval_in_minute=20, + save_s3="${upload_reproducible_setup}", + ), + device_monitor=L(DeviceMonitor)( + every_n=200, + save_s3="${upload_reproducible_setup}", + upload_every_n_mul=5, + ), + termination_signal_checkpoint=L(TerminationSignalCheckpoint)( + min_save_fraction=1 / 3, + ), +) + +OPTIMIZATION_CALLBACKS = dict( + skip_nan_step=L(SkipNaNStep)(max_consecutive_nan=100), + grad_clip=L(GradClip)(clip_norm=1.0, track_per_modality=True), # image/video grad-norm split + low_precision=L(LowPrecisionCallback)(update_iter=1, config=PLACEHOLDER, trainer=PLACEHOLDER), # use model +) + +VIZ_ONLINE_SAMPLING_CALLBACKS = dict( + every_n_sample_reg=L(EveryNDrawSample)( + every_n=5000, + save_s3=True, + do_x0_prediction=False, + ), + every_n_sample_ema=L(EveryNDrawSample)( + every_n=5000, + is_ema=True, + save_s3=True, + do_x0_prediction=False, + ), +) + + +def register_callbacks(): + cs = ConfigStore.instance() + cs.store(group="callbacks", package="trainer.callbacks", name="basic", node=BASIC_CALLBACKS) + cs.store(group="callbacks", package="trainer.callbacks", name="job_monitor", node=JOB_MONITOR_CALLBACKS) + cs.store(group="callbacks", package="trainer.callbacks", name="optimization", node=OPTIMIZATION_CALLBACKS) + # Online sampling generation callback + cs.store( + group="callbacks", package="trainer.callbacks", name="viz_online_sampling", node=VIZ_ONLINE_SAMPLING_CALLBACKS + ) + # Register "generation" as alias for "viz_online_sampling" (expected by base config.py defaults) + cs.store(group="callbacks", package="trainer.callbacks", name="generation", node=VIZ_ONLINE_SAMPLING_CALLBACKS) + + TRAINING_STATS_CALLBACKS = dict( + training_stats=L(TrainingStatsCallback)( + log_freq=100, + ) + ) + cs.store(group="callbacks", package="trainer.callbacks", name="training_stats", node=TRAINING_STATS_CALLBACKS) diff --git a/cosmos_framework/configs/base/defaults/checkpointer.py b/cosmos_framework/configs/base/defaults/checkpointer.py new file mode 100644 index 0000000..1bf252f --- /dev/null +++ b/cosmos_framework/configs/base/defaults/checkpointer.py @@ -0,0 +1,118 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Canonical Hydra-group registry for checkpoint SKUs and checkpointer types.""" + +from typing import Dict + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils import config +from cosmos_framework.checkpoint.dummy import Checkpointer as DummyCheckpointer +from cosmos_framework.utils.config import CheckpointConfig +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.checkpoint.dcp import DistributedCheckpointer + +local_object_store = config.ObjectStoreConfig( + enabled=False, +) + +pdx_object_store = config.ObjectStoreConfig( + enabled=True, + credentials="credentials/pdx_vfm_checkpoint.secret", + bucket="checkpoints", +) + +s3_object_store = config.ObjectStoreConfig( + enabled=True, + credentials="credentials/s3_training.secret", + bucket="checkpoints-us-east-1", +) + +s3_eu_object_store = config.ObjectStoreConfig( + enabled=True, + credentials="credentials/s3_training_eu.secret", + bucket="checkpoints-eu-west-3", +) + +gcp_object_store = config.ObjectStoreConfig( + enabled=True, + credentials="credentials/gcp_checkpoint.secret", + bucket="nv-00-10206-checkpoint-experiments", +) + +neb_eu_object_store = config.ObjectStoreConfig( + enabled=True, + credentials="credentials/neb_eu.secret", + bucket="nv-01-10206-checkpoint-experiments", +) + +CHECKPOINT_LOCAL = CheckpointConfig( + save_to_object_store=local_object_store, + load_from_object_store=local_object_store, + save_iter=5000, + broadcast_via_filesystem=True, + dcp_async_mode_enabled=True, +) + +CHECKPOINT_PDX = CheckpointConfig( + save_to_object_store=pdx_object_store, + load_from_object_store=pdx_object_store, + save_iter=5000, + broadcast_via_filesystem=True, + dcp_async_mode_enabled=True, +) + +CHECKPOINT_S3 = CheckpointConfig( + save_to_object_store=s3_object_store, + load_from_object_store=s3_object_store, + save_iter=5000, + broadcast_via_filesystem=True, + dcp_async_mode_enabled=True, +) + +CHECKPOINT_S3_EU = CheckpointConfig( + save_to_object_store=s3_eu_object_store, + load_from_object_store=s3_eu_object_store, + save_iter=5000, + broadcast_via_filesystem=True, + dcp_async_mode_enabled=True, +) + +CHECKPOINT_GCP = CheckpointConfig( + save_to_object_store=gcp_object_store, + save_iter=1000, + load_from_object_store=gcp_object_store, + load_path="", + load_training_state=False, + strict_resume=True, + enable_gcs_patch_in_boto3=True, + dcp_async_mode_enabled=True, +) + +CHECKPOINT_NEB_EU = CheckpointConfig( + save_to_object_store=neb_eu_object_store, + load_from_object_store=neb_eu_object_store, + save_iter=2000, + broadcast_via_filesystem=True, +) + + +def register_checkpoint(): + cs = ConfigStore.instance() + cs.store(group="checkpoint", package="checkpoint", name="local", node=CHECKPOINT_LOCAL) + cs.store(group="checkpoint", package="checkpoint", name="pdx", node=CHECKPOINT_PDX) + cs.store(group="checkpoint", package="checkpoint", name="s3", node=CHECKPOINT_S3) + cs.store(group="checkpoint", package="checkpoint", name="s3_eu", node=CHECKPOINT_S3_EU) + cs.store(group="checkpoint", package="checkpoint", name="gcp", node=CHECKPOINT_GCP) + cs.store(group="checkpoint", package="checkpoint", name="neb_eu", node=CHECKPOINT_NEB_EU) + + +DUMMY_CHECKPOINTER: Dict[str, str] = L(DummyCheckpointer)() +DISTRIBUTED_CHECKPOINTER: Dict[str, str] = L(DistributedCheckpointer)() + + +def register_ckpt_type(): + cs = ConfigStore.instance() + cs.store(group="ckpt_type", package="checkpoint.type", name="dummy", node=DUMMY_CHECKPOINTER) + cs.store(group="ckpt_type", package="checkpoint.type", name="dcp", node=DISTRIBUTED_CHECKPOINTER) diff --git a/cosmos_framework/configs/base/defaults/cluster.py b/cosmos_framework/configs/base/defaults/cluster.py new file mode 100644 index 0000000..6aee769 --- /dev/null +++ b/cosmos_framework/configs/base/defaults/cluster.py @@ -0,0 +1,46 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import attrs +from hydra.core.config_store import ConfigStore + + +@attrs.define(slots=False) +class ClusterConfig: + """ + Config for the cluster specific information. + Everything cluster specific should be here. + """ + + object_store_bucket_data: str + object_store_bucket_checkpoint: str + object_store_bucket_pretrained: str + + object_store_credential_data: str + object_store_credential_checkpoint: str + object_store_credential_pretrained: str + + +AWSIADH100Config: ClusterConfig = ClusterConfig( + object_store_bucket_data="", + object_store_bucket_checkpoint="checkpoints-us-east-1", + object_store_bucket_pretrained="checkpoints-us-east-1", + object_store_credential_data="credentials/s3_training.secret", + object_store_credential_checkpoint="credentials/s3_checkpoint.secret", + object_store_credential_pretrained="credentials/s3_checkpoint.secret", +) + +GCPIADGB200Config: ClusterConfig = ClusterConfig( + object_store_bucket_data="", + object_store_bucket_checkpoint="nv-00-10206-checkpoint-experiments", + object_store_bucket_pretrained="nv-00-10206-checkpoint", + object_store_credential_data="credentials/gcp_checkpoint.secret", + object_store_credential_checkpoint="credentials/gcp_training.secret", + object_store_credential_pretrained="credentials/gcp_training.secret", +) + + +def register_cluster(): + cs = ConfigStore.instance() + cs.store(group="cluster", package="job.cluster", name="aws_iad_h100", node=AWSIADH100Config) + cs.store(group="cluster", package="job.cluster", name="gcp_iad_gb200", node=GCPIADGB200Config) diff --git a/cosmos_framework/configs/base/defaults/conditioner.py b/cosmos_framework/configs/base/defaults/conditioner.py new file mode 100644 index 0000000..c31aac0 --- /dev/null +++ b/cosmos_framework/configs/base/defaults/conditioner.py @@ -0,0 +1,182 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import random +from dataclasses import dataclass +from typing import Dict, Optional + +import torch +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils.lazy_config import LazyDict +from projects.cosmos3.vfm.conditioner import BooleanFlag, GeneralConditioner, ReMapkey, Text2WorldCondition +from cosmos_framework.utils.vfm.context_parallel import broadcast_split_tensor + + +@dataclass(frozen=True) +class Video2WorldCondition(Text2WorldCondition): + use_video_condition: bool = False + # the following two attributes are used to set the video condition; during training, inference + gt_frames: Optional[torch.Tensor] = None + condition_video_input_mask: Optional[torch.Tensor] = None + + def set_video_condition( + self, + gt_frames: torch.Tensor, + random_min_num_conditional_frames: int, + random_max_num_conditional_frames: int, + num_conditional_frames: Optional[int] = None, + conditional_frames_probs: Optional[Dict[int, float]] = None, + ) -> "Video2WorldCondition": + """ + Sets the video conditioning frames for video-to-video generation. + + This method creates a conditioning mask for the input video frames that determines + which frames will be used as context frames for generating new frames. The method + handles both image batches (T=1) and video batches (T>1) differently. + + Args: + gt_frames: A tensor of ground truth frames with shape [B, C, T, H, W], where: + B = batch size + C = number of channels + T = number of frames + H = height + W = width + + random_min_num_conditional_frames: Minimum number of frames to use for conditioning + when randomly selecting a number of conditioning frames. + + random_max_num_conditional_frames: Maximum number of frames to use for conditioning + when randomly selecting a number of conditioning frames. + + num_conditional_frames: Optional; If provided, all examples in the batch will use + exactly this many frames for conditioning. If None, a random number of frames + between random_min_num_conditional_frames and random_max_num_conditional_frames + will be selected for each example in the batch. + + conditional_frames_probs: Optional; Dictionary mapping number of frames to probabilities. + If provided, overrides the random_min/max_num_conditional_frames with weighted sampling. + Example: {0: 0.5, 1: 0.25, 2: 0.25} for 50% chance of 0 frames, 25% for 1, 25% for 2. + + Returns: + A new Video2WorldCondition object with the gt_frames and conditioning mask set. + The conditioning mask (condition_video_input_mask) is a binary tensor + of shape [B, 1, T, H, W] where 1 indicates frames used for conditioning and 0 + indicates frames to be generated. + + Notes: + - For image batches (T=1), no conditioning frames are used (num_conditional_frames_B = 0). + - For video batches: + - If num_conditional_frames is provided, all examples use that fixed number of frames. + - Otherwise, each example randomly uses between random_min_num_conditional_frames and + random_max_num_conditional_frames frames. + - The mask marks the first N frames as conditioning frames (set to 1) for each example. + """ + kwargs = self.to_dict(skip_underscore=False) + kwargs["gt_frames"] = gt_frames + + # condition_video_input_mask + B, _, T, H, W = gt_frames.shape + condition_video_input_mask = torch.zeros(B, 1, T, H, W, dtype=gt_frames.dtype, device=gt_frames.device) + if T == 1: # handle image batch + num_conditional_frames_B = torch.zeros(B, dtype=torch.int32) + else: # handle video batch + if num_conditional_frames is not None: + num_conditional_frames_B = torch.ones(B, dtype=torch.int32) * num_conditional_frames + elif conditional_frames_probs is not None: + # Use weighted sampling based on provided probabilities + frames_options = list(conditional_frames_probs.keys()) + weights = list(conditional_frames_probs.values()) + num_conditional_frames_B = torch.tensor( + random.choices(frames_options, weights=weights, k=B), dtype=torch.int32 + ) + else: + num_conditional_frames_B = torch.randint( + random_min_num_conditional_frames, random_max_num_conditional_frames + 1, size=(B,) + ) + for idx in range(B): + condition_video_input_mask[idx, :, : num_conditional_frames_B[idx], :, :] += 1 + + kwargs["condition_video_input_mask"] = condition_video_input_mask + return type(self)(**kwargs) + + def edit_for_inference( + self, is_cfg_conditional: bool = True, num_conditional_frames: int = 1 + ) -> "Video2WorldCondition": + _condition = self.set_video_condition( + gt_frames=self.gt_frames, + random_min_num_conditional_frames=0, + random_max_num_conditional_frames=0, + num_conditional_frames=num_conditional_frames, + ) + if not is_cfg_conditional: + # Do not use classifier free guidance on conditional frames. + # YB found that it leads to worse results. + _condition.use_video_condition.fill_(True) + return _condition + + def broadcast(self, process_group: torch.distributed.ProcessGroup) -> "Video2WorldCondition": + if self.is_broadcasted: + return self + # extra efforts + gt_frames = self.gt_frames + condition_video_input_mask = self.condition_video_input_mask + kwargs = self.to_dict(skip_underscore=False) + kwargs["gt_frames"] = None + kwargs["condition_video_input_mask"] = None + new_condition = Text2WorldCondition.broadcast( + type(self)(**kwargs), + process_group, + ) + + kwargs = new_condition.to_dict(skip_underscore=False) + _, _, T, _, _ = gt_frames.shape + if process_group is not None: + if T > 1 and process_group.size() > 1: + gt_frames = broadcast_split_tensor(gt_frames, seq_dim=2, process_group=process_group) + condition_video_input_mask = broadcast_split_tensor( + condition_video_input_mask, seq_dim=2, process_group=process_group + ) + kwargs["gt_frames"] = gt_frames + kwargs["condition_video_input_mask"] = condition_video_input_mask + return type(self)(**kwargs) + + +class Video2WorldConditioner(GeneralConditioner): + def forward( + self, + batch: Dict, + override_dropout_rate: Optional[Dict[str, float]] = None, + ) -> Video2WorldCondition: + output = super()._forward(batch, override_dropout_rate) + return Video2WorldCondition(**output) + + +_SHARED_CONFIG = dict( + fps=L(ReMapkey)( + input_key="fps", + output_key="fps", + dropout_rate=0.0, + dtype=None, + ), + use_video_condition=L(BooleanFlag)( + input_key="fps", + output_key="use_video_condition", + dropout_rate=0.2, + ), +) + +VideoPredictionConditioner: LazyDict = L(Video2WorldConditioner)( + **_SHARED_CONFIG, +) + + +def register_conditioner(): + cs = ConfigStore.instance() + cs.store( + group="conditioner", + package="model.config.conditioner", + name="video_prediction_conditioner", + node=VideoPredictionConditioner, + ) diff --git a/cosmos_framework/configs/base/defaults/ema.py b/cosmos_framework/configs/base/defaults/ema.py new file mode 100644 index 0000000..6b686b9 --- /dev/null +++ b/cosmos_framework/configs/base/defaults/ema.py @@ -0,0 +1,28 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import attrs +from hydra.core.config_store import ConfigStore + + +@attrs.define(slots=False) +class EMAConfig: + """ + Config for the EMA. + """ + + enabled: bool = True + rate: float = 0.1 + iteration_shift: int = 0 + + +PowerEMAConfig: EMAConfig = EMAConfig( + enabled=True, + rate=0.10, + iteration_shift=0, +) + + +def register_ema(): + cs = ConfigStore.instance() + cs.store(group="ema", package="model.config.ema", name="power", node=PowerEMAConfig) diff --git a/cosmos_framework/configs/base/defaults/model.py b/cosmos_framework/configs/base/defaults/model.py new file mode 100644 index 0000000..3273392 --- /dev/null +++ b/cosmos_framework/configs/base/defaults/model.py @@ -0,0 +1,42 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.defaults.model_config import ( + OmniMoTModelConfig, + ParallelismConfig, +) +from cosmos_framework.model.vfm.omni_mot_model import OmniMoTModel + +MOT_DDP_CONFIG = dict( + trainer=dict( + distributed_parallelism="ddp", + ), + model=L(OmniMoTModel)( + config=OmniMoTModelConfig(), + _recursive_=False, + ), +) + + +MOT_FSDP_CONFIG = dict( + trainer=dict( + distributed_parallelism="fsdp", + ), + model=L(OmniMoTModel)( + config=OmniMoTModelConfig( + parallelism=ParallelismConfig( + data_parallel_shard_degree=8, + ), + ), + _recursive_=False, + ), +) + + +def register_model(): + cs = ConfigStore.instance() + cs.store(group="model", package="_global_", name="mot_ddp", node=MOT_DDP_CONFIG) + cs.store(group="model", package="_global_", name="mot_fsdp", node=MOT_FSDP_CONFIG) diff --git a/cosmos_framework/configs/base/defaults/model_config.py b/cosmos_framework/configs/base/defaults/model_config.py new file mode 100644 index 0000000..1e6ee71 --- /dev/null +++ b/cosmos_framework/configs/base/defaults/model_config.py @@ -0,0 +1,281 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + + +from typing import Any + +import attrs + +from cosmos_framework.utils.lazy_config import LazyDict +from cosmos_framework.configs.base.defaults.activation_checkpointing import ActivationCheckpointingConfig +from cosmos_framework.configs.base.defaults.ema import EMAConfig +from cosmos_framework.configs.base.defaults.parallelism import ParallelismConfig +from cosmos_framework.configs.base.defaults.vlm import VLMConfig +from cosmos_framework.configs.base.vlm.defaults.training import PolicyConfig, TrainConfig +from cosmos_framework.configs.base.vlm.freeze_config import VLMFreezeConfig + + +@attrs.define(slots=False) +class DiffusionExpertConfig: + # This determines the range of timesteps before the fourier feature embedding is applied. + timestep_range: float = 1.0 + # Whether to load the generation pathway weights from pretrained LLM/VLM weights. + load_weights_from_pretrained: bool = True + + patch_spatial: int = 2 + max_vae_latent_side_after_patchify: int = ( + 20 # Max dimension (h or w) of the VAE latent after patchification (320/(8*2)) + ) + # Position embedding type for vision tokens: + # - "3d_rope": Additive 3D RoPE embeddings (VideoRopePosition3DEmb) + 1D position IDs for attention + # - "flattened_sin_cos": Additive flattened sin/cos embeddings + 1D position IDs for attention + # - "unified_3d_mrope": No additive embedding + 3D position IDs for Qwen3VL-style mRoPE attention + position_embedding_type: str = "3d_rope" + # When finetuning from lower resolution to higher resolution, the spatial resolution of videos increase. + # So, we need to adjust the position embedding. + # We use NTK based RoPE extrapolation to adjust the position embedding. + # Reference: (https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/) + # Design adapted from Cosmos2.5 (https://arxiv.org/pdf/2511.00062) + # extrapolation_ratio here is how the base of the RoPE is scaled + # b' = b * extrapolation_ratio^(dim / (dim - 2)) + rope_h_extrapolation_ratio: float = 1.0 + rope_w_extrapolation_ratio: float = 1.0 + rope_t_extrapolation_ratio: float = 1.0 + enable_fps_modulation: bool = False + base_fps: int = 24 + # For unified_3d_mrope: whether spatial (H, W) indices reset to 0 for each vision segment + unified_3d_mrope_reset_spatial_ids: bool = True + # Setting the temporal gap on the boundary of the different modalities, default is 0, using a value greater than 0 will add an additional offset on the accumulated temporal offset. + unified_3d_mrope_temporal_modality_margin: int = 0 + + +@attrs.define(slots=False) +class LBLConfig: + # For load balancing loss computation. + # - "local": Use the fraction of tokens routed to each expert only for the local rank. + # - "global": Use the fraction of tokens routed to each expert across all ranks. + method: str = "local" + + # Coefficients for the load balancing loss. + # - "und": Coefficient for the load balancing loss for the "und" pathway. + # - "gen": Coefficient for the load balancing loss for the "gen" pathway. + coeff_und: float | None = None + coeff_gen: float | None = None + + +@attrs.define(slots=False) +class RectifiedFlowTrainingConfig: + shift: Any = 5 # Training time shift. If dict, maps resolution (str) to shift value (int) + use_dynamic_shift: bool = False # Whether to use dynamic shifting + train_time_image_distribution: str = "logitnormal" # Training time distribution for images + train_time_video_distribution: str = "logitnormal" # Training time distribution for videos + train_time_action_distribution: str = "logitnormal" # Training time distribution for actions + train_time_sound_distribution: str = "logitnormal" # Training time distribution for sound + train_time_weight: str = "uniform" # Training time weight + loss_scale: float = 1.0 # Loss scale + image_loss_scale: float | None = None # If set, overrides loss_scale for images + sound_loss_scale: float | None = None # If set, overrides loss_scale for sound + use_high_sigma_strategy: bool = False # Whether to use high sigma strategy + high_sigma_ratio: float = 0.05 # Ratio of using high sigmas + high_sigma_timesteps_min: int = 995 # Minimum timestep for high sigma + high_sigma_timesteps_max: int = 1000 # Maximum timestep for high sigma + use_discrete_rf: bool = False # Whether to use discrete formulation of rectified flow + + # user: please adjust this value according to loss_scale to balance the action loss with the video loss. + # default is 10.0 to align with previous training settings. + action_loss_weight: float = 10.0 + + # Independent noise schedule for action. When False (default), action shares the sigma + # sampled from the vision RF on every step — legacy behavior. When True, action samples + # its own sigma from `rectified_flow_action` using `shift_action` and + # `use_high_sigma_strategy_action`. Action always uses a shared scalar sigma per sample + # ([B,1]), independent of vision's DF mode. If action opts in to the high-sigma strategy, + # it reuses the global ratio / min / max. + independent_action_schedule: bool = False + shift_action: int | None = None # must be int; None → inherit `shift` (which must also be int) + use_high_sigma_strategy_action: bool = False + + # Independent noise schedule for sound. When False (default), sound shares the vision + # sigma schedule, reindexed to the dense audio-bearing subset. When True, sound samples + # its own scalar sigma per sample ([B,1]) from `rectified_flow_sound` using `shift_sound` + # and `use_high_sigma_strategy_sound`. + independent_sound_schedule: bool = False + shift_sound: int | None = None # must be int; None → inherit `shift` (which must also be int) + use_high_sigma_strategy_sound: bool = False + + # When True, per-instance flow-matching loss is normalized by the count of + # active (noisy) elements rather than all elements — preserves sum/active_count + # semantics so conditioning-heavy samples (e.g. I2V, forward_dynamics, diffusion + # forcing, AR rollout teacher-forcing) contribute gradient on par with K=0 + # samples. With .mean() the gradient of a K-conditioned sample is scaled by + # (T-K)/T, which undertrains the attend-to-clean-history dynamics. Kept + # False by default to preserve legacy loss magnitudes; enable for AR/DF training. + normalize_loss_by_active: bool = False + + +@attrs.define(slots=False) +class RectifiedFlowInferenceConfig: + scheduler_type: str = "unipc" # Scheduler type + num_train_timesteps: int = 1000 + shift: int = 1 + use_dynamic_shifting: bool = False + + +@attrs.define(slots=False) +class FixedStepSamplerConfig: + """Config for the fixed-step sampler used by distilled models. + + Uses a fixed sigma schedule instead of a smooth multi-step solver. + + Mirrors the constructor args of ``FixedStepSampler``. + """ + + # Discrete noise-level schedule (descending, excluding the final 0.0 step). + # Convention: exclude the final 0.0 step — FixedStepSampler appends it automatically. + # Values must be descending. Using 0.999 instead of 1.0 avoids numeric edge cases at sigma=1. + t_list: list[float] = [0.999, 0.75, 0.5, 0.25] + # Integrator type: "ode" (deterministic Euler) or "sde" (stochastic re-noising at each step). + sample_type: str = "ode" + + +# Don't have any defaults and init only in config file. +@attrs.define(slots=False) +class OmniMoTModelConfig: + """ + Config for Omni MoT model. + """ + + tokenizer: LazyDict = None + net: LazyDict = None + ema: EMAConfig = EMAConfig() + + # Parallelism (CP, CFGP, FSDP, DP) and compilation configuration. + parallelism: ParallelismConfig = ParallelismConfig() + + # Activation-checkpointing policy (trade-off between memory and speed). + activation_checkpointing: ActivationCheckpointingConfig = ActivationCheckpointingConfig() + + # LoRA (parameter-efficient fine-tuning). When `lora_enabled=True`, + # `OmniMoTModel.build_net` injects custom LoRA adapters BEFORE FSDP wrap on + # the meta-device network, then re-initializes lora_A/lora_B after + # to_empty + init_weights. Pair with `optimizer.keys_to_select=["lora_"]` + # and `checkpoint.keys_to_skip_loading=[..., "lora_"]`. + lora_enabled: bool = False + lora_rank: int = 16 + lora_alpha: int = 32 + lora_target_modules: str = "q_proj_moe_gen,k_proj_moe_gen,v_proj_moe_gen,o_proj_moe_gen" + + # Rectified flow configs + rectified_flow_training_config: RectifiedFlowTrainingConfig = RectifiedFlowTrainingConfig() + rectified_flow_inference_config: RectifiedFlowInferenceConfig = RectifiedFlowInferenceConfig() + + # Optional fixed-step sampler for distilled models (None for base models). + fixed_step_sampler_config: FixedStepSamplerConfig | None = None + + # Model configs + vlm_config: VLMConfig = VLMConfig() + diffusion_expert_config: DiffusionExpertConfig = DiffusionExpertConfig() + # Training data keys + input_video_key: str = "video" + input_image_key: str = "images" # key to fetch input image from data_batch + input_caption_key: str = "ai_caption" # Key used to fetch input captions + + # State and sequence shapes + state_ch: int = 16 # for latent model, ref to the latent channel number + state_t: int = 8 # for latent model, ref to the latent number of frames + latent_downsample_factor: int = 8 + resolution: str = "512" + max_num_tokens_after_packing: int = 13312 # Final num tokens after sequence packing + + # Attention implementation for joint understanding + generation + # Note "two_way" and "three_way" disallow and remove "End-of-Vision" or other text token in the generation tower. + # "three_way" must only be used when introducing sparsity + joint_attn_implementation: str = ( + "two_way" # "two_way", "three_way" or "flex" (NOTICE: We are planning to remove "flex" soon) + ) + + # Per-layer NATTEN parameters + # Must use "three_way" attention if used. + # If None, all attention layers remain dense. + # If not None, must be a list exactly the size of number of layers, and each layer can be either + # None (dense) or a dictionary, with at least 'kernel_size' or 'kernel_size_float' keys + # specifying sparsity. NATTEN parameters 'dilation' and 'stride' may also be specified either as + # static integers, or as floating point values that will be mapped to their domain during + # runtime. Integer parameters should never be mixed with floating point ones. + # + # Floating point parameters are highly recommended, unless the use case will have a fixed token + # layout (input resolution). + # + # Examples: + # Interleaved sliding window layers, "GPT-OSS"-style, with static window size: + # natten_parameter_list = [None if layer_idx % 2 != 0 else {"kernel_size": (8, 8)}] + # Layers with odd indices ("None"s) will use dense attention, and layers with an even indices + # will use a static sliding window size of 8x8. + # + # Interleaved sliding window layers, "GPT-OSS"-style, with input-dependent window size: + # natten_parameter_list = [None if layer_idx % 2 != 0 else {"kernel_size_float": (0.5, 0.5)}] + # Layers with odd indices ("None"s) will use dense attention, and layers with an even indices + # will use a dynamic window size that is 50% of the input along each of the two dimensions. + # + # Interleaved sliding window and dilated layers, "DiNAT"-style: + # natten_parameter_list = [ + # { + # "kernel_size_float": (0.5, 0.5), + # "dilation_float": (1.0, 1.0), + # } if layer_idx % 2 != 0 else { + # "kernel_size_float": (0.5, 0.5), + # } + # ] + # All layers will use a dynamic window size that is 50% of the input along each of the two + # dimensions. Layers with odd indices will also dilate to the maximum level possible. + # + natten_parameter_list: list | None = None + + # Temporal causality for training autoregressive video generation models. + # When enabled, applies temporal causal attention to generation supertokens. + # Each supertoken is num_action_tokens_per_supertoken action tokens followed + # by H*W vision tokens; the value is stamped onto the packed sequence by the + # temporal-causal packer and read by attention/KV-cache code unchanged. + # Only supports image2video modes (with or without actions). + # Requires joint_attn_implementation="three_way". + video_temporal_causal: bool = False + # "none": standard joint denoising (shared σ, no clean context) + # "teacher_forcing": all frames noised with shared σ; clean history via cross-attention + # "diffusion_forcing": each latent frame gets independent σ ~ Uniform[0,1] + # "teacher_forcing_dcm": replayed teacher-forcing discrete-time consistency distillation + causal_training_strategy: str = attrs.field( + default="none", + validator=attrs.validators.in_({"none", "teacher_forcing", "diffusion_forcing", "teacher_forcing_dcm"}), + ) + + # Load balancing loss config. + lbl: LBLConfig = LBLConfig() + + # vision configs + vision_gen: bool = True # whether to use vision related parameters and condition/generate vision tokens + + # action configs + action_gen: bool = False # whether to use action related parameters and condition/generate action tokens + max_action_dim: int = 32 # maximum dimension of the action space, we need to pad the data to this dimension. + num_embodiment_domains: int = 32 # number of domains for the domain-aware linear layer + + # sound configs + sound_gen: bool = False # whether to use sound related parameters and condition/generate sound tokens + sound_tokenizer: LazyDict | None = None # Sound tokenizer config (e.g., AVAE) + sound_dim: int | None = None # Sound latent channel size (e.g., 64 for AVAE 48kHz) + sound_latent_fps: int = 25 # Sound tokenizer's latent rate (e.g., 48kHz / 1920 hop = 25 Hz) + + log_enc_time_every_n: int = 100 # Frequency of logging encoding time to W&B + + +@attrs.define(slots=False) +class VLMModelConfig: + """ + Config for VLM model. + """ + + policy: PolicyConfig = PolicyConfig() + train: TrainConfig = TrainConfig() + # Applied at model construction, before the optimizer is built. + freeze: VLMFreezeConfig = VLMFreezeConfig() + ema: EMAConfig = EMAConfig(enabled=False) diff --git a/cosmos_framework/configs/base/defaults/multiview_dataloader.py b/cosmos_framework/configs/base/defaults/multiview_dataloader.py new file mode 100644 index 0000000..a646ac6 --- /dev/null +++ b/cosmos_framework/configs/base/defaults/multiview_dataloader.py @@ -0,0 +1,150 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +""" +Hydra ConfigStore registration for multiview dataloaders. + +Registers named dataloader configs that can be referenced via Hydra overrides +(e.g. ``{override /data_train: video_control_mads_multiview_0823_gcs_720p_10fps_93frames_7views}``) +or used as templates for inline ``L(get_multiview_video_loader)(...)`` in +experiment configs. + +Two naming conventions: + + **Transfer** (with control signal): + ``video_control_{dataset}_{store}_{res}_{fps}_{frames}_{views}`` + + **Predict** (no control signal): + ``video_{dataset}_{store}_{res}_{fps}_{frames}_{views}`` +""" + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.data.vfm.multiview.multiview_data_source import ( + DEFAULT_CAMERAS, + INDEX_TO_CAMERA_MAPPING, + TRANSFER_CAPTION_KEY_MAPPING, + TRANSFER_CONTROL_KEY_MAPPING, + TRANSFER_VIDEO_KEY_MAPPING, +) +from cosmos_framework.data.vfm.multiview.multiview_dataset import ( + MultiviewAugmentationConfig, + get_multiview_video_loader, +) + +# --------------------------------------------------------------------------- +# Camera view subsets +# --------------------------------------------------------------------------- + +CAMERA_VIEW_CONFIGS: dict[str, tuple[str, ...]] = { + "7views": DEFAULT_CAMERAS, + "1view_front": ("camera_front_wide_120fov",), + "4views": ( + "camera_front_wide_120fov", + "camera_cross_right_120fov", + "camera_rear_tele_30fov", + "camera_cross_left_120fov", + ), +} + +# --------------------------------------------------------------------------- +# Grid dimensions +# --------------------------------------------------------------------------- + +_TRANSFER_DATASETS = ["mads_multiview_0823"] +_OBJECT_STORES = ["gcs"] + +_RESOLUTIONS: list[tuple[str, tuple[int, int]]] = [ + ("720p", (720, 1280)), +] + +_FPS: list[tuple[str, int]] = [ + ("10fps", 1), # MADS transfer data is already at 10 fps +] + +_NUM_VIDEO_FRAMES: list[tuple[str, int]] = [ + ("29frames", 29), + ("61frames", 61), + ("93frames", 93), +] + + +def register_multiview_dataloaders() -> None: + """Register all multiview dataloader configs with Hydra ConfigStore.""" + + cs = ConfigStore.instance() + + # ----- Transfer dataloaders (with control signals) ----- + for dataset in _TRANSFER_DATASETS: + for object_store in _OBJECT_STORES: + for resolution_str, resolution_hw in _RESOLUTIONS: + for fps_str, downsample_factor in _FPS: + for num_frames_str, num_frames in _NUM_VIDEO_FRAMES: + for views_str, camera_keys in CAMERA_VIEW_CONFIGS.items(): + name = ( + f"video_control_{dataset}_{object_store}_{resolution_str}_" + f"{fps_str}_{num_frames_str}_{views_str}" + ) + cs.store( + group="data_train", + package="dataloader_train", + name=name, + node=L(get_multiview_video_loader)( + dataset_name=dataset, + is_train=True, + augmentation_config=L(MultiviewAugmentationConfig)( + resolution_hw=resolution_hw, + fps_downsample_factor=downsample_factor, + num_video_frames=num_frames, + camera_keys=camera_keys, + camera_video_key_mapping=TRANSFER_VIDEO_KEY_MAPPING, + camera_caption_key_mapping=TRANSFER_CAPTION_KEY_MAPPING, + camera_control_key_mapping=TRANSFER_CONTROL_KEY_MAPPING, + position_to_camera_mapping=INDEX_TO_CAMERA_MAPPING, + single_caption_camera_name="camera_front_wide_120fov", + ), + ), + ) + + # ----- Predict dataloaders (no control signals, for future use) ----- + # These use named keys (video_camera_front_wide_120fov, etc.) and need + # different datasets (e.g. alpamayo_dec2024) with 30 fps native data. + # Uncomment and add predict datasets to the catalog when needed. + # + # _PREDICT_DATASETS = ["alpamayo_dec2024"] + # _PREDICT_FPS = [("10fps", 3), ("15fps", 2)] # 30 fps native → downsample + # for dataset in _PREDICT_DATASETS: + # for object_store in _OBJECT_STORES: + # for resolution_str, resolution_hw in _RESOLUTIONS: + # for fps_str, downsample_factor in _PREDICT_FPS: + # for num_frames_str, num_frames in _NUM_VIDEO_FRAMES: + # for views_str, camera_keys in CAMERA_VIEW_CONFIGS.items(): + # name = ( + # f"video_{dataset}_{object_store}_{resolution_str}_" + # f"{fps_str}_{num_frames_str}_{views_str}" + # ) + # cs.store( + # group="data_train", + # package="dataloader_train", + # name=name, + # node=L(get_multiview_video_loader)( + # dataset_name=dataset, + # is_train=True, + # augmentation_config=L(MultiviewAugmentationConfig)( + # resolution_hw=resolution_hw, + # fps_downsample_factor=downsample_factor, + # num_video_frames=num_frames, + # camera_keys=camera_keys, + # camera_video_key_mapping=PREDICT_VIDEO_KEY_MAPPING, + # camera_caption_key_mapping=PREDICT_CAPTION_KEY_MAPPING, + # camera_control_key_mapping=None, + # position_to_camera_mapping=None, + # single_caption_camera_name=None, + # ), + # ), + # ) + + +# Auto-register on import +register_multiview_dataloaders() diff --git a/cosmos_framework/configs/base/defaults/open_source_dataloader.py b/cosmos_framework/configs/base/defaults/open_source_dataloader.py new file mode 100644 index 0000000..dc84e8f --- /dev/null +++ b/cosmos_framework/configs/base/defaults/open_source_dataloader.py @@ -0,0 +1,197 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +""" +Hydra ConfigStore registration for the open-source SFT dataloader. + +This mirrors the inline ``dataloader_train`` block in +``configs/experiment/mixed_modality_sft_8b.yaml`` (cosmos-inference) so users +can pick it up via the Hydra defaults group:: + + defaults: + - data_train: open_source_sft_video_256p + +or as a base to override in their own experiment configs:: + + L(get_open_source_sft_dataloader)( + jsonl_paths=["/path/to/data.jsonl"], + resolution="256", + max_sequence_length=45056, + ) + +Original YAML reference target paths use the ``cosmos3._src.vfm.*`` namespace +(the OSS-release form of ``projects.cosmos3.vfm.*``); inside this released +tree the same modules live under ``cosmos_framework.data.vfm.*``. +""" + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.data.vfm.joint_dataloader import ( + PackingDataLoader, + RankPartitionedDataLoader, +) +from cosmos_framework.data.vfm.local_datasets.sft_dataset import get_sft_dataset +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.defaults.vlm import create_qwen2_tokenizer_with_download + + +# --------------------------------------------------------------------------- +# Inner: SFT video dataset (matches the inline ``get_sft_dataset`` call in the +# reference YAML). +# --------------------------------------------------------------------------- + +def get_sft_video_dataset( + *, + jsonl_paths: list[str], + resolution: str = "256", + pretrained_model_name: str = "Qwen/Qwen3-VL-8B-Instruct", + tokenizer_config_variant: str = "hf", + num_video_frames: int = -1, + temporal_compression_factor: int = 4, + temporal_interval_mode: str = "max_30fps", + min_short_edge: int = 0, + frame_selection_mode: str = "first", + sample_by_window: bool = False, + append_duration_fps_timestamps: bool = True, + append_resolution_info: bool = True, + use_system_prompt: bool = False, + caption_suffix: str = "", + cfg_dropout_rate: float = 0.1, + cfg_dropout_keep_metadata: bool = False, + conditioning_config: dict[int, float] | None = None, + conditioning_fps: int = -1, + conditioning_fps_noise_std: float = 0.0, +): + """LazyCall'd version of ``get_sft_dataset`` matching the reference YAML. + + Defaults reproduce ``mixed_modality_sft_8b.yaml``: 70% T2V / 20% I2V / + 10% V2V conditioning mix at 256p with the Qwen3-VL-8B tokenizer. + """ + if conditioning_config is None: + # 0: T2V (text-to-video) — 70% + # 1: I2V (first-frame conditioning) — 20% + # 2: V2V (first 5 frames → 2 latent frames) — 10% + conditioning_config = {0: 0.7, 1: 0.2, 2: 0.1} + + return L(get_sft_dataset)( + jsonl_paths=jsonl_paths, + resolution=resolution, + num_video_frames=num_video_frames, + temporal_compression_factor=temporal_compression_factor, + temporal_interval_mode=temporal_interval_mode, + min_short_edge=min_short_edge, + frame_selection_mode=frame_selection_mode, + sample_by_window=sample_by_window, + append_duration_fps_timestamps=append_duration_fps_timestamps, + append_resolution_info=append_resolution_info, + use_system_prompt=use_system_prompt, + caption_suffix=caption_suffix, + cfg_dropout_rate=cfg_dropout_rate, + cfg_dropout_keep_metadata=cfg_dropout_keep_metadata, + conditioning_config=conditioning_config, + conditioning_fps=conditioning_fps, + conditioning_fps_noise_std=conditioning_fps_noise_std, + tokenizer_config=L(create_qwen2_tokenizer_with_download)( + pretrained_model_name=pretrained_model_name, + config_variant=tokenizer_config_variant, + ), + ) + + +# --------------------------------------------------------------------------- +# Outer: full PackingDataLoader → RankPartitionedDataLoader → SFT dataset +# pipeline. This is the registered config_store node. +# --------------------------------------------------------------------------- + +def get_open_source_sft_dataloader( + *, + jsonl_paths: list[str] | None = None, + resolution: str = "256", + batch_size: int = 1, + max_sequence_length: int = 45056, + max_samples_per_batch: int | None = None, + num_workers: int = 4, + prefetch_factor: int = 4, + audio_sample_rate: int = 48000, + patch_spatial: int = 2, + tokenizer_spatial_compression_factor: int = 16, + tokenizer_temporal_compression_factor: int = 4, + sound_latent_fps: int = 0, + dataset_name: str = "default", + video_stream_ratio: float = 1.0, +): + """Build the full open-source SFT dataloader (PackingDataLoader at top). + + ``jsonl_paths`` defaults to a Hydra-MISSING marker (``"???"``) so the + user MUST override it at experiment time:: + + ... dataloader_train.dataloader.datasets.video.dataset.jsonl_paths='[".../data.jsonl"]' + """ + if jsonl_paths is None: + # Hydra/OmegaConf "mandatory" sentinel — must be overridden. + jsonl_paths = "???" # type: ignore[assignment] + + return L(PackingDataLoader)( + dataloader=L(RankPartitionedDataLoader)( + batch_size=batch_size, + datasets=dict( + video=dict( + dataset=get_sft_video_dataset( + jsonl_paths=jsonl_paths, + resolution=resolution, + ), + ratio=video_stream_ratio, + ), + ), + in_order=True, + num_workers=num_workers, + persistent_workers=True, + pin_memory=True, + prefetch_factor=prefetch_factor, + sampler=None, + ), + audio_sample_rate=audio_sample_rate, + dataset_name=dataset_name, + max_samples_per_batch=max_samples_per_batch, + max_sequence_length=max_sequence_length, + patch_spatial=patch_spatial, + sound_latent_fps=sound_latent_fps, + tokenizer_spatial_compression_factor=tokenizer_spatial_compression_factor, + tokenizer_temporal_compression_factor=tokenizer_temporal_compression_factor, + ) + + +# --------------------------------------------------------------------------- +# ConfigStore registration. +# --------------------------------------------------------------------------- + +def register_open_source_dataloaders() -> None: + """Register named dataloader configs under the ``data_train`` Hydra group. + + Pick them via experiment config:: + + defaults: + - data_train: open_source_sft_video_256p + - data_train: open_source_sft_video_480p + - data_train: open_source_sft_video_720p + """ + cs = ConfigStore.instance() + + for res_str, max_seq in [ + ("256", 45056), + ("480", 45056), + ("720", 45056), + ]: + cs.store( + group="data_train", + package="dataloader_train", + name=f"open_source_sft_video_{res_str}p", + node=get_open_source_sft_dataloader( + resolution=res_str, + max_sequence_length=max_seq, + ), + ) + + +# Auto-register on import. +register_open_source_dataloaders() diff --git a/cosmos_framework/configs/base/defaults/optimizer.py b/cosmos_framework/configs/base/defaults/optimizer.py new file mode 100644 index 0000000..245a5df --- /dev/null +++ b/cosmos_framework/configs/base/defaults/optimizer.py @@ -0,0 +1,105 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Canonical Hydra-group registry for the optimizer and scheduler SKUs.""" + +from typing import Any + +from cosmos_framework.utils.lazy_config import PLACEHOLDER +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils.config_helper import ConfigStore +from cosmos_framework.utils.vfm.optimizer import build_lr_scheduler, build_optimizer + +OPTIMIZER_KWARGS: dict[str, Any] = dict( + # Learning rate for the optimizer. + lr=1e-4, + # Weight decay for the optimizer. + weight_decay=0.1, + # Beta1 and beta2 for the optimizer. + betas=[0.9, 0.99], + # Epsilon for the optimizer. + eps=1e-8, + # Whether to use fuse updates to all parameters. + fused=True, + # Keys to select for the optimizer. + keys_to_select=[], + # Per-key LR multipliers. Maps parameter name patterns to LR multipliers. + # E.g. {"sound2llm": 5.0, "llm2sound": 5.0} gives those params 5x the base LR. + lr_multipliers={}, + # Whether to disable weight decay for one-dimensional params such as norm weights and biases. + # Default is False to preserve historical optimizer behavior. + disable_weight_decay_for_1d_params=False, +) + +LAMBDACOSINE_KWARGS: dict[str, Any] = dict( + warm_up_steps=[2000], + cycle_lengths=[100000], + f_start=[0.0], + f_max=[1.0], + f_min=[0.0], + verbosity_interval=0, +) + + +def register_optimizers(optimizer_kwargs: dict[str, Any]) -> None: + """Register the ``fusedadamw`` and ``adamw`` SKUs.""" + cs = ConfigStore.instance() + cs.store( + group="optimizer", + package="optimizer", + name="fusedadamw", + node=L(build_optimizer)( + model=PLACEHOLDER, + optimizer_type="FusedAdam", + **optimizer_kwargs, + ), + ) + cs.store( + group="optimizer", + package="optimizer", + name="adamw", + node=L(build_optimizer)( + model=PLACEHOLDER, + optimizer_type="AdamW", + **optimizer_kwargs, + ), + ) + + +def register_schedulers(lambdacosine_kwargs: dict[str, Any]) -> None: + """Register the ``lambdalinear`` and ``lambdacosine`` SKUs.""" + cs = ConfigStore.instance() + cs.store( + group="scheduler", + package="scheduler", + name="lambdalinear", + node=L(build_lr_scheduler)( + optimizer=PLACEHOLDER, + lr_scheduler_type="LambdaLinear", + warm_up_steps=[1000], + cycle_lengths=[10000000000000], + f_start=[1.0e-6], + f_max=[1.0], + f_min=[1.0], + ), + ) + cs.store( + group="scheduler", + package="scheduler", + name="lambdacosine", + node=L(build_lr_scheduler)( + optimizer=PLACEHOLDER, + lr_scheduler_type="LambdaCosine", + **lambdacosine_kwargs, + ), + ) + + +def register_optimizer() -> None: + """VFM project-root entry point.""" + register_optimizers(OPTIMIZER_KWARGS) + + +def register_scheduler() -> None: + """VFM project-root entry point.""" + register_schedulers(LAMBDACOSINE_KWARGS) diff --git a/cosmos_framework/configs/base/defaults/parallelism.py b/cosmos_framework/configs/base/defaults/parallelism.py new file mode 100644 index 0000000..c46c47d --- /dev/null +++ b/cosmos_framework/configs/base/defaults/parallelism.py @@ -0,0 +1,69 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Shared user-facing parallelism schema for VFM and VLM. + +Both project trees (vfm/, vfm/configs/base/vlm/) instantiate the same +ParallelDims runtime at vfm/utils/parallelism.py. They now also share this +single user-facing config schema. Trainer-side translation from the long +descriptive field names here to the short ParallelDims constructor kwargs +happens at the read site (see vfm/models/{omni_mot_model,vlm_model}.py). +""" + +import attrs + + +@attrs.define(slots=False) +class ParallelismConfig: + # Torch compile is used to compile the model for faster training. + use_torch_compile: bool = False + + # Whether to use CUDA graphs for faster inference. This option does not work during training. + use_cuda_graphs: bool = False + + # Whether the entire Cosmos3 VFM network is compiled, or only a specific region is compiled. + # Use "language" to compile only individual layers in the MOT model. + # Use "all" to compile the the MOT model, as well as encode/decode functions. + compiled_region: str = attrs.field( + default="language", + validator=attrs.validators.in_({"all", "language"}), + ) + + # Whether torch.compile should generate symbolic-shape (dynamic) kernels + # (maps to ``torch.compile(dynamic=...)``). Defaults to True for training, + # which sees varying shapes across batches (sequence length, CP sharding, ...); + # specializing would recompile continuously. See ParallelismOverrides in + # cosmos_framework/inference/common/args.py for the inference-side rationale + # (where dynamic=False is preferred for stable AR shapes). + compile_dynamic: bool = True + + # Enable autotuning for pointwise/reduction Triton kernels (e.g. RMSNorm). + # Explores 6 candidate configs instead of the default 1, improving kernel performance + # at the cost of longer first-iteration compilation time. + max_autotune_pointwise: bool = False + + # Enable coordinate descent tuning after autotuning. Starts from the best autotuned + # config and explores nearby configs by adjusting one parameter at a time. + # Requires max_autotune_pointwise=True to have effect on reduction kernels. + coordinate_descent_tuning: bool = False + + # Whether to enable inference mode. + enable_inference_mode: bool = False + + # Number of ranks for sharding the model weights (FSDP). The default -1 + # auto-infers to world_size at runtime via ParallelDims. + data_parallel_shard_degree: int = -1 + + # Number of ranks for replicating the model weights (HSDP outer dim). + # data_parallel_replicate_degree x data_parallel_shard_degree must divide + # world_size when both are explicitly set. + data_parallel_replicate_degree: int = 1 + + # Number of ranks for context parallelism. + context_parallel_shard_degree: int = 1 + + # Number of ranks for CFG parallelism. + cfg_parallel_shard_degree: int = 1 + + # Precision for the model. + precision: str = "bfloat16" diff --git a/cosmos_framework/configs/base/defaults/tokenizer.py b/cosmos_framework/configs/base/defaults/tokenizer.py new file mode 100644 index 0000000..55cb01c --- /dev/null +++ b/cosmos_framework/configs/base/defaults/tokenizer.py @@ -0,0 +1,222 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import PLACEHOLDER, LazyDict +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.model.vfm.tokenizers.audio.avae import AVAEInterface +from cosmos_framework.model.vfm.tokenizers.dc_ae.dc_ae_4x32x32 import DCAE4x32x32Interface +from cosmos_framework.model.vfm.tokenizers.flux_vae_8x8 import FluxVAEInterface +from cosmos_framework.model.vfm.tokenizers.uniae.noncausal_4x16x16 import UniAEVAEInterface +from cosmos_framework.model.vfm.tokenizers.wan2pt1_vae_4x8x8 import Wan2pt1VAEInterface +from cosmos_framework.model.vfm.tokenizers.wan2pt2_vae_4x16x16 import Wan2pt2VAEInterface + +PRETRAINED_TOKENIZER_WAN2PT1_VAE_PTH = "pretrained/tokenizers/video/wan2pt1/Wan2.1_VAE.pth" +PRETRAINED_TOKENIZER_WAN2PT2_VAE_PTH = "pretrained/tokenizers/video/wan2pt2/Wan2.2_VAE.pth" +PRETRAINED_TOKENIZER_FLUX_VAE_PTH = "pretrained/tokenizers/image/flux/ae.safetensors" + +# UniAE checkpoint paths +PRETRAINED_TOKENIZER_UNIAE_4X16X16_C48_T8TO24_64TO512P_FPS_ALL_ENCODER_NONCAUSAL_DECODER_NONCAUSAL_NOGAN_BEST_S1_VAE_PTH = "pretrained/tokenizers/video/cosmos/uniae4x16x16_c48_t8to24_64to512p_fps_all_encoder_noncausal_decoder_noncausal_nogan_best_s1.pt" + +# DCAE checkpoint paths +PRETRAINED_TOKENIZER_DCAE_PTH = "pretrained/tokenizers/video/cosmos/dc-ae-v-1.0-f32t4c64-cosmos-encoder-causal-decoder-chunk-causal-4-frame-120-pad-7-no-gan.pt" +PRETRAINED_TOKENIZER_DCAE_4X32X32_C64_T120_256P_FPS_ALL_ENCODER_CAUSAL_DECODER_CHUNKCAUSAL4_NOGAN_COSMOS_PAD_7_V0PT2_PTH = "pretrained/tokenizers/video/cosmos/dcae4x32x32_c64_t120_256p_fps_all_encoder_causal_decoder_chunk_causal_4_nogan_cosmos_pad_7_v0.2.pt" +PRETRAINED_TOKENIZER_DCAE_4X32X32_C96_T120_256P_FPS_ALL_ENCODER_CAUSAL_DECODER_CHUNKCAUSAL4_NOGAN_COSMOS_PAD_7_V0PT2_PTH = "pretrained/tokenizers/video/cosmos/dcae4x32x32_c96_t120_256p_fps_all_encoder_causal_decoder_chunk_causal_4_nogan_cosmos_pad_7_v0.2.pt" +PRETRAINED_TOKENIZER_DCAE_4X32X32_C128_T120_256P_FPS_ALL_ENCODER_CAUSAL_DECODER_CHUNKCAUSAL4_NOGAN_COSMOS_PAD_7_V0PT2_PTH = "pretrained/tokenizers/video/cosmos/dcae4x32x32_c128_t120_256p_fps_all_encoder_causal_decoder_chunk_causal_4_nogan_cosmos_pad_7_v0.2.pt" + +# AVAE (Audio VAE) checkpoint paths +PRETRAINED_TOKENIZER_AVAE_PTH = "pretrained/tokenizers/audio/avae/model_unwrap.ckpt" +PRETRAINED_TOKENIZER_AVAE_44K_NONCAUSAL = "pretrained/tokenizers/audio/avae/avae_44k_noncausal_21hz_64ch.ckpt" +PRETRAINED_TOKENIZER_AVAE_44K_CAUSAL = "pretrained/tokenizers/audio/avae/avae_44k_causal_21hz_64ch.ckpt" +PRETRAINED_TOKENIZER_AVAE_48K_25HZ = "pretrained/tokenizers/audio/avae/avae_48k_noncausal_25hz_64ch.ckpt" +PRETRAINED_TOKENIZER_AVAE_48K_6HZ = "pretrained/tokenizers/audio/avae/avae_48k_noncausal_6hz_64ch.ckpt" + + +# Flux tokenizer config +FluxVAEConfig: LazyDict = L(FluxVAEInterface)( + # This is the flux image tokenizer. + # We use it for bagel inference. + # We do not use it for Cosmos3. + bucket_name=PLACEHOLDER, + object_store_credential_path_pretrained=PLACEHOLDER, + vae_path=PRETRAINED_TOKENIZER_FLUX_VAE_PTH, + chunk_duration=1, + spatial_compression_factor=8, + temporal_compression_factor=1, +) + +Wan2pt1VAEConfig: LazyDict = L(Wan2pt1VAEInterface)( + # 4x8x8 tokenizer + bucket_name=PLACEHOLDER, + object_store_credential_path_pretrained=PLACEHOLDER, + vae_path=PRETRAINED_TOKENIZER_WAN2PT1_VAE_PTH, + spatial_compression_factor=8, + temporal_compression_factor=4, +) + +Wan2pt2VAEConfig: LazyDict = L(Wan2pt2VAEInterface)( + bucket_name=PLACEHOLDER, + object_store_credential_path_pretrained=PLACEHOLDER, + vae_path=PRETRAINED_TOKENIZER_WAN2PT2_VAE_PTH, + spatial_compression_factor=16, + temporal_compression_factor=4, +) + +DCAE4x32x32Config: LazyDict = L(DCAE4x32x32Interface)( + bucket_name=PLACEHOLDER, + object_store_credential_path_pretrained=PLACEHOLDER, + vae_path=PRETRAINED_TOKENIZER_DCAE_PTH, + spatial_compression_factor=32, + temporal_compression_factor=4, +) + +DCAE4x32x32C64T120_256pFpsAllEncoderCausalDecoderChunkCausal4NoganCosmosPad7V0pt2Config: LazyDict = L( + DCAE4x32x32Interface +)( + bucket_name=PLACEHOLDER, + object_store_credential_path_pretrained=PLACEHOLDER, + vae_path=PRETRAINED_TOKENIZER_DCAE_4X32X32_C64_T120_256P_FPS_ALL_ENCODER_CAUSAL_DECODER_CHUNKCAUSAL4_NOGAN_COSMOS_PAD_7_V0PT2_PTH, + model_name="dcae4x32x32_c64_t120_256p_fps_all_encoder_causal_decoder_chunk_causal_4_nogan_cosmos_pad_7_v0.2", + spatial_compression_factor=32, + temporal_compression_factor=4, +) + +DCAE4x32x32C96T120_256pFpsAllEncoderCausalDecoderChunkCausal4NoganCosmosPad7V0pt2Config: LazyDict = L( + DCAE4x32x32Interface +)( + bucket_name=PLACEHOLDER, + object_store_credential_path_pretrained=PLACEHOLDER, + vae_path=PRETRAINED_TOKENIZER_DCAE_4X32X32_C96_T120_256P_FPS_ALL_ENCODER_CAUSAL_DECODER_CHUNKCAUSAL4_NOGAN_COSMOS_PAD_7_V0PT2_PTH, + model_name="dcae4x32x32_c96_t120_256p_fps_all_encoder_causal_decoder_chunk_causal_4_nogan_cosmos_pad_7_v0.2", + spatial_compression_factor=32, + temporal_compression_factor=4, +) + +DCAE4x32x32C128T120_256pFpsAllEncoderCausalDecoderChunkCausal4NoganCosmosPad7V0pt2Config: LazyDict = L( + DCAE4x32x32Interface +)( + bucket_name=PLACEHOLDER, + object_store_credential_path_pretrained=PLACEHOLDER, + vae_path=PRETRAINED_TOKENIZER_DCAE_4X32X32_C128_T120_256P_FPS_ALL_ENCODER_CAUSAL_DECODER_CHUNKCAUSAL4_NOGAN_COSMOS_PAD_7_V0PT2_PTH, + model_name="dcae4x32x32_c128_t120_256p_fps_all_encoder_causal_decoder_chunk_causal_4_nogan_cosmos_pad_7_v0.2", + spatial_compression_factor=32, + temporal_compression_factor=4, +) + +UniAE4x16x16C48T8to24_64to512pFpsAllEncoderNoncausalDecoderNoncausalNoganBestS1Config: LazyDict = L(UniAEVAEInterface)( + bucket_name=PLACEHOLDER, + object_store_credential_path_pretrained=PLACEHOLDER, + vae_path=PRETRAINED_TOKENIZER_UNIAE_4X16X16_C48_T8TO24_64TO512P_FPS_ALL_ENCODER_NONCAUSAL_DECODER_NONCAUSAL_NOGAN_BEST_S1_VAE_PTH, + spatial_compression_factor=16, + temporal_compression_factor=4, +) + +# ============================================================================= +# AVAE (Audio VAE) Tokenizer Configs +# ============================================================================= + +# Legacy config with tanh companding (non-commercial use only) +# Latent rate: 44100 / 2048 = 21.53Hz +AVAETokenizerConfig: LazyDict = L(AVAEInterface)( + bucket_name=PLACEHOLDER, + object_store_credential_path_pretrained=PLACEHOLDER, + avae_path=PRETRAINED_TOKENIZER_AVAE_PTH, + sample_rate=44100, + audio_channels=2, + io_channels=64, + hop_size=2048, + normalization_type="tanh", + tanh_input_scale=1.0, + tanh_output_scale=3.0, +) + + +# 44.1kHz Non-causal (PRIMARY - used for V2A/T2A training) +# Latent rate: 44100 / 2048 = 21.53Hz +AVAE_44k_NoncausalConfig: LazyDict = L(AVAEInterface)( + bucket_name=PLACEHOLDER, + object_store_credential_path_pretrained=PLACEHOLDER, + avae_path=PRETRAINED_TOKENIZER_AVAE_44K_NONCAUSAL, + sample_rate=44100, + audio_channels=2, + io_channels=64, + hop_size=2048, + normalize_latents=True, + tanh_input_scale=1.5, + tanh_output_scale=3.5, +) + +# 48kHz 25Hz (higher quality audio) +# Latent rate: 48000 / 1920 = 25Hz +AVAE_48k_25hzConfig: LazyDict = L(AVAEInterface)( + bucket_name=PLACEHOLDER, + object_store_credential_path_pretrained=PLACEHOLDER, + avae_path=PRETRAINED_TOKENIZER_AVAE_48K_25HZ, + sample_rate=48000, + audio_channels=2, + io_channels=64, + hop_size=1920, + normalize_latents=True, + tanh_input_scale=1.5, + tanh_output_scale=3.5, +) + + +def register_tokenizer(): + cs = ConfigStore.instance() + + # Wan2pt1 and Wan2pt2 tokenizers + cs.store(group="tokenizer", package="model.config.tokenizer", name="wan2pt1_tokenizer", node=Wan2pt1VAEConfig) + cs.store(group="tokenizer", package="model.config.tokenizer", name="wan2pt2_tokenizer", node=Wan2pt2VAEConfig) + # UniAE tokenizer + cs.store( + group="tokenizer", + package="model.config.tokenizer", + name="uniae_4x16x16_c48_t8to24_64to512p_fps_all_encoder_noncausal_decoder_noncausal_nogan_best_s1_tokenizer", + node=UniAE4x16x16C48T8to24_64to512pFpsAllEncoderNoncausalDecoderNoncausalNoganBestS1Config, + ) + # Flux tokenizer + cs.store(group="tokenizer", package="model.config.tokenizer", name="flux_tokenizer", node=FluxVAEConfig) + # DC AE 4x32x32 tokenizer + cs.store( + group="tokenizer", + package="model.config.tokenizer", + name="dc_ae_4x32x32_tokenizer", + node=DCAE4x32x32Config, + ) + cs.store( + group="tokenizer", + package="model.config.tokenizer", + name="dc_ae_4x32x32_c64_t120_256p_fps_all_encoder_causal_decoder_chunk_causal_4_nogan_cosmos_pad_7_v0.2_tokenizer", + node=DCAE4x32x32C64T120_256pFpsAllEncoderCausalDecoderChunkCausal4NoganCosmosPad7V0pt2Config, + ) + cs.store( + group="tokenizer", + package="model.config.tokenizer", + name="dc_ae_4x32x32_c96_t120_256p_fps_all_encoder_causal_decoder_chunk_causal_4_nogan_cosmos_pad_7_v0.2_tokenizer", + node=DCAE4x32x32C96T120_256pFpsAllEncoderCausalDecoderChunkCausal4NoganCosmosPad7V0pt2Config, + ) + cs.store( + group="tokenizer", + package="model.config.tokenizer", + name="dc_ae_4x32x32_c128_t120_256p_fps_all_encoder_causal_decoder_chunk_causal_4_nogan_cosmos_pad_7_v0.2_tokenizer", + node=DCAE4x32x32C128T120_256pFpsAllEncoderCausalDecoderChunkCausal4NoganCosmosPad7V0pt2Config, + ) + + +def register_sound_tokenizer(): + """Register sound tokenizers in Hydra ConfigStore under model.config.sound_tokenizer.""" + cs = ConfigStore.instance() + cs.store( + group="sound_tokenizer", package="model.config.sound_tokenizer", name="avae_48k_25hz", node=AVAE_48k_25hzConfig + ) + cs.store( + group="sound_tokenizer", + package="model.config.sound_tokenizer", + name="avae_44k_noncausal", + node=AVAE_44k_NoncausalConfig, + ) + cs.store( + group="sound_tokenizer", package="model.config.sound_tokenizer", name="avae_tokenizer", node=AVAETokenizerConfig + ) diff --git a/cosmos_framework/configs/base/defaults/unittest.py b/cosmos_framework/configs/base/defaults/unittest.py new file mode 100644 index 0000000..fdd3b0e --- /dev/null +++ b/cosmos_framework/configs/base/defaults/unittest.py @@ -0,0 +1,31 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 +import attrs + +# from cosmos_framework.configs.base.defaults.cluster import GCPIADGB200Config + +# We are hardcoding the unittest assets in this file. + +# CLUSTER_CONFIG = GCPIADGB200Config + +# add codeowner for cosmos_framework/model/vfm/tokenizers + + +@attrs.define(slots=False) +class SwfitStackPDXrConfig: + """ + Config for the cluster specific information. + Everything cluster specific should be here. + """ + + object_store_bucket_data: str + object_store_credential_data: str + + +UNITTEST_CONFIG = SwfitStackPDXrConfig( + object_store_bucket_data="unittest", + object_store_credential_data="credentials/pdx_dir.secret", +) + +TOKENIZER_RECONSTRUCTION_VIDEO_PATH = "tokenizer/video/panda70m_test_0000039_00000.mp4" +AVAE_RECONSTRUCTION_AUDIO_PATH = "tokenizer/audio/test_audio.wav" diff --git a/cosmos_framework/configs/base/defaults/vlm.py b/cosmos_framework/configs/base/defaults/vlm.py new file mode 100644 index 0000000..ec690e8 --- /dev/null +++ b/cosmos_framework/configs/base/defaults/vlm.py @@ -0,0 +1,924 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# Configs for VLM / LLM models + +import os +from typing import Any + +import attrs +import torch.distributed as dist + +from cosmos_framework.utils.flags import INTERNAL +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils.lazy_config import LazyDict +from cosmos_framework.utils.lazy_config import instantiate as lazy_instantiate +from cosmos_framework.utils import log +from cosmos_framework.utils.config_helper import ConfigStore +from cosmos_framework.utils.easy_io import easy_io +from cosmos_framework.model.vfm.mot.unified_mot import ( + Nemotron3DenseVLMoTConfig, + Nemotron3DenseVLTextForCausalLM, + Qwen3MoTConfig, + Qwen3VLMoeMoTConfig, + Qwen3VLMoeTextForCausalLM, + Qwen3VLMoTConfig, + Qwen3VLTextForCausalLM, +) +from cosmos_framework.data.vfm.processors import LLMTokenizerProcessor, build_processor_lazy +from cosmos_framework.model.vfm.tokenizers.tokenization_qwen2 import Qwen2Tokenizer + + +def create_vlm_config(base_config: LazyDict, **overrides): + vlm_config = lazy_instantiate(base_config) + for key, value in overrides.items(): + setattr(vlm_config, key, value) + return vlm_config + + +def get_rank_safe() -> int: + if dist.is_available() and dist.is_initialized(): + return dist.get_rank() + return 0 # default to rank 0 when not in distributed mode + + +################################################################################ +# Download tokenizer files from s3 +# Download to ~/.cache/imaginaire4/tokenizer_files/{model_name} and then load from there. +def download_tokenizer_files(model_name: str, config_variant: str) -> str: + if config_variant == "hf": + return model_name + + if config_variant == "s3": + ckpt_bucket = "checkpoints-us-east-1" + credentials = "credentials/s3_checkpoint.secret" + elif config_variant == "gcp": + ckpt_bucket = "nv-00-10206-checkpoint" + credentials = "credentials/gcp_checkpoint.secret" + else: + raise ValueError(f"Invalid config variant: {config_variant}") + + model_path = f"s3://{ckpt_bucket}/cosmos3/pretrained/huggingface/{model_name}" + if not INTERNAL: + from cosmos_framework.utils.checkpoint_db import download_checkpoint_v2 + + model_path = download_checkpoint_v2(model_path) + if "://" not in model_path: + return model_path + + imaginaire_cache_dir = os.environ.get("IMAGINAIRE_CACHE_DIR", os.path.expanduser("~/.cache/imaginaire4")) + destination_dir = os.path.join(imaginaire_cache_dir, f"tokenizer_files/{model_name}/rank_{get_rank_safe()}") + s3_backend_args = { + "backend": "s3", + "s3_credential_path": credentials, + } + + extensions = ["json", "txt", "jinja"] + for extension in extensions: + for file_path in easy_io.list_dir_or_file( + model_path, + list_dir=False, + list_file=True, + suffix=extension, + recursive=False, + backend_args=s3_backend_args, + ): + full_path = easy_io.join_path(model_path, file_path, backend_args=s3_backend_args) + local_path = f"{destination_dir}/{file_path}" + if os.path.exists(local_path): + log.debug(f"Skipping already downloaded tokenizer file: {local_path}") + continue + log.info(f"Downloading tokenizer file: {full_path} to {local_path}, cwd: {os.getcwd()}") + # Download the file + file_data = easy_io.get(full_path, backend_args=s3_backend_args) + easy_io.put(file_data, local_path) + return destination_dir + + +def create_qwen2_tokenizer_with_download(pretrained_model_name: str, config_variant: str, **_unused_kwargs): + # **_unused_kwargs absorbs extras (e.g. tokenizer_type) that OmegaConf + # merges in from a vlm_config preset's tokenizer block when an experiment + # overrides the tokenizer with this function but doesn't fully replace + # the preset's kwarg dict. + destination_dir = download_tokenizer_files(pretrained_model_name, config_variant) + return LLMTokenizerProcessor(Qwen2Tokenizer.from_pretrained(destination_dir)) + + +@attrs.define(slots=False) +class PretrainedWeightsConfig: + # Master switch. When False, the trainer skips the pretrained-weights load + # path entirely. + enabled: bool = True + + # Path to the pretrained-weights snapshot for the backbone. Accepts s3://, + # gs://, hf://, or a local filesystem path. Empty means no overlay; the + # trainer falls back to whatever the AutoModel constructor produces. + backbone_path: str = "" + + # Path to the credentials .secret used to fetch backbone_path from object + # storage. Empty means anonymous (works for hf:// and public buckets). + credentials_path: str = "" + + # Apply the boto3 GCS-compatibility patch when loading DCP shards from a + # gs:// URI. Required for DCP loads from GCS; harmless otherwise. + enable_gcs_patch_in_boto3: bool = False + + # Force a specific safetensors weight remapping (e.g. "qwen3" vs + # "nemotron_3_dense_vl" / "nemotron_3_llm"). None lets the loader auto-detect. + checkpoint_format: str | None = None + + +@attrs.define(slots=False) +class VLMConfig: + """VLM backbone identity shared by OmniMoTModelConfig.vlm_config and VLMModelConfig.policy.backbone. + + model_instance and tokenizer are typed Any | None instead of LazyDict | None + because OmegaConf 2.3 rejects LazyDict as a structured-config annotation; the + runtime value is still a LazyDict. + """ + + # HuggingFace model identifier or local path. Drives AutoConfig + AutoModel selection. + model_name: str = "" + + # Safetensor path for model + safetensors_path: str = "" + + # Optional pretrained-weights overlay (separate from the AutoModel structural + # init driven by model_name). + pretrained_weights: PretrainedWeightsConfig = PretrainedWeightsConfig() + + # Optional LazyCall override for the language-model class to instantiate. + # When set, the trainer routes construction through lazy_instantiate(model_instance) + # instead of the AutoModelForCausalLM / AutoModelForVision2Seq from_pretrained path. + model_instance: Any | None = None + + # Optional LazyCall override for the tokenizer/processor (a BaseVLMProcessor + # subclass). When None, callers may auto-derive via AutoTokenizer.from_pretrained. + tokenizer: Any | None = None + + # Override class name for the decoder layer (e.g. "Qwen2MoTDecoderLayer"); the + # substring "Mo" gates MoE detection in cosmos3_vfm_network. None means no swap. + layer_module: str | None = None + + # Apply QK normalization in the language-model decoder. + qk_norm: bool = False + + # Whether input and output word-embedding matrices are tied. Affects the + # safetensors loader (lm_head load is skipped when tied) and the FSDP wrapper. + tie_word_embeddings: bool = False + + # Prepend a system prompt during text tokenization. Checkpoints trained with + # system prompt enabled require this set true at inference time. + use_system_prompt: bool = False + + +# Configs for LLM models +Qwen3MoT_LLM_0p6b_Config: VLMConfig = VLMConfig( + model_name="Qwen/Qwen3-0.6B", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3MoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/llm/qwen3/configs/Qwen3-0.6B.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-0.6B", + config_variant="hf", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://checkpoints-us-east-1/cosmos3/pretrained/huggingface/Qwen/Qwen3-0.6B/", + credentials_path="credentials/s3_training.secret", + ), +) + +Qwen3MoT_LLM_0p6b_GCP_Config: VLMConfig = VLMConfig( + model_name="Qwen/Qwen3-0.6B", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3MoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/llm/qwen3/configs/Qwen3-0.6B.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-0.6B", + config_variant="gcp", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Qwen/Qwen3-0.6B/", + credentials_path="credentials/gcp_checkpoint.secret", + ), +) + +Nemotron3_LLM_2b_GCP_Config: VLMConfig = VLMConfig( + model_name="nvidia/NVIDIA-Nemotron-3-2B-BF16", + model_instance=L(Nemotron3DenseVLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Nemotron3DenseVLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/nemotron_3_dense_vl/configs/Nemotron-2B-Dense-VL.json" + ), + qk_norm_for_text=False, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Nemotron/NVIDIA-Nemotron-3-2B-BF16", + config_variant="gcp", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Nemotron/NVIDIA-Nemotron-3-2B-BF16/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + checkpoint_format="nemotron_3_llm", + ), +) + +# Configs for VL instruct models + +# Config for Qwen3VL 30B A3B Instruct model +# Qwen3VLMoE uses Qwen2Tokenizer +Qwen3VLMoT_VLM_30b_a3b_Instruct_Config: VLMConfig = VLMConfig( + model_name="Qwen/Qwen3-VL-30B-A3B-Instruct", + model_instance=L(Qwen3VLMoeTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoeMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl_moe/configs/Qwen3-VL-30B-A3B-Instruct.json" + ), + layer_module="Qwen3VLMoeTextMoTDecoderLayer", + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-30B-A3B-Instruct", + config_variant="s3", + ), + layer_module="Qwen3VLMoeTextMoTDecoderLayer", + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://checkpoints-us-east-1/cosmos3/pretrained/huggingface/Qwen/Qwen3-VL-30B-A3B-Instruct/", + credentials_path="credentials/s3_training.secret", + ), +) + + +Qwen3VLMoT_VLM_30b_a3b_Instruct_GCP_Config: VLMConfig = VLMConfig( + model_name="Qwen/Qwen3-VL-30B-A3B-Instruct", + model_instance=L(Qwen3VLMoeTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoeMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl_moe/configs/Qwen3-VL-30B-A3B-Instruct.json" + ), + layer_module="Qwen3VLMoeTextMoTDecoderLayer", + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-30B-A3B-Instruct", + config_variant="gcp", + ), + layer_module="Qwen3VLMoeTextMoTDecoderLayer", + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Qwen/Qwen3-VL-30B-A3B-Instruct/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + ), +) + +CosmosReason2_VLM_30b_a3b_Private_GCP_Config: VLMConfig = VLMConfig( + model_name="nvidia/Cosmos-Reason2-30B-A3B-Private", + model_instance=L(Qwen3VLMoeTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoeMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl_moe/configs/Qwen3-VL-30B-A3B-Instruct.json" + ), + layer_module="Qwen3VLMoeTextMoTDecoderLayer", + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-30B-A3B-Instruct", + config_variant="gcp", + ), + layer_module="Qwen3VLMoeTextMoTDecoderLayer", + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Cosmos-Reason/Cosmos-Reason2-30B-A3B-Private/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + ), +) + +# Config for Qwen3VL 235B A22B Instruct model +# Qwen3VLMoE uses Qwen2Tokenizer +Qwen3VLMoT_VLM_235b_a22b_Instruct_GCP_Config: VLMConfig = VLMConfig( + model_name="Qwen/Qwen3-VL-235B-A22B-Instruct", + model_instance=L(Qwen3VLMoeTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoeMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl_moe/configs/Qwen3-VL-235B-A22B-Instruct.json" + ), + layer_module="Qwen3VLMoeTextMoTDecoderLayer", + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-235B-A22B-Instruct", + config_variant="gcp", + ), + layer_module="Qwen3VLMoeTextMoTDecoderLayer", + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Qwen/Qwen3-VL-235B-A22B-Instruct/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + ), +) + + +# Config for Qwen3VL 2B Instruct model +# Qwen3VL uses Qwen2Tokenizer +Qwen3VLMoT_VLM_2b_Instruct_Config: VLMConfig = VLMConfig( + model_name="Qwen/Qwen3-VL-2B-Instruct", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-2B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-2B-Instruct", + config_variant="s3", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://checkpoints-us-east-1/cosmos3/pretrained/huggingface/Qwen/Qwen3-VL-2B-Instruct/", + credentials_path="credentials/s3_training.secret", + ), +) + +Qwen3VLMoT_VLM_2b_Instruct_GCP_Config: VLMConfig = VLMConfig( + model_name="Qwen/Qwen3-VL-2B-Instruct", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-2B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-2B-Instruct", + config_variant="gcp", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Qwen/Qwen3-VL-2B-Instruct/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + ), +) + +Qwen3VLMoT_VLM_2b_Instruct_HF_Config: VLMConfig = VLMConfig( + model_name="Qwen/Qwen3-VL-2B-Instruct", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-2B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-2B-Instruct", + config_variant="hf", + ), +) + +Nemotron3DenseVL_VLM_2b_GCP_Config: VLMConfig = VLMConfig( + model_name="nvidia/Nemotron-3-Dense-VL-2B-BF16-Alignment", + model_instance=L(Nemotron3DenseVLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Nemotron3DenseVLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/nemotron_3_dense_vl/configs/Nemotron-2B-Dense-VL.json" + ), + qk_norm_for_text=False, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Nemotron/NVIDIA-Nemotron-3-Dense-VL-2B-BF16-Alignment", + config_variant="gcp", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Nemotron/NVIDIA-Nemotron-3-Dense-VL-2B-BF16-Alignment/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + checkpoint_format="nemotron_3_dense_vl", + ), +) + +Cosmos3Reasoner_Nemotron_VLM_2b_Private_GCP_Config: VLMConfig = VLMConfig( + model_name="nvidia/Cosmos3-Reasoner-2B-Private", + model_instance=L(Nemotron3DenseVLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Nemotron3DenseVLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/nemotron_3_dense_vl/configs/Nemotron-2B-Dense-VL.json" + ), + qk_norm_for_text=False, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="nvidia/Cosmos3-Reasoner-2B-Private", + config_variant="gcp", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/nvidia/Cosmos3-Reasoner-2B-Private/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + checkpoint_format="nemotron_3_dense_vl", + ), +) + +CosmosReason2_VLM_2b_GCP_Config: VLMConfig = VLMConfig( + model_name="nvidia/Cosmos-Reason2-2B", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-2B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-2B-Instruct", + config_variant="gcp", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Cosmos-Reason/Cosmos-Reason2-2B/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + ), +) + +CosmosReason2_VLM_2b_Private_GCP_Config: VLMConfig = VLMConfig( + model_name="nvidia/Cosmos-Reason2-2B-Private", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-2B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-2B-Instruct", + config_variant="gcp", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Cosmos-Reason/Cosmos-Reason2-2B-Private/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + ), +) + +Cosmos3Reasoner_VLM_2b_Private_GCP_Config: VLMConfig = VLMConfig( + model_name="nvidia/Cosmos3-Reasoner-2B-Private", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-2B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-2B-Instruct", + config_variant="gcp", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Cosmos-Reason/Cosmos3-Reasoner-2B-Private/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + ), +) + +# Config for Qwen3VL 4B Instruct model +# Qwen3VL uses Qwen2Tokenizer +Qwen3VLMoT_VLM_4b_Instruct_Config: VLMConfig = VLMConfig( + model_name="Qwen/Qwen3-VL-4B-Instruct", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-4B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-4B-Instruct", + config_variant="s3", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://checkpoints-us-east-1/cosmos3/pretrained/huggingface/Qwen/Qwen3-VL-4B-Instruct/", + credentials_path="credentials/s3_training.secret", + ), +) + +Qwen3VLMoT_VLM_4b_Instruct_GCP_Config: VLMConfig = VLMConfig( + model_name="Qwen/Qwen3-VL-4B-Instruct", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-4B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-4B-Instruct", + config_variant="gcp", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Qwen/Qwen3-VL-4B-Instruct/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + ), +) + +# Config for Qwen3VL 8B Instruct model +# Qwen3VL uses Qwen2Tokenizer +Qwen3VLMoT_VLM_8b_Instruct_Config: VLMConfig = VLMConfig( + model_name="Qwen/Qwen3-VL-8B-Instruct", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-8B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-8B-Instruct", + config_variant="s3", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://checkpoints-us-east-1/cosmos3/pretrained/huggingface/Qwen/Qwen3-VL-8B-Instruct/", + credentials_path="credentials/s3_training.secret", + ), +) + +Qwen3VLMoT_VLM_8b_Instruct_GCP_Config: VLMConfig = VLMConfig( + model_name="Qwen/Qwen3-VL-8B-Instruct", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-8B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-8B-Instruct", + config_variant="gcp", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Qwen/Qwen3-VL-8B-Instruct/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + ), +) + +CosmosReason2_VLM_8b_Private_GCP_Config: VLMConfig = VLMConfig( + model_name="nvidia/Cosmos-Reason2-8B-Private", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-8B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-8B-Instruct", + config_variant="gcp", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Cosmos-Reason/Cosmos-Reason2-8B-Private/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + ), +) + +Cosmos3Reasoner_VLM_8b_Private_GCP_Config: VLMConfig = VLMConfig( + model_name="nvidia/Cosmos3-Reasoner-8B-Private", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-8B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-8B-Instruct", + config_variant="gcp", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Cosmos-Reason/Cosmos3-Reasoner-8B-Private/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + ), +) + +Cosmos3NanoReasoner_VLM_GCP_Config: VLMConfig = VLMConfig( + model_name="nvidia/Cosmos3-Nano-Reasoner", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-8B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(create_qwen2_tokenizer_with_download)( + pretrained_model_name="Qwen/Qwen3-VL-8B-Instruct", + config_variant="gcp", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Cosmos-Reason/Cosmos3-Nano-Reasoner/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + ), +) + + +# Config for Qwen3VL 32B Instruct model +# Qwen3VL uses Qwen2Tokenizer +Qwen3VLMoT_VLM_32b_Instruct_Config: VLMConfig = VLMConfig( + model_name="Qwen/Qwen3-VL-32B-Instruct", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-32B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-32B-Instruct", + config_variant="s3", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://checkpoints-us-east-1/cosmos3/pretrained/huggingface/Qwen/Qwen3-VL-32B-Instruct/", + credentials_path="credentials/s3_training.secret", + ), +) + +Qwen3VLMoT_VLM_32b_Instruct_GCP_Config: VLMConfig = VLMConfig( + model_name="Qwen/Qwen3-VL-32B-Instruct", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-32B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-32B-Instruct", + config_variant="gcp", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Qwen/Qwen3-VL-32B-Instruct/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + ), +) + +CosmosReason2_VLM_32b_Private_GCP_Config: VLMConfig = VLMConfig( + model_name="nvidia/Cosmos-Reason2-32B-Private", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-32B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-32B-Instruct", + config_variant="gcp", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Cosmos-Reason/Cosmos-Reason2-32B-Private/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + ), +) + +Cosmos3Reasoner_VLM_32b_Private_GCP_Config: VLMConfig = VLMConfig( + model_name="nvidia/Cosmos3-Reasoner-32B-Private", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-32B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(build_processor_lazy)( + tokenizer_type="Qwen/Qwen3-VL-32B-Instruct", + config_variant="gcp", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Cosmos-Reason/Cosmos3-Reasoner-32B-Private/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + ), +) + +Cosmos3SuperReasoner_VLM_GCP_Config: VLMConfig = VLMConfig( + model_name="nvidia/Cosmos3-Super-Reasoner", + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file="cosmos_framework/model/vfm/vlm/qwen3_vl/configs/Qwen3-VL-32B-Instruct.json" + ), + qk_norm_for_text=True, + ), + ), + tokenizer=L(create_qwen2_tokenizer_with_download)( + pretrained_model_name="Qwen/Qwen3-VL-32B-Instruct", + config_variant="gcp", + ), + pretrained_weights=PretrainedWeightsConfig( + backbone_path="s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/Cosmos-Reason/Cosmos3-Super-Reasoner/", + credentials_path="credentials/gcp_checkpoint.secret", + enable_gcs_patch_in_boto3=True, + ), +) + + + +def register_vlm(): + cs = ConfigStore.instance() + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="qwen3_mot_0p6b", + node=Qwen3MoT_LLM_0p6b_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="qwen3_mot_0p6b_gcp", + node=Qwen3MoT_LLM_0p6b_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="nemotron_3_llm_2b_gcp", + node=Nemotron3_LLM_2b_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="qwen3_vl_mot_vlm_30b_a3b_instruct", + node=Qwen3VLMoT_VLM_30b_a3b_Instruct_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="qwen3_vl_mot_vlm_30b_a3b_instruct_gcp", + node=Qwen3VLMoT_VLM_30b_a3b_Instruct_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="qwen3_vl_mot_vlm_235b_a22b_instruct_gcp", + node=Qwen3VLMoT_VLM_235b_a22b_Instruct_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="qwen3_vl_mot_vlm_2b_instruct", + node=Qwen3VLMoT_VLM_2b_Instruct_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="qwen3_vl_mot_vlm_2b_instruct_gcp", + node=Qwen3VLMoT_VLM_2b_Instruct_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="qwen3_vl_mot_vlm_2b_instruct_hf", + node=Qwen3VLMoT_VLM_2b_Instruct_HF_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="nemotron_3_dense_vl_2b_gcp", + node=Nemotron3DenseVL_VLM_2b_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="cosmos3_reasoner_nemotron_vlm_2b_private_gcp", + node=Cosmos3Reasoner_Nemotron_VLM_2b_Private_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="cosmos_reason2_vlm_2b_gcp", + node=CosmosReason2_VLM_2b_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="cosmos_reason2_vlm_2b_private_gcp", + node=CosmosReason2_VLM_2b_Private_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="cosmos3_reasoner_vlm_2b_private_gcp", + node=Cosmos3Reasoner_VLM_2b_Private_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="cosmos_reason2_vlm_8b_private_gcp", + node=CosmosReason2_VLM_8b_Private_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="cosmos3_reasoner_vlm_8b_private_gcp", + node=Cosmos3Reasoner_VLM_8b_Private_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="cosmos3_nano_reasoner_vlm_gcp", + node=Cosmos3NanoReasoner_VLM_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="cosmos_reason2_vlm_32b_private_gcp", + node=CosmosReason2_VLM_32b_Private_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="cosmos3_reasoner_vlm_32b_private_gcp", + node=Cosmos3Reasoner_VLM_32b_Private_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="cosmos3_super_reasoner_vlm_gcp", + node=Cosmos3SuperReasoner_VLM_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="cosmos_reason2_vlm_30b_a3b_private_gcp", + node=CosmosReason2_VLM_30b_a3b_Private_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="qwen3_vl_mot_vlm_4b_instruct", + node=Qwen3VLMoT_VLM_4b_Instruct_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="qwen3_vl_mot_vlm_4b_instruct_gcp", + node=Qwen3VLMoT_VLM_4b_Instruct_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="qwen3_vl_mot_vlm_8b_instruct", + node=Qwen3VLMoT_VLM_8b_Instruct_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="qwen3_vl_mot_vlm_8b_instruct_gcp", + node=Qwen3VLMoT_VLM_8b_Instruct_GCP_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="qwen3_vl_mot_vlm_32b_instruct", + node=Qwen3VLMoT_VLM_32b_Instruct_Config, + ) + cs.store( + group="vlm_config", + package="model.config.vlm_config", + name="qwen3_vl_mot_vlm_32b_instruct_gcp", + node=Qwen3VLMoT_VLM_32b_Instruct_GCP_Config, + ) diff --git a/cosmos_framework/configs/base/experiment/action/__init__.py b/cosmos_framework/configs/base/experiment/action/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/configs/base/experiment/action/_experiment_helpers.py b/cosmos_framework/configs/base/experiment/action/_experiment_helpers.py new file mode 100644 index 0000000..440cce6 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/_experiment_helpers.py @@ -0,0 +1,159 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import copy +import os +from typing import Any + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.libero_dataset import LIBERO_ROOTS, LIBERODataset +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry + +MODES = [ + ("fd", "forward_dynamics"), + ("id", "inverse_dynamics"), + ("policy", "policy"), + ("i2v", "image2video"), + ("joint", "joint"), +] + + +def register_modes(cs: ConfigStore, parent_name: str, base_config: dict, *, dataloader_key: str = "dataloader") -> None: + """Register mode variants (fd, id, policy, i2v, joint) for a parent experiment. + + Args: + dataloader_key: Key inside ``dataloaders`` dict. Legacy configs use + ``"dataloader"``; new ``make_2b/8b_experiment`` configs use ``"action_data"``. + """ + for suffix, mode in MODES: + name = f"{parent_name}_{suffix}" + list_of_datasets = copy.deepcopy( + base_config["dataloader_train"]["dataloaders"][dataloader_key]["dataloader"]["dataset"]["list_of_datasets"] + ) + list_of_datasets[0]["dataset"]["mode"] = mode + node = dict( + defaults=[f"/experiment/{parent_name}", "_self_"], + dataloader_train=dict( + dataloaders={dataloader_key: dict(dataloader=dict(dataset=dict(list_of_datasets=list_of_datasets)))} + ), + job=dict(name=f"${{now:%Y-%m-%d_%H-%M-%S}}_{name}"), + ) + cs.store(name, node, group="experiment", package="_global_") + + +def register_embodiment_type( + cs: ConfigStore, + parent_name: str, + suffix: str, + embodiment_type: str, + base_config: dict, + *, + dataloader_key: str = "dataloader", + dataset_target: Any | None = None, +) -> dict: + """Register an embodiment-type variant for a parent experiment. + + Args: + dataloader_key: Key inside ``dataloaders`` dict. Legacy configs use + ``"dataloader"``; new ``make_2b/8b_experiment`` configs use ``"action_data"``. + """ + name = f"{parent_name}_{suffix}" + list_of_datasets = copy.deepcopy( + base_config["dataloader_train"]["dataloaders"][dataloader_key]["dataloader"]["dataset"]["list_of_datasets"] + ) + list_of_datasets[0]["dataset"]["embodiment_type"] = embodiment_type + if dataset_target is not None: + list_of_datasets[0]["dataset"]["_target_"] = dataset_target + node = dict( + defaults=[f"/experiment/{parent_name}", "_self_"], + dataloader_train=dict( + dataloaders={dataloader_key: dict(dataloader=dict(dataset=dict(list_of_datasets=list_of_datasets)))} + ), + ) + cs.store(name, node, group="experiment", package="_global_") + return node + + +# --------------------------------------------------------------------------- +# LIBERO ``list_of_datasets`` helper +# --------------------------------------------------------------------------- +# The default ``root`` for each LIBERO suite resolves at import time: +# 1. If ``$LIBERO_LOCAL_DATA_ROOT`` is set, use it as the base directory and +# look up each suite's local ``_20260124``-style subfolder. +# 2. Otherwise fall back to ``LIBERO_ROOTS`` from ``libero_dataset.py`` (the +# cluster lustre layout). + +LIBERO_LOCAL_ROOT_ENV = "LIBERO_LOCAL_DATA_ROOT" + +# Shared training-loop baseline used by libero_exp and all libero_*_experiment +# variants (fd, joint, policy). Individual experiments may still override these +LIBERO_BASELINE_BATCH_SIZE: int = 4 +LIBERO_BASELINE_NUM_WORKERS: int = 16 +LIBERO_BASELINE_TRAINING_ITERATIONS: int = 4_000 + +LIBERO_REPO_IDS: list[str] = [ + "libero_10", + "libero_90", + "libero_object", + "libero_spatial", + "libero_goal", +] + +# Subfolder name (under ${LIBERO_LOCAL_DATA_ROOT}) for each repo_id. Order matches +# LIBERO_REPO_IDS. These mirror the gs://nv-00-10206-robot/lerobot_v30/ layout. +LIBERO_LOCAL_SUITE_DIRS: list[str] = [ + "libero_10_no_noops_1.0.0_lerobot_aligned_20260124", + "libero_90_no_noops_lerobot_shuffled_20260124", + "libero_object_no_noops_1.0.0_lerobot_aligned_20260124", + "libero_spatial_no_noops_1.0.0_lerobot_20260124", + "libero_goal_no_noops_1.0.0_lerobot_20260124", +] + + +def _resolve_libero_default_roots() -> list[str]: + """Resolve the default ``root`` list for ``LIBERODataset``. + + Reads ``$LIBERO_LOCAL_DATA_ROOT`` and joins each suite subdir. Falls back to + ``LIBERO_ROOTS`` (cluster lustre layout) when the env var is unset. + """ + base = os.environ.get(LIBERO_LOCAL_ROOT_ENV) + if not base: + return list(LIBERO_ROOTS) + + candidates = [os.path.join(base, suite) for suite in LIBERO_LOCAL_SUITE_DIRS] + missing = [p for p in candidates if not os.path.isdir(p)] + if missing: + raise FileNotFoundError( + f"${LIBERO_LOCAL_ROOT_ENV}={base} is set but the following LIBERO suite directories are missing: {missing}" + ) + log.info(f"[libero] using local LIBERO mount at {base} (via ${LIBERO_LOCAL_ROOT_ENV})") + return candidates + + +LIBERO_DEFAULT_KWARGS: dict = dict( + repo_id=list(LIBERO_REPO_IDS), + root=_resolve_libero_default_roots(), + split="train", + camera_mode="image", +) + + +def make_libero_dataset(**libero_dataset_kwargs): + """Build a single-entry ``list_of_datasets`` for a libero experiment. + + Returns a one-element list containing one ``L(dataset_entry)(...)`` LazyCall + whose inner ``LIBERODataset`` kwargs are ``LIBERO_DEFAULT_KWARGS`` layered + with the caller-provided overrides. Pass the returned value as the + ``datasets=`` argument to ``make_2b_experiment``. + """ + kwargs = {**LIBERO_DEFAULT_KWARGS, **libero_dataset_kwargs} + return [ + L(dataset_entry)( + name="libero", + dataset=L(LIBERODataset)(**kwargs), + ratio=1.0, + ), + ] diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/__init__.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/agibotworld_beta_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/agibotworld_beta_experiment.py new file mode 100644 index 0000000..f714515 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/agibotworld_beta_experiment.py @@ -0,0 +1,52 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# AgiBotWorld Beta experiment — Cosmos3 2B pretrained base +# +# Base experiment (policy mode) + mode variants (fd, id, policy, i2v, joint). +# Plus keep-aspect-ratio variant. + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.experiment.action._experiment_helpers import register_modes +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_2b import make_2b_experiment +from cosmos_framework.data.vfm.action.agibotworld_beta_dataset import AgiBotWorldBetaDataset +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry + +cs = ConfigStore.instance() + +# --------------------------------------------------------------------------- +# Dataset definition (shared across all modes) +# --------------------------------------------------------------------------- +AGIBOTWORLD_BETA_DATASET = [ + L(dataset_entry)( + name="agibotworld_beta", + dataset=L(AgiBotWorldBetaDataset)(chunk_length=16, split="train"), + ratio=1.0, + ), +] + +# --------------------------------------------------------------------------- +# Base experiment — 2B, 4k iters (policy mode default) +# --------------------------------------------------------------------------- +agibotworld_beta = make_2b_experiment( + exp_name="agibotworld_beta", + datasets=AGIBOTWORLD_BETA_DATASET, + training_iterations=4_000, +) +agibotworld_beta["job"]["group"] = "agibotworld_beta" + +cs.store("agibotworld_beta", agibotworld_beta, group="experiment", package="_global_") +register_modes(cs, "agibotworld_beta", agibotworld_beta, dataloader_key="action_data") + + +# --------------------------------------------------------------------------- +# Keep aspect ratio. +# --------------------------------------------------------------------------- +agibotworld_beta_kar = dict( + defaults=["/experiment/agibotworld_beta", "_self_"], + dataloader_train=dict(dataloaders=dict(action_data=dict(dataloader=dict(dataset=dict(keep_aspect_ratio=True))))), +) +cs.store("agibotworld_beta_kar", agibotworld_beta_kar, group="experiment", package="_global_") +register_modes(cs, "agibotworld_beta_kar", agibotworld_beta, dataloader_key="action_data") diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/av_multires_8b_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/av_multires_8b_experiment.py new file mode 100644 index 0000000..282ef67 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/av_multires_8b_experiment.py @@ -0,0 +1,192 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# AV policy experiment on the multires 8B model with modality-offset mRoPE. +# +# Built on make_8b_experiment() from cosmos3_8b.py with overrides for: +# - 720p multires (256/480/720) with per-resolution shift +# - Modality-offset mRoPE (temporal_modality_margin=15000) +# - exp302_003 checkpoint (multires modality offset pretrained) +# - Extended tokenizer warmup for multi-resolution + +import copy + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils.lazy_config import LazyDict +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_8b import make_8b_experiment +from cosmos_framework.data.vfm.action.av_dataset import AVDataset +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry + +cs = ConfigStore.instance() + + +# --------------------------------------------------------------------------- +# Dataset +# --------------------------------------------------------------------------- +AV_DATASET_I2V = [ + L(dataset_entry)( + name="av", + dataset=L(AVDataset)( + split="train", + fps=10, + mode="image2video", + history_len=0.1, + future_len=6.0, + rotation_format="rot6d", + pose_convention="backward_anchored", + credential_path="${job.cluster.object_store_credential_data}", + shuffle=True, + translation_scale=0.01, + ), + ratio=6, + ), +] + +# --------------------------------------------------------------------------- +# Base experiment — i2v mode +# --------------------------------------------------------------------------- +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v = make_8b_experiment( + exp_name="av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v", + datasets=AV_DATASET_I2V, + training_iterations=25_000, +) + +# -- Override job -- +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["job"]["group"] = "action_av" +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["job"]["name"] = ( + "av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v" +) + +# -- Override model for multires 720p -- +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["model"]["config"]["resolution"] = "720" +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["model"]["config"]["max_num_tokens_after_packing"] = 45056 +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["model"]["config"]["rectified_flow_training_config"]["shift"] = { + "256": 1, + "480": 2, + "720": 3, +} +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["model"]["config"]["diffusion_expert_config"][ + "unified_3d_mrope_temporal_modality_margin" +] = 15000 +# Remove extrapolation ratios and max_vae_latent_side (not in original multires config) +for _key in [ + "rope_h_extrapolation_ratio", + "rope_w_extrapolation_ratio", + "rope_t_extrapolation_ratio", + "max_vae_latent_side_after_patchify", +]: + av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["model"]["config"]["diffusion_expert_config"].pop(_key, None) + +# -- Override tokenizer -- +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["model"]["config"]["tokenizer"]["encode_chunk_frames"] = { + "256": 68, + "480": 24, + "720": 12, +} +# Remove fields not in original +for _key in ["encode_exact_durations"]: + av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["model"]["config"]["tokenizer"].pop(_key, None) + +# -- Override optimizer -- +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["optimizer"]["lr"] = 1e-4 + +# -- Override scheduler -- +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["scheduler"]["warm_up_steps"] = [0] +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["scheduler"]["cycle_lengths"] = [200_000] + +# -- Override trainer -- +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["trainer"]["logging_iter"] = 50 +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["trainer"]["callbacks"]["norm_monitor"] = dict(every_n=100) +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["trainer"]["callbacks"]["sigma_loss_analysis"] = dict( + every_n=500, + every_n_viz=500, +) +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["trainer"]["callbacks"]["compile_tokenizer"] = dict( + enabled=True, + warmup_resolutions=["256", "480", "720"], +) +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["trainer"]["compile_config"]["recompile_limit"] = 100 + +# -- Override checkpoint (exp302_003 multires) -- +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["checkpoint"]["save_iter"] = 250 +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["checkpoint"]["load_path"] = ( + "cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/" + "t2w_mot_exp302_003_qwen3_vl_8b_multires_modality_offset/" + "checkpoints/iter_000006750/" +) +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["checkpoint"]["load_from_object_store"] = dict(enabled=True) + +# Remove vlm_config.pretrained_weights.enabled (not in original — inherits default) +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v["model"]["config"]["vlm_config"]["pretrained_weights"].pop( + "enabled", None +) + +# Wrap in LazyDict for allow_objects support +av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v = LazyDict( + av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v, + flags={"allow_objects": True}, +) + +cs.store( + group="experiment", + package="_global_", + name="av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v", + node=av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v, +) + +# --------------------------------------------------------------------------- +# Policy variant +# --------------------------------------------------------------------------- +AV_DATASET_POLICY = copy.deepcopy(AV_DATASET_I2V) +AV_DATASET_POLICY[0]["dataset"]["mode"] = "policy" + +av_exp302_003_qwen3_vl_8b_multires_modality_offset_policy = LazyDict( + dict( + defaults=[ + "/experiment/av_exp302_003_qwen3_vl_8b_multires_modality_offset_i2v", + "_self_", + ], + job=dict( + group="action_av", + name="av_exp302_003_qwen3_vl_8b_multires_modality_offset_policy", + ), + dataloader_train=dict( + dataloaders=dict( + action_data=dict( + dataloader=dict( + dataset=dict( + list_of_datasets=[ + L(dataset_entry)( + name="av", + dataset=L(AVDataset)( + split="train", + fps=10, + mode="policy", + history_len=0.1, + future_len=6.0, + rotation_format="rot6d", + pose_convention="backward_anchored", + credential_path="${job.cluster.object_store_credential_data}", + shuffle=True, + translation_scale=0.01, + ), + ratio=6, + ), + ], + ), + ), + ), + ), + ), + ), + flags={"allow_objects": True}, +) + +cs.store( + group="experiment", + package="_global_", + name="av_exp302_003_qwen3_vl_8b_multires_modality_offset_policy", + node=av_exp302_003_qwen3_vl_8b_multires_modality_offset_policy, +) diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/bridge_clean_lerobot_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/bridge_clean_lerobot_experiment.py new file mode 100644 index 0000000..2c14d9f --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/bridge_clean_lerobot_experiment.py @@ -0,0 +1,333 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Bridge clean LeRobot experiment configs (post-training). + +====================================================================================================================================================== +Runnable experiments — Columns: action | rot | norm | chunk | model | iter | mode | resolution | notes +====================================================================================================================================================== +8B backward_framewise policy from exp302_000 (SoT 36 base VFM checkpoint) +8B backward_framewise fdm/joint from exp506_000 midtraining_v1 + Policy loads from t2w_mot_exp302_000_qwen3_vl_8b_multires_recipe_v7/iter_000043500 + FDM/joint loads from t2w_mot_exp506_000_qwen3_vl_8b_multires_recipe_midtraining_v1/iter_000040000 + + bridge_clean_8b_framewise_from_exp302_000_chunk16_quantile_norm_policy backward_framewise rot6d quantile c16 8B 4k policy 256 (resize) alr=5 + bridge_clean_8b_framewise_from_exp302_000_chunk16_nonorm_policy backward_framewise rot6d none c16 8B 4k policy 256 (resize) alr=5 + bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_fdm backward_framewise rot6d quantile_rot c16 8B 4k fdm 256 (resize) alr=5 + bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_fdm backward_framewise rot6d none c16 8B 4k fdm 256 (resize) alr=5 + bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_joint backward_framewise rot6d quantile_rot c16 8B 4k joint 256 (resize) alr=5 + bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_joint backward_framewise rot6d none c16 8B 4k joint 256 (resize) alr=5 + bridge_clean_8b_framewise_from_exp302_000_chunk16_quantile_norm_policy_highres backward_framewise rot6d quantile c16 8B 4k policy 480 (native) alr=5 + bridge_clean_8b_framewise_from_exp302_000_chunk16_nonorm_policy_highres backward_framewise rot6d none c16 8B 4k policy 480 (native) alr=5 + bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_fdm_highres backward_framewise rot6d quantile_rot c16 8B 4k fdm 480 (native) alr=5 + bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_fdm_highres backward_framewise rot6d none c16 8B 4k fdm 480 (native) alr=5 + bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_joint_highres backward_framewise rot6d quantile_rot c16 8B 4k joint 480 (native) alr=5 + bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_joint_highres backward_framewise rot6d none c16 8B 4k joint 480 (native) alr=5 + bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_rot_norm_idm_highres backward_framewise rot6d quantile_rot c16 8B 4k idm 480 (native) alr=5 + bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_idm_highres backward_framewise rot6d none c16 8B 4k idm 480 (native) alr=5 + +====================================================================================================================================================== +""" + +import copy + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.configs.base.experiment.action.midtrain_config.action_datasets import DATASET_BRIDGE +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_8b import make_8b_experiment + +cs = ConfigStore.instance() + +# --------------------------------------------------------------------------- +# Constants +# --------------------------------------------------------------------------- + +_ACTION_KEYS_TO_SKIP = ["action2llm", "llm2action", "action_modality_embed", "action_pos_embed"] + +_EXP302_000_CKPT = ( + "cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/t2w_mot_exp302_000_qwen3_vl_8b_multires_recipe_v7/checkpoints/iter_000043500/" +) + +_EXP506_000_MIDTRAIN_CKPT = ( + "cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/" + "t2w_mot_exp506_000_qwen3_vl_8b_multires_recipe_midtraining_v1/checkpoints/iter_000040000/" +) + +# Per-resolution batch budget. Highres pipeline uses ~6x more pixels (native +# Bridge 640x480 mapped to the 480 bucket "4,3" = 736x544), so we shrink batch. +_BATCH_BY_RES: dict[str, int] = { + "256": 256, + "480": 64, +} + + +# --------------------------------------------------------------------------- +# Dataset: derive from midtrain DATASET_BRIDGE (mode=joint, no norm, backward_framewise) +# Override mode and stamp the resolution bucket so wrap_dataset +# performs the corresponding resize + reflection-pad (see unified_dataset.py / +# VIDEO_RES_SIZE_INFO). +# --------------------------------------------------------------------------- +def _make_bridge_posttrain_dataset( + *, + pose_convention: str = "backward_framewise", + action_normalization: str | None = "quantile", + mode: str = "policy", + resolution: str = "256", +) -> list: + ds = copy.deepcopy(DATASET_BRIDGE) + ds.dataset.mode = mode + ds.dataset.pose_convention = pose_convention + ds.dataset.action_normalization = action_normalization + ds.resolution = resolution + return [ds] + + +# --------------------------------------------------------------------------- +# Helper: common 8B posttrain overrides +# --------------------------------------------------------------------------- +def _apply_8b_posttrain_overrides(exp: dict, *, checkpoint_path: str, resolution: str = "256") -> dict: + exp["checkpoint"]["save_iter"] = 1000 + exp["checkpoint"]["load_path"] = checkpoint_path + exp["checkpoint"]["load_from_object_store"] = dict(bucket="nv-00-10206-checkpoint-experiments") + + exp["model"]["config"]["max_num_tokens_after_packing"] = -1 + exp["model"]["config"]["rectified_flow_training_config"]["shift"] = {"256": 3, "480": 5, "720": 10} + exp["model"]["config"]["diffusion_expert_config"]["unified_3d_mrope_temporal_modality_margin"] = 15000 + exp["model"]["config"]["diffusion_expert_config"]["max_vae_latent_side_after_patchify"] = 52 + exp["model"]["config"]["tokenizer"]["encode_exact_durations"] = [17] + + exp["dataloader_train"]["max_sequence_length"] = None + exp["dataloader_train"]["max_samples_per_batch"] = _BATCH_BY_RES[resolution] + + exp["optimizer"]["lr"] = 2e-4 + exp["scheduler"]["warm_up_steps"] = [100] + exp["scheduler"]["f_min"] = [0.05] + exp["trainer"]["run_validation"] = False + exp["trainer"]["callbacks"]["compile_tokenizer"]["warmup_resolutions"] = [resolution] + + exp["job"]["project"] = "cosmos3_action" + exp["job"]["group"] = "bridge_clean_lerobot" + return exp + + +def _build_exp( + exp_name: str, + *, + action_normalization: str | None, + mode: str = "policy", + resolution: str, + checkpoint_path: str = _EXP302_000_CKPT, +) -> dict: + return _apply_8b_posttrain_overrides( + make_8b_experiment( + exp_name, + _make_bridge_posttrain_dataset( + action_normalization=action_normalization, + mode=mode, + resolution=resolution, + ), + training_iterations=4_000, + action_param_lr_multipliers=5.0, + keys_to_skip_loading=_ACTION_KEYS_TO_SKIP, + ), + checkpoint_path=checkpoint_path, + resolution=resolution, + ) + + +# =========================================================================== +# 8B backward_framewise from exp302_000 — 256 (resize) +# =========================================================================== + +bridge_clean_8b_framewise_from_exp302_000_chunk16_quantile_norm_policy = _build_exp( + "bridge_clean_8b_framewise_from_exp302_000_chunk16_quantile_norm_policy", + action_normalization="quantile", + resolution="256", +) +cs.store( + "bridge_clean_8b_framewise_from_exp302_000_chunk16_quantile_norm_policy", + bridge_clean_8b_framewise_from_exp302_000_chunk16_quantile_norm_policy, + group="experiment", + package="_global_", +) + +bridge_clean_8b_framewise_from_exp302_000_chunk16_nonorm_policy = _build_exp( + "bridge_clean_8b_framewise_from_exp302_000_chunk16_nonorm_policy", + action_normalization=None, + resolution="256", +) +cs.store( + "bridge_clean_8b_framewise_from_exp302_000_chunk16_nonorm_policy", + bridge_clean_8b_framewise_from_exp302_000_chunk16_nonorm_policy, + group="experiment", + package="_global_", +) + +bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_fdm = _build_exp( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_fdm", + action_normalization="quantile_rot", + mode="forward_dynamics", + resolution="256", + checkpoint_path=_EXP506_000_MIDTRAIN_CKPT, +) +cs.store( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_fdm", + bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_fdm, + group="experiment", + package="_global_", +) + +bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_fdm = _build_exp( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_fdm", + action_normalization=None, + mode="forward_dynamics", + resolution="256", + checkpoint_path=_EXP506_000_MIDTRAIN_CKPT, +) +cs.store( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_fdm", + bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_fdm, + group="experiment", + package="_global_", +) + +bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_joint = _build_exp( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_joint", + action_normalization="quantile_rot", + mode="joint", + resolution="256", + checkpoint_path=_EXP506_000_MIDTRAIN_CKPT, +) +cs.store( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_joint", + bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_joint, + group="experiment", + package="_global_", +) + +bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_joint = _build_exp( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_joint", + action_normalization=None, + mode="joint", + resolution="256", + checkpoint_path=_EXP506_000_MIDTRAIN_CKPT, +) +cs.store( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_joint", + bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_joint, + group="experiment", + package="_global_", +) + + +# =========================================================================== +# 8B backward_framewise from exp302_000 — highres (480 bucket, native 640x480) +# =========================================================================== + +bridge_clean_8b_framewise_from_exp302_000_chunk16_quantile_norm_policy_highres = _build_exp( + "bridge_clean_8b_framewise_from_exp302_000_chunk16_quantile_norm_policy_highres", + action_normalization="quantile", + resolution="480", +) +cs.store( + "bridge_clean_8b_framewise_from_exp302_000_chunk16_quantile_norm_policy_highres", + bridge_clean_8b_framewise_from_exp302_000_chunk16_quantile_norm_policy_highres, + group="experiment", + package="_global_", +) + +bridge_clean_8b_framewise_from_exp302_000_chunk16_nonorm_policy_highres = _build_exp( + "bridge_clean_8b_framewise_from_exp302_000_chunk16_nonorm_policy_highres", + action_normalization=None, + resolution="480", +) +cs.store( + "bridge_clean_8b_framewise_from_exp302_000_chunk16_nonorm_policy_highres", + bridge_clean_8b_framewise_from_exp302_000_chunk16_nonorm_policy_highres, + group="experiment", + package="_global_", +) + +bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_fdm_highres = _build_exp( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_fdm_highres", + action_normalization="quantile_rot", + mode="forward_dynamics", + resolution="480", + checkpoint_path=_EXP506_000_MIDTRAIN_CKPT, +) +cs.store( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_fdm_highres", + bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_fdm_highres, + group="experiment", + package="_global_", +) + +bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_fdm_highres = _build_exp( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_fdm_highres", + action_normalization=None, + mode="forward_dynamics", + resolution="480", + checkpoint_path=_EXP506_000_MIDTRAIN_CKPT, +) +cs.store( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_fdm_highres", + bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_fdm_highres, + group="experiment", + package="_global_", +) + +bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_joint_highres = _build_exp( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_joint_highres", + action_normalization="quantile_rot", + mode="joint", + resolution="480", + checkpoint_path=_EXP506_000_MIDTRAIN_CKPT, +) +cs.store( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_joint_highres", + bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_norm_joint_highres, + group="experiment", + package="_global_", +) + +bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_joint_highres = _build_exp( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_joint_highres", + action_normalization=None, + mode="joint", + resolution="480", + checkpoint_path=_EXP506_000_MIDTRAIN_CKPT, +) +cs.store( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_joint_highres", + bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_joint_highres, + group="experiment", + package="_global_", +) + +bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_rot_norm_idm_highres = _build_exp( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_rot_norm_idm_highres", + action_normalization="quantile_rot", + mode="inverse_dynamics", + resolution="480", + checkpoint_path=_EXP506_000_MIDTRAIN_CKPT, +) +cs.store( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_rot_norm_idm_highres", + bridge_clean_8b_framewise_from_exp506_000_chunk16_quantile_rot_norm_idm_highres, + group="experiment", + package="_global_", +) + +bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_idm_highres = _build_exp( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_idm_highres", + action_normalization=None, + mode="inverse_dynamics", + resolution="480", + checkpoint_path=_EXP506_000_MIDTRAIN_CKPT, +) +cs.store( + "bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_idm_highres", + bridge_clean_8b_framewise_from_exp506_000_chunk16_nonorm_idm_highres, + group="experiment", + package="_global_", +) diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/bridge_orig_lerobot_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/bridge_orig_lerobot_experiment.py new file mode 100644 index 0000000..d08f269 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/bridge_orig_lerobot_experiment.py @@ -0,0 +1,140 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# Bridge Original LeRobot experiment — Cosmos3 2B & 8B pretrained base +# +# Base experiment (policy mode) + mode variants (fd, id, policy, i2v, joint). +# Historical 8B variant configs (pre302003, pre302000v7, mid003001) are rebuilt +# on the centralized 8B pretrained base with custom checkpoint paths. + +import copy + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.experiment.action._experiment_helpers import register_modes +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_2b import make_2b_experiment +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_8b import make_8b_experiment +from cosmos_framework.data.vfm.action.bridge_orig_lerobot_dataset import BridgeOrigLeRobotDataset +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry + +cs = ConfigStore.instance() + +# --------------------------------------------------------------------------- +# Dataset definition (shared across all modes) +# --------------------------------------------------------------------------- +BRIDGE_DATASET = [ + L(dataset_entry)( + name="bridge", + dataset=L(BridgeOrigLeRobotDataset)(chunk_length=16, split="train"), + ratio=1.0, + ), +] + +# --------------------------------------------------------------------------- +# Base experiment — 2B, 4k iters (policy mode default) +# --------------------------------------------------------------------------- +bridge_orig_lerobot = make_2b_experiment( + exp_name="bridge_orig_lerobot", + datasets=BRIDGE_DATASET, + training_iterations=4_000, +) +bridge_orig_lerobot["job"]["group"] = "bridge_orig_lerobot" + +cs.store("bridge_orig_lerobot", bridge_orig_lerobot, group="experiment", package="_global_") +register_modes(cs, "bridge_orig_lerobot", bridge_orig_lerobot, dataloader_key="action_data") + + +# --------------------------------------------------------------------------- +# Training iteration: 20000. +# --------------------------------------------------------------------------- +bridge_orig_lerobot_iter2e4 = dict( + defaults=["/experiment/bridge_orig_lerobot", "_self_"], + scheduler=dict(cycle_lengths=[20000]), + trainer=dict(max_iter=20000), +) +cs.store("bridge_orig_lerobot_iter2e4", bridge_orig_lerobot_iter2e4, group="experiment", package="_global_") +register_modes(cs, "bridge_orig_lerobot_iter2e4", bridge_orig_lerobot, dataloader_key="action_data") + + +# --------------------------------------------------------------------------- +# 8B variants — built on cosmos3_8b pretrained base +# --------------------------------------------------------------------------- + +# pre 302_003 +bridge_orig_lerobot_pre302003 = make_8b_experiment( + exp_name="bridge_orig_lerobot_pre302003", + datasets=copy.deepcopy(BRIDGE_DATASET), + training_iterations=4_000, +) +bridge_orig_lerobot_pre302003["job"]["group"] = "bridge_orig_lerobot" +bridge_orig_lerobot_pre302003["model"]["config"]["rectified_flow_training_config"]["shift"] = 3 +bridge_orig_lerobot_pre302003["model"]["config"]["rectified_flow_training_config"]["train_time_video_distribution"] = ( + "waver" +) +bridge_orig_lerobot_pre302003["model"]["config"]["diffusion_expert_config"][ + "unified_3d_mrope_temporal_modality_margin" +] = 15000 +bridge_orig_lerobot_pre302003["checkpoint"]["load_path"] = ( + "cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/" + "t2w_mot_exp302_003_qwen3_vl_8b_multires_modality_offset/checkpoints/iter_000006750" +) + +cs.store("bridge_orig_lerobot_pre302003", bridge_orig_lerobot_pre302003, group="experiment", package="_global_") +register_modes(cs, "bridge_orig_lerobot_pre302003", bridge_orig_lerobot_pre302003, dataloader_key="action_data") + + +# pre 302_000_v7 +bridge_orig_lerobot_pre302000v7 = make_8b_experiment( + exp_name="bridge_orig_lerobot_pre302000v7", + datasets=copy.deepcopy(BRIDGE_DATASET), + training_iterations=4_000, +) +bridge_orig_lerobot_pre302000v7["job"]["group"] = "bridge_orig_lerobot" +bridge_orig_lerobot_pre302000v7["model"]["config"]["rectified_flow_training_config"]["shift"] = 3 +bridge_orig_lerobot_pre302000v7["model"]["config"]["rectified_flow_training_config"][ + "train_time_video_distribution" +] = "waver" +bridge_orig_lerobot_pre302000v7["model"]["config"]["diffusion_expert_config"][ + "unified_3d_mrope_temporal_modality_margin" +] = 15000 +bridge_orig_lerobot_pre302000v7["checkpoint"]["load_path"] = ( + "cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/t2w_mot_exp302_000_qwen3_vl_8b_multires_recipe_v7/checkpoints/iter_000043500" +) + +cs.store("bridge_orig_lerobot_pre302000v7", bridge_orig_lerobot_pre302000v7, group="experiment", package="_global_") +register_modes(cs, "bridge_orig_lerobot_pre302000v7", bridge_orig_lerobot_pre302000v7, dataloader_key="action_data") + + +# mid 003_001 (base — no specific checkpoint, used by iteration variants below) +bridge_orig_lerobot_mid003001 = make_8b_experiment( + exp_name="bridge_orig_lerobot_mid003001", + datasets=copy.deepcopy(BRIDGE_DATASET), + training_iterations=4_000, +) +bridge_orig_lerobot_mid003001["job"]["group"] = "bridge_orig_lerobot" +bridge_orig_lerobot_mid003001["checkpoint"]["keys_to_skip_loading"] = [] +bridge_orig_lerobot_mid003001["model"]["config"]["rectified_flow_training_config"]["shift"] = 3 +bridge_orig_lerobot_mid003001["model"]["config"]["rectified_flow_training_config"]["train_time_video_distribution"] = ( + "waver" +) +bridge_orig_lerobot_mid003001["model"]["config"]["diffusion_expert_config"][ + "unified_3d_mrope_temporal_modality_margin" +] = 15000 + +cs.store("bridge_orig_lerobot_mid003001", bridge_orig_lerobot_mid003001, group="experiment", package="_global_") + +for i in (16250, 25000, 50000): + bridge_orig_lerobot_mid003001_iter = dict( + defaults=["/experiment/bridge_orig_lerobot_mid003001", "_self_"], + checkpoint=dict(load_path=f"cosmos3_vfm/video_uva_joint/exp003_001/checkpoints/iter_{i:09d}"), + ) + cs.store( + f"bridge_orig_lerobot_mid003001_{i}", + bridge_orig_lerobot_mid003001_iter, + group="experiment", + package="_global_", + ) + register_modes( + cs, f"bridge_orig_lerobot_mid003001_{i}", bridge_orig_lerobot_mid003001, dataloader_key="action_data" + ) diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/camera_av_joint.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/camera_av_joint.py new file mode 100644 index 0000000..66ff325 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/camera_av_joint.py @@ -0,0 +1,815 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.experiment.action.midtrain_config.action_datasets_v1p2 import ( + DATASET_AV_480, + DATASET_CAMERA_256, + DATASET_CAMERA_480, + DATASET_CAMERA_720, +) +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_8b import make_8b_experiment +from cosmos_framework.data.vfm.action.av_dataset import AVDataset +from cosmos_framework.data.vfm.action.camera_dataset_sharded import CAMERA_WDINFOS, CameraDatasetSharded +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry + +cs = ConfigStore.instance() + +_RECIPE_V5_CKPT = ( + "cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/t2w_mot_exp302_000_qwen3_vl_8b_multires_recipe_v5/checkpoints/iter_000005500/" +) +_RECIPE_V7_CKPT = ( + "cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/t2w_mot_exp302_000_qwen3_vl_8b_multires_recipe_v7/checkpoints/iter_000030000/" +) +_MIDTRAIN_CKPT = "cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/t2w_mot_exp506_000_qwen3_vl_8b_multires_recipe_midtraining_v1/checkpoints/iter_000050000/" + +_COMPILE_TOKENIZER = dict( + enabled=True, + warmup_resolutions=["256", "480", "720"], +) + + +def _make_exp(name: str, datasets: list) -> dict: + """Create an 8B experiment with common camera/AV overrides applied.""" + exp = make_8b_experiment(name, datasets=datasets, num_workers=4, training_iterations=100_000) + exp["job"]["group"] = "uva_camera" + return exp + + +### framewise experiments +# camera-only joint +camera_8b_joint_300 = _make_exp( + "camera_8b_joint_300", + [DATASET_CAMERA_256, DATASET_CAMERA_480, DATASET_CAMERA_720], +) +camera_8b_joint_300["checkpoint"]["load_path"] = _MIDTRAIN_CKPT +cs.store(group="experiment", package="_global_", name="camera_8b_joint_300", node=camera_8b_joint_300) + +# camera-only fd +DATASET_CAMERA_256_FD = L(dataset_entry)( + name="camera_256_20260501", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_01_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_02_filtered"], + CAMERA_WDINFOS["pretrained_clips_260313_10s_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt256", + max_frames=400, + translation_scale=10, + max_action_translation_norm=10, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_framewise", + ), + ratio=1, + resolution="256", +) + +DATASET_CAMERA_480_FD = L(dataset_entry)( + name="camera_480_20260501", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_01_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_02_filtered"], + CAMERA_WDINFOS["pretrained_clips_260313_10s_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt480", # 480, 720, resize to 480 + max_frames=300, + translation_scale=10, + max_action_translation_norm=10, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_framewise", + ), + ratio=1, + resolution="480", +) + +DATASET_CAMERA_720_FD = L(dataset_entry)( + name="camera_720_20260501", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_01_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_02_filtered"], + CAMERA_WDINFOS["pretrained_clips_260313_10s_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt720", # only 720 + max_frames=200, + translation_scale=10, + max_action_translation_norm=10, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_framewise", + ), + ratio=1, + resolution="720", +) +camera_8b_fd_300 = _make_exp( + "camera_8b_fd_300", + [DATASET_CAMERA_256_FD, DATASET_CAMERA_480_FD, DATASET_CAMERA_720_FD], +) +camera_8b_fd_300["checkpoint"]["load_path"] = _MIDTRAIN_CKPT +cs.store(group="experiment", package="_global_", name="camera_8b_fd_300", node=camera_8b_fd_300) + +# av-only joint +av_8b_joint_fw_011 = _make_exp( + "av_8b_joint_fw_011", + [DATASET_AV_480], +) +cs.store(group="experiment", package="_global_", name="av_8b_joint_fw_011", node=av_8b_joint_fw_011) + +# camera-av joint +camera_av_8b_joint_fw_012 = _make_exp( + "camera_av_8b_joint_fw_012", + [DATASET_CAMERA_480, DATASET_AV_480], +) +cs.store(group="experiment", package="_global_", name="camera_av_8b_joint_fw_012", node=camera_av_8b_joint_fw_012) + +# camera-av joint, continued from fw_012 iter 60k (dataset updated inplace) +camera_av_8b_joint_fw_013 = _make_exp( + "camera_av_8b_joint_fw_013", + [DATASET_CAMERA_480, DATASET_AV_480], +) +camera_av_8b_joint_fw_013["checkpoint"]["load_path"] = ( + "cosmos3_action/uva_camera/camera_av_8b_joint_fw_012/checkpoints/iter_000060000/" +) +camera_av_8b_joint_fw_013["checkpoint"]["load_training_state"] = False +camera_av_8b_joint_fw_013["checkpoint"]["keys_to_skip_loading"] = [] +cs.store(group="experiment", package="_global_", name="camera_av_8b_joint_fw_013", node=camera_av_8b_joint_fw_013) + +### new 16f framewise experiments + +DATASET_AV_480_16F = L(dataset_entry)( + name="av_480", + dataset=L(AVDataset)( + root=[ + "s3://nv-00-10206-robot/cosmos3_action_data/av_v2_02182026_wdinfo/", + "s3://nv-00-10206-robot/cosmos3_action_data/av_v2_03292026_wdinfo/", + ], + split="train", + fps=10, + mode="joint", + history_len=0.1, + future_len=1.6, + rotation_format="rot6d", + pose_convention="backward_framewise", + translation_scale=1.35, + resolution="480", + credential_path="${job.cluster.object_store_credential_data}", + shuffle=True, + include_route_in_prompt=True, + use_semantic_route_prompt=True, + ), + ratio=1, + resolution="480", +) + +# av-only joint +av_8b_16f_joint_fw_011 = _make_exp( + "av_8b_16f_joint_fw_011", + [DATASET_AV_480_16F], +) +cs.store(group="experiment", package="_global_", name="av_8b_16f_joint_fw_011", node=av_8b_16f_joint_fw_011) + + +DATASET_CAMERA_480_16F = L(dataset_entry)( + name="camera_480", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_01_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_02_filtered"], + CAMERA_WDINFOS["pretrained_clips_260313_10s_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt480", # 480, 720, resize to 480 + max_frames=17, + translation_scale=50, + mode="joint", + rotation_format="rot6d", + pose_convention="backward_framewise", + max_action_translation_norm=10, + ), + ratio=1, + resolution="480", +) + +camera_8b_16f_joint_fw_thresh_011 = _make_exp( + "camera_8b_16f_joint_fw_thresh_011", + [DATASET_CAMERA_480_16F], +) +cs.store( + group="experiment", + package="_global_", + name="camera_8b_16f_joint_fw_thresh_011", + node=camera_8b_16f_joint_fw_thresh_011, +) + +# ablate scale +DATASET_CAMERA_480_16F_THRESH_S30 = L(dataset_entry)( + name="camera_480", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_01_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_02_filtered"], + CAMERA_WDINFOS["pretrained_clips_260313_10s_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt480", # 480, 720, resize to 480 + max_frames=17, + translation_scale=30, + mode="joint", + rotation_format="rot6d", + pose_convention="backward_framewise", + max_action_translation_norm=10, + ), + ratio=1, + resolution="480", +) + +camera_8b_16f_joint_fw_s30_011 = _make_exp( + "camera_8b_16f_joint_fw_s30_011", + [DATASET_CAMERA_480_16F_THRESH_S30], +) +cs.store( + group="experiment", + package="_global_", + name="camera_8b_16f_joint_fw_s30_011", + node=camera_8b_16f_joint_fw_s30_011, +) + + +DATASET_CAMERA_480_16F_THRESH_S10 = L(dataset_entry)( + name="camera_480", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_01_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_02_filtered"], + CAMERA_WDINFOS["pretrained_clips_260313_10s_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt480", # 480, 720, resize to 480 + max_frames=17, + translation_scale=10, + mode="joint", + rotation_format="rot6d", + pose_convention="backward_framewise", + max_action_translation_norm=10, + ), + ratio=1, + resolution="480", +) + +camera_8b_16f_joint_fw_s10_011 = _make_exp( + "camera_8b_16f_joint_fw_s10_011", + [DATASET_CAMERA_480_16F_THRESH_S10], +) +cs.store( + group="experiment", + package="_global_", + name="camera_8b_16f_joint_fw_s10_011", + node=camera_8b_16f_joint_fw_s10_011, +) + +DATASET_CAMERA_480_16F_THRESH_S5 = L(dataset_entry)( + name="camera_480", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_01_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_02_filtered"], + CAMERA_WDINFOS["pretrained_clips_260313_10s_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt480", # 480, 720, resize to 480 + max_frames=17, + translation_scale=5, + mode="joint", + rotation_format="rot6d", + pose_convention="backward_framewise", + max_action_translation_norm=10, + ), + ratio=1, + resolution="480", +) + +camera_8b_16f_joint_fw_s5_011 = _make_exp( + "camera_8b_16f_joint_fw_s5_011", + [DATASET_CAMERA_480_16F_THRESH_S5], +) +cs.store( + group="experiment", + package="_global_", + name="camera_8b_16f_joint_fw_s5_011", + node=camera_8b_16f_joint_fw_s5_011, +) + + +DATASET_CAMERA_480_16F_THRESH_S1 = L(dataset_entry)( + name="camera_480", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_01_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_02_filtered"], + CAMERA_WDINFOS["pretrained_clips_260313_10s_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt480", # 480, 720, resize to 480 + max_frames=17, + translation_scale=1, + mode="joint", + rotation_format="rot6d", + pose_convention="backward_framewise", + max_action_translation_norm=10, + ), + ratio=1, + resolution="480", +) + +camera_8b_16f_joint_fw_s1_011 = _make_exp( + "camera_8b_16f_joint_fw_s1_011", + [DATASET_CAMERA_480_16F_THRESH_S1], +) +cs.store( + group="experiment", + package="_global_", + name="camera_8b_16f_joint_fw_s1_011", + node=camera_8b_16f_joint_fw_s1_011, +) + +### camera fd-only experiments + +DATASET_CAMERA_FD_480 = L(dataset_entry)( + name="camera_480", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_01_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_02_filtered"], + CAMERA_WDINFOS["pretrained_clips_260313_10s_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt480", # 480, 720, resize to 480 + max_frames=149, + translation_scale=10, + max_action_translation_norm=10, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_framewise", + ), + ratio=1, + resolution="480", +) + +camera_8b_fd_fw_step1_001 = _make_exp( + "camera_8b_fd_fw_step1_001", + [DATASET_CAMERA_FD_480], +) +cs.store( + group="experiment", + package="_global_", + name="camera_8b_fd_fw_step1_001", + node=camera_8b_fd_fw_step1_001, +) + +DATASET_CAMERA_FD_LONG_256 = L(dataset_entry)( + name="camera_256", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_01_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_02_filtered"], + CAMERA_WDINFOS["pretrained_clips_260313_10s_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt256", + max_frames=400, + translation_scale=10, + max_action_translation_norm=10, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_framewise", + ), + ratio=1, + resolution="256", +) + +DATASET_CAMERA_FD_LONG_480 = L(dataset_entry)( + name="camera_480", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_01_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_02_filtered"], + CAMERA_WDINFOS["pretrained_clips_260313_10s_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt480", # 480, 720, resize to 480 + max_frames=300, + translation_scale=10, + max_action_translation_norm=10, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_framewise", + ), + ratio=1, + resolution="480", +) + +DATASET_CAMERA_FD_LONG_720 = L(dataset_entry)( + name="camera_720", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_01_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_02_filtered"], + CAMERA_WDINFOS["pretrained_clips_260313_10s_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt720", # 480, 720, resize to 480 + max_frames=200, + translation_scale=10, + max_action_translation_norm=10, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_framewise", + ), + ratio=1, + resolution="720", +) + +camera_8b_fd_long_001 = _make_exp( + "camera_8b_fd_long_001", + [ + DATASET_CAMERA_FD_LONG_256, + DATASET_CAMERA_FD_LONG_480, + DATASET_CAMERA_FD_LONG_720, + ], +) +camera_8b_fd_long_001["checkpoint"]["load_path"] = ( + "cosmos3_action/uva_camera/camera_8b_fd_fw_step1_001/checkpoints/iter_000075000/" +) +camera_8b_fd_long_001["checkpoint"]["load_training_state"] = False +camera_8b_fd_long_001["checkpoint"]["keys_to_skip_loading"] = [] +cs.store( + group="experiment", + package="_global_", + name="camera_8b_fd_long_001", + node=camera_8b_fd_long_001, +) + +### AV ID-only experiments +DATASET_AV_ID_480 = L(dataset_entry)( + name="av_id_480", + dataset=L(AVDataset)( + root=[ + "s3://nv-00-10206-robot/cosmos3_action_data/av_v2_02182026_wdinfo/", + "s3://nv-00-10206-robot/cosmos3_action_data/av_v2_03292026_wdinfo/", + ], + split="train", + fps=10, + mode="inverse_dynamics", + history_len=0.1, + future_len=6.0, + rotation_format="rot6d", + pose_convention="backward_framewise", + translation_scale=1.35, + max_action_translation_norm=10, + resolution="480", + credential_path="${job.cluster.object_store_credential_data}", + shuffle=True, + include_route_in_prompt=False, + use_semantic_route_prompt=False, + align_opencv_pose=False, + ), + ratio=1, + resolution="480", +) + +av_8b_id_fw_step1_003 = _make_exp( + "av_8b_id_fw_step1_003", + [DATASET_AV_ID_480], +) +cs.store(group="experiment", package="_global_", name="av_8b_id_fw_step1_003", node=av_8b_id_fw_step1_003) + + +DATASET_AV_JOINT_480 = L(dataset_entry)( + name="av_joint_480", + dataset=L(AVDataset)( + root=[ + "s3://nv-00-10206-robot/cosmos3_action_data/av_v2_02182026_wdinfo/", + "s3://nv-00-10206-robot/cosmos3_action_data/av_v2_03292026_wdinfo/", + ], + split="train", + fps=10, + mode="joint", + history_len=0.1, + future_len=6.0, + rotation_format="rot6d", + pose_convention="backward_framewise", + translation_scale=1.35, + max_action_translation_norm=10, + resolution="480", + credential_path="${job.cluster.object_store_credential_data}", + shuffle=True, + include_route_in_prompt=False, + use_semantic_route_prompt=False, + align_opencv_pose=False, + ), + ratio=1, + resolution="480", +) + +av_8b_joint_fw_step1_003 = _make_exp( + "av_8b_joint_fw_step1_003", + [DATASET_AV_JOINT_480], +) +cs.store(group="experiment", package="_global_", name="av_8b_joint_fw_step1_003", node=av_8b_joint_fw_step1_003) + +# same exp with 1 node +av_8b_joint_fw_step1_004 = _make_exp( + "av_8b_joint_fw_step1_004", + [DATASET_AV_JOINT_480], +) +cs.store(group="experiment", package="_global_", name="av_8b_joint_fw_step1_004", node=av_8b_joint_fw_step1_004) + +### Rotation format and scale experiments + +DATASET_AV_JOINT_480_AA = L(dataset_entry)( + name="av_joint_480", + dataset=L(AVDataset)( + root=[ + "s3://nv-00-10206-robot/cosmos3_action_data/av_v2_02182026_wdinfo/", + "s3://nv-00-10206-robot/cosmos3_action_data/av_v2_03292026_wdinfo/", + ], + split="train", + fps=10, + mode="joint", + history_len=0.1, + future_len=6.0, + rotation_format="axisangle", + pose_convention="backward_framewise", + translation_scale=1.35, + rotation_scale=1.35 * 50, + max_action_translation_norm=10, + resolution="480", + credential_path="${job.cluster.object_store_credential_data}", + shuffle=True, + include_route_in_prompt=False, + use_semantic_route_prompt=False, + align_opencv_pose=False, + ), + ratio=1, + resolution="480", +) + +av_8b_joint_fwaa_001 = _make_exp( + "av_8b_joint_fwaa_001", + [DATASET_AV_JOINT_480_AA], +) +cs.store(group="experiment", package="_global_", name="av_8b_joint_fwaa_001", node=av_8b_joint_fwaa_001) + +DATASET_AV_ID_480_AA = L(dataset_entry)( + name="av_id_480", + dataset=L(AVDataset)( + root=[ + "s3://nv-00-10206-robot/cosmos3_action_data/av_v2_02182026_wdinfo/", + "s3://nv-00-10206-robot/cosmos3_action_data/av_v2_03292026_wdinfo/", + ], + split="train", + fps=10, + mode="inverse_dynamics", + history_len=0.1, + future_len=6.0, + rotation_format="axisangle", + pose_convention="backward_framewise", + translation_scale=1.35, + rotation_scale=1.35 * 50, + max_action_translation_norm=10, + resolution="480", + credential_path="${job.cluster.object_store_credential_data}", + shuffle=True, + include_route_in_prompt=False, + use_semantic_route_prompt=False, + align_opencv_pose=False, + ), + ratio=1, + resolution="480", +) + +av_8b_id_fwaa_001 = _make_exp( + "av_8b_id_fwaa_001", + [DATASET_AV_ID_480_AA], +) +cs.store(group="experiment", package="_global_", name="av_8b_id_fwaa_001", node=av_8b_id_fwaa_001) + +DATASET_CAMERA_JOINT_480_AA = L(dataset_entry)( + name="camera_480", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_01_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_02_filtered"], + CAMERA_WDINFOS["pretrained_clips_260313_10s_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt480", # 480, 720, resize to 480 + max_frames=149, + translation_scale=10, + rotation_scale=10 * 8, + max_action_translation_norm=10, + mode="joint", + rotation_format="axisangle", + pose_convention="backward_framewise", + ), + ratio=1, + resolution="480", +) + +camera_8b_joint_fwaa_001 = _make_exp( + "camera_8b_joint_fwaa_001", + [DATASET_CAMERA_JOINT_480_AA], +) +cs.store( + group="experiment", + package="_global_", + name="camera_8b_joint_fwaa_001", + node=camera_8b_joint_fwaa_001, +) + +DATASET_CAMERA_FD_480_AA = L(dataset_entry)( + name="camera_fd_480", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_01_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_02_filtered"], + CAMERA_WDINFOS["pretrained_clips_260313_10s_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt480", # 480, 720, resize to 480 + max_frames=149, + translation_scale=10, + rotation_scale=10 * 8, + max_action_translation_norm=10, + mode="forward_dynamics", + rotation_format="axisangle", + pose_convention="backward_framewise", + ), + ratio=1, + resolution="480", +) + +camera_8b_fd_fwaa_001 = _make_exp( + "camera_8b_fd_fwaa_001", + [DATASET_CAMERA_FD_480_AA], +) +cs.store( + group="experiment", + package="_global_", + name="camera_8b_fd_fwaa_001", + node=camera_8b_fd_fwaa_001, +) + +DATASET_CAMERA_FD_480_AA_WOROTSCL = L(dataset_entry)( + name="camera_fd_480", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_01_filtered"], + CAMERA_WDINFOS["pretrained_clips_260325_500k_02_filtered"], + CAMERA_WDINFOS["pretrained_clips_260313_10s_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt480", # 480, 720, resize to 480 + max_frames=149, + translation_scale=10, + rotation_scale=10, + max_action_translation_norm=10, + mode="forward_dynamics", + rotation_format="axisangle", + pose_convention="backward_framewise", + ), + ratio=1, + resolution="480", +) + +camera_8b_fd_fwaa_worotscl_001 = _make_exp( + "camera_8b_fd_fwaa_worotscl_001", + [DATASET_CAMERA_FD_480_AA_WOROTSCL], +) +cs.store( + group="experiment", + package="_global_", + name="camera_8b_fd_fwaa_worotscl_001", + node=camera_8b_fd_fwaa_worotscl_001, +) + +### SDG data ablation +# DATASET_CAMERA_PRETRAIN_480 = L(dataset_entry)( +# name="camera_480", +# dataset=L(CameraDatasetSharded)( +# wdinfo_paths=[ +# CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], +# CAMERA_WDINFOS["pretrained_clips_260325_500k_01_filtered"], +# CAMERA_WDINFOS["pretrained_clips_260325_500k_02_filtered"], +# CAMERA_WDINFOS["pretrained_clips_260313_10s_100k_filtered"], +# ], +# split="train", +# shuffle=True, +# fix_caption=False, +# wdinfo_resolution="gt480", # 480, 720, resize to 480 +# max_frames=61, +# translation_scale=50, +# max_action_translation_norm=10, +# mode="joint", +# rotation_format="6D", +# rel_pose_format="backward_framewise", +# ), +# ratio=1, +# resolution="480", +# ) + +# DATASET_CAMERA_ENDEAVOR_480 = L(dataset_entry)( +# name="camera_endeavor_480", +# dataset=L(CameraDatasetSharded)( +# wdinfo_paths=[ +# CAMERA_WDINFOS["endeavor_forever_480"], +# ], +# split="train", +# shuffle=True, +# fix_caption=False, +# wdinfo_resolution="gt480", # 480, 720, resize to 480 +# max_frames=61, +# translation_scale=50, +# max_action_translation_norm=10, +# mode="joint", +# rotation_format="6D", +# rel_pose_format="backward_framewise", +# ), +# ratio=1, +# resolution="480", +# ) + +# DATASET_CAMERA_SYNHUMAN_480 = L(dataset_entry)( +# name="camera_synhuman_480", +# dataset=L(CameraDatasetSharded)( +# wdinfo_paths=[ +# CAMERA_WDINFOS["synhuman_20260223_480"], +# ], +# split="train", +# shuffle=True, +# fix_caption=False, +# wdinfo_resolution="gt480", # 480, 720, resize to 480 +# max_frames=61, +# translation_scale=50, +# max_action_translation_norm=10, +# mode="joint", +# rotation_format="6D", +# rel_pose_format="backward_framewise", +# ), +# ratio=1, +# resolution="480", +# ) diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/camera_fwd_sharded.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/camera_fwd_sharded.py new file mode 100644 index 0000000..90f643e --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/camera_fwd_sharded.py @@ -0,0 +1,538 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_8b import make_8b_experiment +from cosmos_framework.data.vfm.action.camera_dataset_sharded import CAMERA_WDINFOS, CameraDatasetSharded +from cosmos_framework.data.vfm.action.dataloaders import InfiniteDataLoader +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry, wrap_dataset +from cosmos_framework.data.vfm.joint_dataloader import IterativeJointDataLoader + +cs = ConfigStore.instance() + +RECIPE_V5_CHECKPOINT = ( + "cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/t2w_mot_exp302_000_qwen3_vl_8b_multires_recipe_v5/checkpoints/iter_000005500/" +) + +_COMPILE_TOKENIZER_CONFIG = dict( + enabled=True, + warmup_resolutions=["256", "480", "720"], +) + + +def _apply_camera_overrides(exp: dict) -> dict: + """Apply overrides common to all camera forward-dynamics sharded experiments.""" + exp["job"]["group"] = "uva_camera" + exp["model"]["config"]["max_action_dim"] = 32 + exp["model"]["config"]["num_embodiment_domains"] = 32 + exp["trainer"]["callbacks"]["compile_tokenizer"] = _COMPILE_TOKENIZER_CONFIG.copy() + exp["trainer"]["compile_config"]["recompile_limit"] = 100 + return exp + + +def _make_multires_dataloader(datasets: list) -> L: + """Build an IterativeJointDataLoader for multi-resolution camera datasets.""" + return L(IterativeJointDataLoader)( + dataloaders={ + "camera_data": dict( + dataloader=L(InfiniteDataLoader)( + dataset=L(wrap_dataset)( + list_of_datasets=datasets, + tokenizer_config="${model.config.vlm_config.tokenizer}", + cfg_dropout_rate=0.1, + max_action_dim="${model.config.max_action_dim}", + ), + batch_size=4, + num_workers=4, + pin_memory=True, + drop_last=True, + use_deterministic_seed=False, + in_order=False, + multiprocessing_context="spawn", + ), + ratio=1, + ), + }, + tokenizer_spatial_compression_factor="${model.config.tokenizer.spatial_compression_factor}", + tokenizer_temporal_compression_factor="${model.config.tokenizer.temporal_compression_factor}", + patch_spatial="${model.config.diffusion_expert_config.patch_spatial}", + max_sequence_length="${model.config.max_num_tokens_after_packing}", + ) + + +# --------------------------------------------------------------------------- +# Experiment 1: 8b multires (modality_offset checkpoint, 480p, no waver) +# s3://nv-00-10206-checkpoint-experiments/cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/t2w_mot_exp302_003_qwen3_vl_8b_multires_modality_offset/checkpoints/iter_000006750/ +# --------------------------------------------------------------------------- +camera_8b_480res_fd_sharded_v2 = _apply_camera_overrides( + make_8b_experiment("camera_8b_480res_fd_sharded_v2", datasets=[], max_action_dim=32, training_iterations=100_000) +) +camera_8b_480res_fd_sharded_v2["model"]["config"]["resolution"] = "480" +del camera_8b_480res_fd_sharded_v2["model"]["config"]["rectified_flow_training_config"]["train_time_video_distribution"] +camera_8b_480res_fd_sharded_v2["checkpoint"]["save_iter"] = 500 +camera_8b_480res_fd_sharded_v2["checkpoint"]["load_path"] = ( + "cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/" + "t2w_mot_exp302_003_qwen3_vl_8b_multires_modality_offset/" + "checkpoints/iter_000006750/" +) +camera_8b_480res_fd_sharded_v2["dataloader_train"] = L(IterativeJointDataLoader)( + dataloaders={ + "camera_data": dict( + dataloader=L(InfiniteDataLoader)( + dataset=L(wrap_dataset)( + list_of_datasets=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["tartanair_480"], + CAMERA_WDINFOS["endeavor_forever_480"], + CAMERA_WDINFOS["synhuman_20251218_480"], + CAMERA_WDINFOS["pretrained_clips_260131_10k_480"], + ], + split="train", + shuffle=True, + fix_caption=True, + max_frames=149, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_anchored", + ), + tokenizer_config="${model.config.vlm_config.tokenizer}", + cfg_dropout_rate=0.1, + max_action_dim="${model.config.max_action_dim}", + ), + batch_size=4, + shuffle=False, + num_workers=4, + pin_memory=True, + drop_last=True, + in_order=False, + seed=0, + ), + ratio=1, + ), + }, + tokenizer_spatial_compression_factor="${model.config.tokenizer.spatial_compression_factor}", + tokenizer_temporal_compression_factor="${model.config.tokenizer.temporal_compression_factor}", + patch_spatial="${model.config.diffusion_expert_config.patch_spatial}", + max_sequence_length="${model.config.max_num_tokens_after_packing}", +) +camera_8b_480res_fd_sharded_v2["dataloader_val"] = L(IterativeJointDataLoader)( + dataloaders={ + "camera_data": dict( + dataloader=L(InfiniteDataLoader)( + dataset=L(wrap_dataset)( + list_of_datasets=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["tartanair_480"], + CAMERA_WDINFOS["endeavor_forever_480"], + CAMERA_WDINFOS["synhuman_20251218_480"], + CAMERA_WDINFOS["pretrained_clips_260131_10k_480"], + ], + split="val", + shuffle=False, + fix_caption=True, + max_frames=149, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_anchored", + ), + tokenizer_config="${model.config.vlm_config.tokenizer}", + cfg_dropout_rate=0.1, + max_action_dim="${model.config.max_action_dim}", + ), + batch_size=1, + shuffle=False, + num_workers=0, + pin_memory=True, + drop_last=True, + in_order=False, + seed=0, + ), + ratio=1, + ), + }, + tokenizer_spatial_compression_factor="${model.config.tokenizer.spatial_compression_factor}", + tokenizer_temporal_compression_factor="${model.config.tokenizer.temporal_compression_factor}", + patch_spatial="${model.config.diffusion_expert_config.patch_spatial}", + max_sequence_length="${model.config.max_num_tokens_after_packing}", +) + +cs.store( + group="experiment", + package="_global_", + name="camera_8b_480res_fd_sharded_v2", + node=camera_8b_480res_fd_sharded_v2, +) + + +# --------------------------------------------------------------------------- +# Experiment 2: 480p, recipe_v5, pretrain10k with captions +# base model: +# s3://nv-00-10206-checkpoint-experiments/cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/t2w_mot_exp302_000_qwen3_vl_8b_multires_recipe_v5/checkpoints/iter_000005500/model/ +# --------------------------------------------------------------------------- +camera_8b_480res_fd_sharded_v3_pretrain10k_wcap = _apply_camera_overrides( + make_8b_experiment( + "camera_8b_480res_fd_sharded_v3_pretrain10k_wcap", datasets=[], max_action_dim=32, training_iterations=100_000 + ) +) +camera_8b_480res_fd_sharded_v3_pretrain10k_wcap["model"]["config"]["resolution"] = "480" +camera_8b_480res_fd_sharded_v3_pretrain10k_wcap["checkpoint"]["save_iter"] = 1000 +camera_8b_480res_fd_sharded_v3_pretrain10k_wcap["checkpoint"]["load_path"] = RECIPE_V5_CHECKPOINT +camera_8b_480res_fd_sharded_v3_pretrain10k_wcap["dataloader_train"] = L(IterativeJointDataLoader)( + dataloaders={ + "camera_data": dict( + dataloader=L(InfiniteDataLoader)( + dataset=L(wrap_dataset)( + list_of_datasets=L(CameraDatasetSharded)( + wdinfo_paths=[ + # CAMERA_WDINFOS["tartanair_480"], + # CAMERA_WDINFOS["endeavor_forever_480"], + # CAMERA_WDINFOS["synhuman_20251218_480"], + CAMERA_WDINFOS["pretrained_clips_260131_10k_480"], + ], + split="train", + shuffle=True, + fix_caption=False, + max_frames=149, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_anchored", + discard_varying_intrinsics=False, + ), + tokenizer_config="${model.config.vlm_config.tokenizer}", + cfg_dropout_rate=0.1, + max_action_dim="${model.config.max_action_dim}", + ), + batch_size=4, + shuffle=False, + num_workers=4, + pin_memory=True, + drop_last=True, + in_order=False, + seed=0, + ), + ratio=1, + ), + }, + tokenizer_spatial_compression_factor="${model.config.tokenizer.spatial_compression_factor}", + tokenizer_temporal_compression_factor="${model.config.tokenizer.temporal_compression_factor}", + patch_spatial="${model.config.diffusion_expert_config.patch_spatial}", + max_sequence_length="${model.config.max_num_tokens_after_packing}", +) +camera_8b_480res_fd_sharded_v3_pretrain10k_wcap["dataloader_val"] = L(IterativeJointDataLoader)( + dataloaders={ + "camera_data": dict( + dataloader=L(InfiniteDataLoader)( + dataset=L(wrap_dataset)( + list_of_datasets=L(CameraDatasetSharded)( + wdinfo_paths=[ + # CAMERA_WDINFOS["tartanair_480"], + # CAMERA_WDINFOS["endeavor_forever_480"], + # CAMERA_WDINFOS["synhuman_20251218_480"], + CAMERA_WDINFOS["pretrained_clips_260131_10k_480"], + ], + split="val", + shuffle=False, + fix_caption=False, + max_frames=149, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_anchored", + discard_varying_intrinsics=False, + ), + tokenizer_config="${model.config.vlm_config.tokenizer}", + cfg_dropout_rate=0.1, + max_action_dim="${model.config.max_action_dim}", + ), + batch_size=1, + shuffle=False, + num_workers=0, + pin_memory=True, + drop_last=True, + in_order=False, + seed=0, + ), + ratio=1, + ), + }, + tokenizer_spatial_compression_factor="${model.config.tokenizer.spatial_compression_factor}", + tokenizer_temporal_compression_factor="${model.config.tokenizer.temporal_compression_factor}", + patch_spatial="${model.config.diffusion_expert_config.patch_spatial}", + max_sequence_length="${model.config.max_num_tokens_after_packing}", +) + +cs.store( + group="experiment", + package="_global_", + name="camera_8b_480res_fd_sharded_v3_pretrain10k_wcap", + node=camera_8b_480res_fd_sharded_v3_pretrain10k_wcap, +) + + +# --------------------------------------------------------------------------- +# Dataset constants: pretrain 100k filtered +# --------------------------------------------------------------------------- +DATASET_CAMERA_P100K_FILTERED_256 = L(dataset_entry)( + name="camera_p100k_filtered_256", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="all", # 256, 480, 720, resize to 256 + max_frames=400, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_anchored", + discard_varying_intrinsics=False, + ), + ratio=3, + resolution="256", +) + +DATASET_CAMERA_P100K_FILTERED_480 = L(dataset_entry)( + name="camera_p100k_filtered_480", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt480", # 480, 720, resize to 480 + max_frames=300, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_anchored", + discard_varying_intrinsics=False, + ), + ratio=2, + resolution="480", +) + +DATASET_CAMERA_P100K_FILTERED_720 = L(dataset_entry)( + name="camera_p100k_filtered_720", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt720", # only 720 + max_frames=200, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_anchored", + discard_varying_intrinsics=False, + ), + ratio=1, + resolution="720", +) + + +# --------------------------------------------------------------------------- +# Experiment 3: pretrain 100k filtered (720p, recipe_v5) +# --------------------------------------------------------------------------- +camera_8b_fd_sharded_v3_p100k_filtered = _apply_camera_overrides( + make_8b_experiment( + "camera_8b_fd_sharded_v3_p100k_filtered", datasets=[], max_action_dim=32, training_iterations=100_000 + ) +) +camera_8b_fd_sharded_v3_p100k_filtered["checkpoint"]["save_iter"] = 1000 +camera_8b_fd_sharded_v3_p100k_filtered["checkpoint"]["load_path"] = RECIPE_V5_CHECKPOINT +camera_8b_fd_sharded_v3_p100k_filtered["dataloader_train"] = _make_multires_dataloader( + [ + DATASET_CAMERA_P100K_FILTERED_256, + DATASET_CAMERA_P100K_FILTERED_480, + DATASET_CAMERA_P100K_FILTERED_720, + ] +) + +cs.store( + group="experiment", + package="_global_", + name="camera_8b_fd_sharded_v3_p100k_filtered", + node=camera_8b_fd_sharded_v3_p100k_filtered, +) + + +# --------------------------------------------------------------------------- +# Dataset constants: pretrain 100k filtered framewise +# --------------------------------------------------------------------------- +DATASET_CAMERA_P100K_FILTERED_FRAMEWISE_256 = L(dataset_entry)( + name="camera_p100k_filtered_framewise_256", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="all", # 256, 480, 720, resize to 256 + max_frames=400, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_framewise", + discard_varying_intrinsics=False, + ), + ratio=3, + resolution="256", +) + +DATASET_CAMERA_P100K_FILTERED_FRAMEWISE_480 = L(dataset_entry)( + name="camera_p100k_filtered_framewise_480", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt480", # 480, 720, resize to 480 + max_frames=300, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_framewise", + discard_varying_intrinsics=False, + ), + ratio=2, + resolution="480", +) + +DATASET_CAMERA_P100K_FILTERED_FRAMEWISE_720 = L(dataset_entry)( + name="camera_p100k_filtered_framewise_720", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt720", # only 720 + max_frames=200, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_framewise", + discard_varying_intrinsics=False, + ), + ratio=1, + resolution="720", +) + + +# --------------------------------------------------------------------------- +# Experiment 4: pretrain 100k filtered framewise (720p, recipe_v5) +# --------------------------------------------------------------------------- +camera_8b_fd_sharded_v3_p100k_filtered_fw = _apply_camera_overrides( + make_8b_experiment( + "camera_8b_fd_sharded_v3_p100k_filtered_fw", datasets=[], max_action_dim=32, training_iterations=100_000 + ) +) +camera_8b_fd_sharded_v3_p100k_filtered_fw["checkpoint"]["save_iter"] = 1000 +camera_8b_fd_sharded_v3_p100k_filtered_fw["checkpoint"]["load_path"] = RECIPE_V5_CHECKPOINT +camera_8b_fd_sharded_v3_p100k_filtered_fw["dataloader_train"] = _make_multires_dataloader( + [ + DATASET_CAMERA_P100K_FILTERED_FRAMEWISE_256, + DATASET_CAMERA_P100K_FILTERED_FRAMEWISE_480, + DATASET_CAMERA_P100K_FILTERED_FRAMEWISE_720, + ] +) + +cs.store( + group="experiment", + package="_global_", + name="camera_8b_fd_sharded_v3_p100k_filtered_fw", + node=camera_8b_fd_sharded_v3_p100k_filtered_fw, +) + + +# --------------------------------------------------------------------------- +# Dataset constants: pretrain 100k +# --------------------------------------------------------------------------- +DATASET_CAMERA_P100K_256 = L(dataset_entry)( + name="camera_p100k_256", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="all", # 256, 480, 720, resize to 256 + max_frames=400, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_anchored", + discard_varying_intrinsics=False, + ), + ratio=3, + resolution="256", +) + +DATASET_CAMERA_P100K_480 = L(dataset_entry)( + name="camera_p100k_480", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt480", # 480, 720, resize to 480 + max_frames=300, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_anchored", + discard_varying_intrinsics=False, + ), + ratio=2, + resolution="480", +) + +DATASET_CAMERA_P100K_720 = L(dataset_entry)( + name="camera_p100k_720", + dataset=L(CameraDatasetSharded)( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260307_100k"], + ], + split="train", + shuffle=True, + fix_caption=False, + wdinfo_resolution="gt720", # only 720 + max_frames=200, + mode="forward_dynamics", + rotation_format="rot6d", + pose_convention="backward_anchored", + discard_varying_intrinsics=False, + ), + ratio=1, + resolution="720", +) + + +# --------------------------------------------------------------------------- +# Experiment 6: pretrain 100k (720p, recipe_v5) +# --------------------------------------------------------------------------- +camera_8b_fd_sharded_v3_p100k = _apply_camera_overrides( + make_8b_experiment("camera_8b_fd_sharded_v3_p100k", datasets=[], max_action_dim=32, training_iterations=100_000) +) +camera_8b_fd_sharded_v3_p100k["checkpoint"]["save_iter"] = 1000 +camera_8b_fd_sharded_v3_p100k["checkpoint"]["load_path"] = RECIPE_V5_CHECKPOINT +camera_8b_fd_sharded_v3_p100k["dataloader_train"] = _make_multires_dataloader( + [ + DATASET_CAMERA_P100K_256, + DATASET_CAMERA_P100K_480, + DATASET_CAMERA_P100K_720, + ] +) + +cs.store( + group="experiment", + package="_global_", + name="camera_8b_fd_sharded_v3_p100k", + node=camera_8b_fd_sharded_v3_p100k, +) diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/droid_lerobot_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/droid_lerobot_experiment.py new file mode 100644 index 0000000..49398e7 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/droid_lerobot_experiment.py @@ -0,0 +1,107 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.experiment.action._experiment_helpers import register_modes +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_2b import make_2b_experiment +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_8b import make_8b_experiment +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_32b import make_32b_experiment +from cosmos_framework.data.vfm.action.droid_lerobot_dataset import DROIDLeRobotDataset +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry + +cs = ConfigStore.instance() + +DROID_DATASET = [ + L(dataset_entry)( + name="droid", + dataset=L(DROIDLeRobotDataset)( + root="", + viewpoint="wrist_view", + use_success_only=True, + video_mode="wrist", + action_space="ee_pose_delta", + use_filter_dict=True, + ), + ) +] + + +droid_lerobot_2b = make_2b_experiment( + exp_name="${now:%Y-%m-%d_%H-%M-%S}", + datasets=DROID_DATASET, + batch_size=19, + num_workers=39, + training_iterations=4_000, + use_deterministic_seed=True, +) +for i, d in enumerate(droid_lerobot_2b["defaults"]): + if isinstance(d, dict) and "override /callbacks" in d: + droid_lerobot_2b["defaults"][i] = { + "override /callbacks": ["basic", "optimization", "job_monitor", "training_stats"] + } + break +droid_lerobot_2b["model"]["config"]["tokenizer"]["encode_exact_durations"] = [17] +droid_lerobot_2b["model"]["config"]["vlm_config"]["pretrained_weights"]["enabled"] = False +droid_lerobot_2b["model"]["config"]["max_num_tokens_after_packing"] = -1 +droid_lerobot_2b["scheduler"]["warm_up_steps"] = [0] +droid_lerobot_2b["dataloader_train"]["dataloaders"]["action_data"]["dataloader"]["dataset"]["list_of_datasets"][0][ + "resolution" +] = "256" +droid_lerobot_2b["dataloader_train"]["max_sequence_length"] = None +droid_lerobot_2b["dataloader_train"]["max_samples_per_batch"] = 256 +droid_lerobot_2b["job"]["group"] = "cosmos3_action_2b_posttrain" +droid_lerobot_2b["trainer"]["callbacks"]["compile_tokenizer"]["warmup_resolutions"] = ["256"] +droid_lerobot_2b["checkpoint"]["load_from_object_store"] = {"bucket": "nv-00-10206-checkpoint"} + +cs.store("droid_lerobot_2b", droid_lerobot_2b, group="experiment", package="_global_") +register_modes(cs, "droid_lerobot_2b", droid_lerobot_2b, dataloader_key="action_data") + + +droid_lerobot_8b = make_8b_experiment( + exp_name="${now:%Y-%m-%d_%H-%M-%S}", + datasets=DROID_DATASET, + batch_size=19, + num_workers=39, + training_iterations=4_000, + use_deterministic_seed=True, +) +droid_lerobot_8b["model"]["config"]["tokenizer"]["encode_exact_durations"] = [17] +droid_lerobot_8b["model"]["config"]["vlm_config"]["pretrained_weights"]["enabled"] = False +droid_lerobot_8b["model"]["config"]["max_num_tokens_after_packing"] = -1 +droid_lerobot_8b["dataloader_train"]["dataloaders"]["action_data"]["dataloader"]["dataset"]["list_of_datasets"][0][ + "resolution" +] = "256" +droid_lerobot_8b["dataloader_train"]["max_sequence_length"] = None +droid_lerobot_8b["dataloader_train"]["max_samples_per_batch"] = 256 +droid_lerobot_8b["job"]["group"] = "cosmos3_action_8b_posttrain" +droid_lerobot_8b["trainer"]["callbacks"]["compile_tokenizer"]["warmup_resolutions"] = ["256"] +droid_lerobot_8b["checkpoint"]["load_from_object_store"] = {"bucket": "nv-00-10206-checkpoint"} + +cs.store("droid_lerobot_8b", droid_lerobot_8b, group="experiment", package="_global_") +register_modes(cs, "droid_lerobot_8b", droid_lerobot_8b, dataloader_key="action_data") + + +droid_lerobot_32b = make_32b_experiment( + exp_name="${now:%Y-%m-%d_%H-%M-%S}", + datasets=DROID_DATASET, + batch_size=19, + num_workers=39, + training_iterations=4_000, + use_deterministic_seed=True, +) +droid_lerobot_32b["model"]["config"]["tokenizer"]["encode_exact_durations"] = [17] +droid_lerobot_32b["model"]["config"]["vlm_config"]["pretrained_weights"]["enabled"] = False +droid_lerobot_32b["model"]["config"]["max_num_tokens_after_packing"] = -1 +droid_lerobot_32b["dataloader_train"]["dataloaders"]["action_data"]["dataloader"]["dataset"]["list_of_datasets"][0][ + "resolution" +] = "256" +droid_lerobot_32b["dataloader_train"]["max_sequence_length"] = None +droid_lerobot_32b["dataloader_train"]["max_samples_per_batch"] = 256 +droid_lerobot_32b["job"]["group"] = "cosmos3_action_32b_posttrain" +droid_lerobot_32b["trainer"]["callbacks"]["compile_tokenizer"]["warmup_resolutions"] = ["256"] +droid_lerobot_32b["checkpoint"]["load_from_object_store"] = {"bucket": "nv-00-10206-checkpoint"} + +cs.store("droid_lerobot_32b", droid_lerobot_32b, group="experiment", package="_global_") +register_modes(cs, "droid_lerobot_32b", droid_lerobot_32b, dataloader_key="action_data") diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/dummy_dataset_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/dummy_dataset_experiment.py new file mode 100644 index 0000000..66f3202 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/dummy_dataset_experiment.py @@ -0,0 +1,46 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# Dummy dataset experiment — Cosmos3 2B pretrained base (for debugging) +# +# CUDA_VISIBLE_DEVICES=0 PYTHONPATH=. torchrun --nproc_per_node=1 \ +# --master_port=12341 cosmos_framework/scripts/train.py \ +# --config=cosmos_framework/configs/base/config.py \ +# -- experiment=action_dummy_dataset_exp + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_2b import make_2b_experiment +from cosmos_framework.data.vfm.action.dummy_dataset import DummyDataset +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry + +cs = ConfigStore.instance() + +# --------------------------------------------------------------------------- +# Dataset definition +# --------------------------------------------------------------------------- +DUMMY_TRAIN_DATASET = [ + L(dataset_entry)( + name="default", + dataset=L(DummyDataset)(length=1e6), + ratio=1.0, + ), +] + +# --------------------------------------------------------------------------- +# Base experiment — 2B, long run for debugging +# --------------------------------------------------------------------------- +action_dummy_dataset_exp = make_2b_experiment( + exp_name="dummy_dataset_exp", + datasets=DUMMY_TRAIN_DATASET, + batch_size=1, + num_workers=2, + training_iterations=1_000_000, +) + +# --- Experiment-specific overrides --- +action_dummy_dataset_exp["job"]["group"] = "debugging" +action_dummy_dataset_exp["checkpoint"]["save_iter"] = 100_000_000 + +cs.store(group="experiment", package="_global_", name="action_dummy_dataset_exp", node=action_dummy_dataset_exp) diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/embodiment_b_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/embodiment_b_experiment.py new file mode 100644 index 0000000..72b3392 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/embodiment_b_experiment.py @@ -0,0 +1,101 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# Embodiment_b experiment — Cosmos3 2B pretrained base +# +# Base experiment (policy mode) + mode variants (fd, id, policy, i2v, joint). +# Plus wrist camera and keep-aspect-ratio variants. + + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.experiment.action._experiment_helpers import register_modes +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_2b import make_2b_experiment +from cosmos_framework.data.vfm.action.embodiment_b_dataset import Embodiment_bDataset +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry + +cs = ConfigStore.instance() + +# --------------------------------------------------------------------------- +# Dataset definition (shared across all modes) +# --------------------------------------------------------------------------- +EMBODIMENT_B_DATASET = [ + L(dataset_entry)( + name="embodiment_b", + dataset=L(Embodiment_bDataset)(chunk_length=16, split="train"), + ratio=1.0, + ), +] + +# --------------------------------------------------------------------------- +# Base experiment — 2B, 4k iters (policy mode default) +# --------------------------------------------------------------------------- +embodiment_b = make_2b_experiment( + exp_name="embodiment_b", + datasets=EMBODIMENT_B_DATASET, + training_iterations=4_000, +) +embodiment_b["job"]["group"] = "embodiment_b" + +cs.store("embodiment_b", embodiment_b, group="experiment", package="_global_") +register_modes(cs, "embodiment_b", embodiment_b, dataloader_key="action_data") + + +# --------------------------------------------------------------------------- +# Training iteration: 20000. +# --------------------------------------------------------------------------- +embodiment_b_iter2e4 = dict( + defaults=["/experiment/embodiment_b", "_self_"], + scheduler=dict(cycle_lengths=[20000]), + trainer=dict(max_iter=20000), +) +cs.store("embodiment_b_iter2e4", embodiment_b_iter2e4, group="experiment", package="_global_") +register_modes(cs, "embodiment_b_iter2e4", embodiment_b, dataloader_key="action_data") + + +# --------------------------------------------------------------------------- +# Left wrist camera. +# --------------------------------------------------------------------------- +embodiment_b_wrist_left = dict( + defaults=["/experiment/embodiment_b", "_self_"], + dataloader_train=dict( + dataloaders=dict( + action_data=dict( + dataloader=dict(dataset=dict(list_of_datasets=dict(video_key="observation.images.camera_wrist_left"))) + ) + ) + ), + job=dict(name="${now:%Y-%m-%d_%H-%M-%S}_embodiment_b_wrist_left"), +) +cs.store("embodiment_b_wrist_left", embodiment_b_wrist_left, group="experiment", package="_global_") +register_modes(cs, "embodiment_b_wrist_left", embodiment_b, dataloader_key="action_data") + + +# --------------------------------------------------------------------------- +# Right wrist camera. +# --------------------------------------------------------------------------- +embodiment_b_wrist_right = dict( + defaults=["/experiment/embodiment_b", "_self_"], + dataloader_train=dict( + dataloaders=dict( + action_data=dict( + dataloader=dict(dataset=dict(list_of_datasets=dict(video_key="observation.images.camera_wrist_right"))) + ) + ) + ), + job=dict(name="${now:%Y-%m-%d_%H-%M-%S}_embodiment_b_wrist_right"), +) +cs.store("embodiment_b_wrist_right", embodiment_b_wrist_right, group="experiment", package="_global_") +register_modes(cs, "embodiment_b_wrist_right", embodiment_b, dataloader_key="action_data") + + +# --------------------------------------------------------------------------- +# Keep aspect ratio. +# --------------------------------------------------------------------------- +embodiment_b_kar = dict( + defaults=["/experiment/embodiment_b", "_self_"], + dataloader_train=dict(dataloaders=dict(action_data=dict(dataloader=dict(dataset=dict(keep_aspect_ratio=True))))), +) +cs.store("embodiment_b_kar", embodiment_b_kar, group="experiment", package="_global_") +register_modes(cs, "embodiment_b_kar", embodiment_b, dataloader_key="action_data") diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/embodiment_c_gripper_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/embodiment_c_gripper_experiment.py new file mode 100644 index 0000000..afd0ab5 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/embodiment_c_gripper_experiment.py @@ -0,0 +1,87 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# Embodiment C Gripper experiments — 2B and 8B pretrained bases +# +# 2B: make_2b_experiment() (480p mrope, exp202_001 2B checkpoint) +# 8B: make_8b_experiment() (480p mrope, exp202_001 8B checkpoint) +# Both include train dataloaders and mode variants (fd, id, policy, video, joint). + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_2b import make_2b_experiment +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_8b import make_8b_experiment +from cosmos_framework.data.vfm.action.embodiment_c_dataset import EmbodimentCGripperDataset +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry + +cs = ConfigStore.instance() + +# --------------------------------------------------------------------------- +# Constants +# --------------------------------------------------------------------------- +TRAINING_ITERATIONS_2B = 4_000 +TRAINING_ITERATIONS_8B = 50_000 + +MODES = [ + ("exp_fd", "forward_dynamics"), + ("exp_id", "inverse_dynamics"), + ("exp_policy", "policy"), + ("exp_video", "image2video"), + ("exp_joint", "joint"), +] + +# --------------------------------------------------------------------------- +# 2B base experiment +# --------------------------------------------------------------------------- +GRIPPER_DATASET = [ + L(dataset_entry)( + name="embodiment_c_gripper", + dataset=L(EmbodimentCGripperDataset)(split="train"), + ratio=1.0, + ), +] + +embodiment_c_gripper = make_2b_experiment( + exp_name="embodiment_c_gripper", + datasets=GRIPPER_DATASET, + training_iterations=TRAINING_ITERATIONS_2B, +) +embodiment_c_gripper["job"]["group"] = "embodiment_c_gripper" +embodiment_c_gripper["checkpoint"]["save_iter"] = 500 + +cs.store("embodiment_c_gripper", embodiment_c_gripper, group="experiment", package="_global_") + +# 2B mode variants +for suffix, mode in MODES: + name = f"embodiment_c_gripper_{suffix}" + node = dict( + defaults=["/experiment/embodiment_c_gripper", "_self_"], + dataloader_train=dict( + dataloaders=dict(action_data=dict(dataloader=dict(dataset=dict(list_of_datasets=dict(mode=mode))))), + ), + job=dict(name=f"{name}_${{now:%Y%m%d_%H%M%S}}"), + ) + cs.store(name, node, group="experiment", package="_global_") + +# --------------------------------------------------------------------------- +# 8B base experiment +# --------------------------------------------------------------------------- +embodiment_c_gripper_8b = make_8b_experiment( + exp_name="embodiment_c_gripper_8b", + datasets=GRIPPER_DATASET, + training_iterations=TRAINING_ITERATIONS_8B, +) +embodiment_c_gripper_8b["job"]["group"] = "embodiment_c_gripper" +embodiment_c_gripper_8b["checkpoint"]["save_iter"] = 2000 + +cs.store("embodiment_c_gripper_8b", embodiment_c_gripper_8b, group="experiment", package="_global_") + +# 8B mode variants +for suffix, mode in MODES: + name = f"embodiment_c_gripper_8b_{suffix}" + node = dict( + defaults=["/experiment/embodiment_c_gripper_8b", "_self_"], + job=dict(name=f"{name}_${{now:%Y%m%d_%H%M%S}}"), + ) + cs.store(name, node, group="experiment", package="_global_") diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/fractal_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/fractal_experiment.py new file mode 100644 index 0000000..4c41045 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/fractal_experiment.py @@ -0,0 +1,42 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# Fractal (Google RT-1) experiment — Cosmos3 2B pretrained base +# +# Base experiment (policy mode) + mode variants (fd, id, policy, i2v, joint). + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.experiment.action._experiment_helpers import register_modes +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_2b import ( + make_2b_experiment, +) +from cosmos_framework.data.vfm.action.fractal import FractalLeRobotDataset +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry + +cs = ConfigStore.instance() + +# --------------------------------------------------------------------------- +# Dataset definition (shared across all modes) +# --------------------------------------------------------------------------- +FRACTAL_DATASET = [ + L(dataset_entry)( + name="fractal", + dataset=L(FractalLeRobotDataset)(chunk_length=16, split="train"), + ratio=1.0, + ), +] + +# --------------------------------------------------------------------------- +# Base experiment — 2B, 4k iters (policy mode default) +# --------------------------------------------------------------------------- +fractal = make_2b_experiment( + exp_name="fractal", + datasets=FRACTAL_DATASET, + training_iterations=4_000, +) +fractal["job"]["group"] = "fractal" + +cs.store("fractal", fractal, group="experiment", package="_global_") +register_modes(cs, "fractal", fractal, dataloader_key="action_data") diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/hand_pose_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/hand_pose_experiment.py new file mode 100644 index 0000000..0279851 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/hand_pose_experiment.py @@ -0,0 +1,382 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Hand-pose Action experiments built on top of the centralized pretrained configs. + +Base configs are constructed via ``make_2b_experiment`` / ``make_8b_experiment`` +from the pretrained-config module and then patched with hand-pose-specific +overrides (dataloaders, action dims, tokenizer durations, …). + +Example (single node interactive run):: + + CUDA_VISIBLE_DEVICES=0 PYTHONPATH=. torchrun --nproc_per_node=1 \ + --master_port=12347 cosmos_framework/scripts/train.py \ + --config=cosmos_framework/configs/base/config.py \ + -- experiment=embodiment_a_500hrNew_camWristRel_fd job.wandb_mode=disabled +""" + +from typing import Any + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_2b import make_2b_experiment +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_8b import make_8b_experiment +from cosmos_framework.data.vfm.action.dataloaders import InfiniteDataLoader +from cosmos_framework.data.vfm.action.hand_pose_dataset import HandPoseDataset +from cosmos_framework.data.vfm.action.hand_pose_dataset_config import HAND_POSE_DATASETS, EMBODIMENT_A_ALL +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry, wrap_dataset +from cosmos_framework.data.vfm.joint_dataloader import IterativeJointDataLoader + +# -------------------------------------------------------------------------- +# Data roots & experiment factory +# -------------------------------------------------------------------------- + +_EMBODIMENT_A_500HR_UPDATED_ROOT_FIXED = HAND_POSE_DATASETS["embodiment_a_feb08_500hr"] +_VITRA_EGO4D_ROOT = HAND_POSE_DATASETS["vitra_ego4d"] + + +def _make_hand_pose_experiment( + name: str, + base: str = "hand_pose_exp_8b_base_config", + max_action_dim: int = 256, + save_iter: int | None = 500, + mode: str = "forward_dynamics", + pose_convention: str = "backward_framewise", + chunk_length: int = 72, + root: str | list[str] = _EMBODIMENT_A_500HR_UPDATED_ROOT_FIXED, + keypoint_option: str = "wrist_plus_fingers", + rotation_format: str = "rot9d", + intra_episode_val_ratio: float = 0.0, + max_episodes: int | None = None, + snap_to_subtask: bool = True, + skip_no_action: bool = True, + max_subtasks_per_episode: int | None = 5, +) -> dict: + """Build a complete hand-pose posttrain experiment config. + + Creates a Hydra experiment node with matched train/val dataloaders that + differ only in ``split``. Defaults match ``HandPoseDataset`` class defaults. + """ + ds_kwargs: dict[str, Any] = dict( + mode=mode, + pose_convention=pose_convention, + chunk_length=chunk_length, + root=root, + keypoint_option=keypoint_option, + rotation_format=rotation_format, + intra_episode_val_ratio=intra_episode_val_ratio, + snap_to_subtask=snap_to_subtask, + skip_no_action=skip_no_action, + ) + if max_episodes is not None: + ds_kwargs["max_episodes"] = max_episodes + if max_subtasks_per_episode is not None: + ds_kwargs["max_subtasks_per_episode"] = max_subtasks_per_episode + + def dataloader_override(split: str) -> dict: + ds_list = [ + L(dataset_entry)( + name="hand_pose", + dataset=L(HandPoseDataset)(split=split, **ds_kwargs), + ratio=1.0, + resolution="480", + ), + ] + return {"dataloaders": {"uva_data": {"dataloader": {"dataset": {"list_of_datasets": ds_list}}}}} + + node: dict[str, Any] = dict( + defaults=[f"/experiment/{base}", "_self_"], + model=dict(config=dict(max_action_dim=max_action_dim)), + dataloader_train=dataloader_override("train"), + dataloader_val=dataloader_override("val"), + job=dict(name=name), + ) + if save_iter is not None: + node["checkpoint"] = dict(save_iter=save_iter) + return node + + +def _hand_pose_base_dataloader(split: str) -> Any: + """Default hand-pose dataloader for base configs (train or val).""" + is_train = split == "train" + loader_kwargs: dict[str, Any] = dict( + batch_size=4 if is_train else 1, + shuffle=is_train, + num_workers=4 if is_train else 0, + pin_memory=True, + drop_last=True, + ) + if is_train: + loader_kwargs["seed"] = 42 + + return L(IterativeJointDataLoader)( + dataloaders={ + "uva_data": dict( + dataloader=L(InfiniteDataLoader)( + dataset=L(wrap_dataset)( + list_of_datasets=[ + L(dataset_entry)( + name="hand_pose", + dataset=L(HandPoseDataset)(chunk_length=72, split=split), + ratio=1.0, + resolution="480", + ), + ], + tokenizer_config="${model.config.vlm_config.tokenizer}", + cfg_dropout_rate=0.1 if is_train else 0.0, + max_action_dim="${model.config.max_action_dim}", + shard_across_workers=True, + ), + **loader_kwargs, + ), + ratio=1, + ), + }, + tokenizer_spatial_compression_factor="${model.config.tokenizer.spatial_compression_factor}", + tokenizer_temporal_compression_factor="${model.config.tokenizer.temporal_compression_factor}", + patch_spatial="${model.config.diffusion_expert_config.patch_spatial}", + max_sequence_length="${model.config.max_num_tokens_after_packing}", + ) + + +# ========================================================================== +# Base configs (built from centralized pretrained-config factories) +# ========================================================================== + +# ---- 2B: 480p, unified_3d_mrope, 300 frames ---- +hand_pose_exp_2b_base_config = make_2b_experiment( + "hand_pose_exp_2b_base_config", + datasets=[], + max_action_dim=64, + training_iterations=50_000, +) +hand_pose_exp_2b_base_config["job"]["group"] = "hand_pose" +hand_pose_exp_2b_base_config["model"]["config"]["max_action_dim"] = 64 +hand_pose_exp_2b_base_config["model"]["config"]["num_embodiment_domains"] = 32 +hand_pose_exp_2b_base_config["model"]["config"]["tokenizer"]["encode_exact_durations"] = [17, 61, 73, 93] +hand_pose_exp_2b_base_config["dataloader_train"] = _hand_pose_base_dataloader("train") +hand_pose_exp_2b_base_config["dataloader_val"] = _hand_pose_base_dataloader("val") + +# ---- 8B: 480p single-res, unified_3d_mrope with modality offset ---- +hand_pose_exp_8b_base_config = make_8b_experiment( + "hand_pose_exp_8b_base_config", + datasets=[], + max_action_dim=64, + training_iterations=50_000, +) +hand_pose_exp_8b_base_config["job"]["group"] = "hand_pose" +hand_pose_exp_8b_base_config["model"]["config"]["resolution"] = "480" +hand_pose_exp_8b_base_config["trainer"]["callbacks"]["compile_tokenizer"]["enabled"] = True +hand_pose_exp_8b_base_config["trainer"]["callbacks"]["compile_tokenizer"]["warmup_resolutions"] = ["480"] +hand_pose_exp_8b_base_config["dataloader_train"] = _hand_pose_base_dataloader("train") +hand_pose_exp_8b_base_config["dataloader_val"] = _hand_pose_base_dataloader("val") + +# ========================================================================== +# Reusable dataset entries for multi-domain joint training. +# ========================================================================== + +_HAND_POSE_DATASET_DEFAULTS: dict[str, object] = dict( + root=_EMBODIMENT_A_500HR_UPDATED_ROOT_FIXED, + chunk_length=72, + pose_convention="backward_framewise", + keypoint_option="wrist_plus_finger_tips", + rotation_format="rot6d", + snap_to_subtask=True, + skip_no_action=True, + max_subtasks_per_episode=5, +) + +DATASET_HAND_POSE_480 = L(dataset_entry)( + name="hand_pose", + dataset=L(HandPoseDataset)(split="train", mode="forward_dynamics", **_HAND_POSE_DATASET_DEFAULTS), + ratio=1.0, + resolution="480", +) + +DATASET_HAND_POSE_480_I2V = L(dataset_entry)( + name="hand_pose", + dataset=L(HandPoseDataset)(split="train", mode="image2video", **_HAND_POSE_DATASET_DEFAULTS), + ratio=1.0, + resolution="480", +) + +DATASET_HAND_POSE_480_JOINT = L(dataset_entry)( + name="hand_pose", + dataset=L(HandPoseDataset)(split="train", mode="joint", **_HAND_POSE_DATASET_DEFAULTS), + ratio=1.0, + resolution="480", +) + +# ========================================================================== +# Posttrain ablation experiments +# +# Experiment names keep legacy "cam*" prefixes for W&B / checkpoint continuity; +# actual pose_convention values use the unified names. +# +# Axes varied across experiments: +# Pose convention – backward_anchored, backward_framewise +# Keypoints – wrist_plus_fingers (all 21, default) · wrist_plus_finger_tips +# Rotation format – rot9d (default) · rot6d +# Model size – 2B · 8B +# ========================================================================== + +# -- Group 1: backward_anchored ± egocam (8B) ----------------------------- +embodiment_a_500hrNew_camAnchored_fd_8b_32node = _make_hand_pose_experiment( + "embodiment_a_500hrNew_camAnchored_fd_8b_32node", + pose_convention="backward_anchored", +) +embodiment_a_500hrNew_camAnchoredWithCamPose_fd_8b_32node = _make_hand_pose_experiment( + "embodiment_a_500hrNew_camAnchoredWithCamPose_fd_8b_32node", + pose_convention="backward_anchored", +) +embodiment_a_500hrNew_camAnchoredWithCamPose_rot6D_fd_8b_32node = _make_hand_pose_experiment( + "embodiment_a_500hrNew_camAnchoredWithCamPose_rot6D_fd_8b_32node", + pose_convention="backward_anchored", + rotation_format="rot6d", +) +embodiment_a_500hrNew_camAnchoredWithCamPose_wristAndFingerTips_rot6D_fd_8b_32node = _make_hand_pose_experiment( + "embodiment_a_500hrNew_camAnchoredWithCamPose_wristAndFingerTips_rot6D_fd_8b_32node", + max_action_dim=64, + pose_convention="backward_anchored", + keypoint_option="wrist_plus_finger_tips", + rotation_format="rot6d", +) + +# -- Group 2: backward_framewise (8B) ----------------------------- +embodiment_a_500hrNew_camWristRelWithCamPose_fd_8b_32node = _make_hand_pose_experiment( + "embodiment_a_500hrNew_camWristRelWithCamPose_fd_8b_32node", + pose_convention="backward_framewise", +) +embodiment_a_500hrNew_camWristRelWithCamPose_wristAndFingerTips_fd_8b_32node = _make_hand_pose_experiment( + "embodiment_a_500hrNew_camWristRelWithCamPose_wristAndFingerTips_fd_8b_32node", + pose_convention="backward_framewise", + keypoint_option="wrist_plus_finger_tips", +) +embodiment_a_500hrNew_camWristRelWithCamPose_wristAndFingerTips_rot6D_fd_8b_32node = _make_hand_pose_experiment( + "embodiment_a_500hrNew_camWristRelWithCamPose_wristAndFingerTips_rot6D_fd_8b_32node", + max_action_dim=64, + pose_convention="backward_framewise", + keypoint_option="wrist_plus_finger_tips", + rotation_format="rot6d", +) +embodiment_a_500hrNew_camWristRelWithCamPose_wristAndFingerTips_deltaTminus1_fd_8b_32node = _make_hand_pose_experiment( + "embodiment_a_500hrNew_camWristRelWithCamPose_wristAndFingerTips_deltaTminus1_fd_8b_32node", + pose_convention="backward_framewise", + keypoint_option="wrist_plus_finger_tips", +) + +# -- Group 3: backward_framewise (midtraining default, 57D, 8B) ------------ +embodiment_a_500hrNew_backwardFramewise_fingerTips_rot6D_fd_8b_maxep10 = _make_hand_pose_experiment( + "embodiment_a_500hrNew_backwardFramewise_fingerTips_rot6D_fd_8b_maxep10", + max_action_dim=64, + pose_convention="backward_framewise", + keypoint_option="wrist_plus_finger_tips", + rotation_format="rot6d", + max_episodes=10, +) +embodiment_a_500hrNew_backwardFramewise_fingerTips_rot6D_id_8b_maxep10 = _make_hand_pose_experiment( + "embodiment_a_500hrNew_backwardFramewise_fingerTips_rot6D_id_8b_maxep10", + max_action_dim=64, + mode="inverse_dynamics", + pose_convention="backward_framewise", + keypoint_option="wrist_plus_finger_tips", + rotation_format="rot6d", + max_episodes=10, +) +embodiment_a_all_backwardFramewise_fingerTips_rot6D_fd_8b_32node = _make_hand_pose_experiment( + "embodiment_a_all_backwardFramewise_fingerTips_rot6D_fd_8b_32node", + max_action_dim=64, + pose_convention="backward_framewise", + keypoint_option="wrist_plus_finger_tips", + rotation_format="rot6d", + root=EMBODIMENT_A_ALL, +) + +# -- Group 4: vitra_ego4d backward_framewise (57D, 8B, overfit) ------------- +vitra_ego4d_backwardFramewise_fingerTips_rot6D_fd_8b_maxep10 = _make_hand_pose_experiment( + "vitra_ego4d_backwardFramewise_fingerTips_rot6D_fd_8b_maxep10", + max_action_dim=64, + pose_convention="backward_framewise", + keypoint_option="wrist_plus_finger_tips", + rotation_format="rot6d", + chunk_length=16, + root=_VITRA_EGO4D_ROOT, + max_episodes=10, +) +vitra_ego4d_backwardFramewise_fingerTips_rot6D_id_8b_maxep10 = _make_hand_pose_experiment( + "vitra_ego4d_backwardFramewise_fingerTips_rot6D_id_8b_maxep10", + max_action_dim=64, + mode="inverse_dynamics", + pose_convention="backward_framewise", + keypoint_option="wrist_plus_finger_tips", + rotation_format="rot6d", + chunk_length=16, + root=_VITRA_EGO4D_ROOT, + max_episodes=10, +) + +# -- test all data ----------------------------- +embodiment_a_all_camAnchored_fd_8b_32node = _make_hand_pose_experiment( + "embodiment_a_500hrNew_camAnchored_fd_8b_32node", + pose_convention="backward_anchored", + root=EMBODIMENT_A_ALL, +) + +# ========================================================================== +# Register all experiments +# ========================================================================== + +cs = ConfigStore.instance() + +_ALL_EXPERIMENTS: list[tuple[str, dict]] = [ + ("hand_pose_exp_2b_base_config", hand_pose_exp_2b_base_config), + ("hand_pose_exp_8b_base_config", hand_pose_exp_8b_base_config), + ("embodiment_a_500hrNew_camAnchored_fd_8b_32node", embodiment_a_500hrNew_camAnchored_fd_8b_32node), + ("embodiment_a_500hrNew_camAnchoredWithCamPose_fd_8b_32node", embodiment_a_500hrNew_camAnchoredWithCamPose_fd_8b_32node), + ( + "embodiment_a_500hrNew_camAnchoredWithCamPose_rot6D_fd_8b_32node", + embodiment_a_500hrNew_camAnchoredWithCamPose_rot6D_fd_8b_32node, + ), + ( + "embodiment_a_500hrNew_camAnchoredWithCamPose_wristAndFingerTips_rot6D_fd_8b_32node", + embodiment_a_500hrNew_camAnchoredWithCamPose_wristAndFingerTips_rot6D_fd_8b_32node, + ), + ("embodiment_a_500hrNew_camWristRelWithCamPose_fd_8b_32node", embodiment_a_500hrNew_camWristRelWithCamPose_fd_8b_32node), + ( + "embodiment_a_500hrNew_camWristRelWithCamPose_wristAndFingerTips_fd_8b_32node", + embodiment_a_500hrNew_camWristRelWithCamPose_wristAndFingerTips_fd_8b_32node, + ), + ( + "embodiment_a_500hrNew_camWristRelWithCamPose_wristAndFingerTips_rot6D_fd_8b_32node", + embodiment_a_500hrNew_camWristRelWithCamPose_wristAndFingerTips_rot6D_fd_8b_32node, + ), + ( + "embodiment_a_500hrNew_camWristRelWithCamPose_wristAndFingerTips_deltaTminus1_fd_8b_32node", + embodiment_a_500hrNew_camWristRelWithCamPose_wristAndFingerTips_deltaTminus1_fd_8b_32node, + ), + ( + "embodiment_a_500hrNew_backwardFramewise_fingerTips_rot6D_fd_8b_maxep10", + embodiment_a_500hrNew_backwardFramewise_fingerTips_rot6D_fd_8b_maxep10, + ), + ( + "embodiment_a_500hrNew_backwardFramewise_fingerTips_rot6D_id_8b_maxep10", + embodiment_a_500hrNew_backwardFramewise_fingerTips_rot6D_id_8b_maxep10, + ), + ( + "embodiment_a_all_backwardFramewise_fingerTips_rot6D_fd_8b_32node", + embodiment_a_all_backwardFramewise_fingerTips_rot6D_fd_8b_32node, + ), + ( + "vitra_ego4d_backwardFramewise_fingerTips_rot6D_fd_8b_maxep10", + vitra_ego4d_backwardFramewise_fingerTips_rot6D_fd_8b_maxep10, + ), + ( + "vitra_ego4d_backwardFramewise_fingerTips_rot6D_id_8b_maxep10", + vitra_ego4d_backwardFramewise_fingerTips_rot6D_id_8b_maxep10, + ), + ("embodiment_a_all_camAnchored_fd_8b_32node", embodiment_a_all_camAnchored_fd_8b_32node), +] + +for _name, _node in _ALL_EXPERIMENTS: + cs.store(_name, _node, group="experiment", package="_global_") diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/libero_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/libero_experiment.py new file mode 100644 index 0000000..c31df11 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/libero_experiment.py @@ -0,0 +1,73 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# LIBERO experiment — Cosmos3 2B pretrained base + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.experiment.action._experiment_helpers import ( + LIBERO_BASELINE_BATCH_SIZE, + LIBERO_BASELINE_NUM_WORKERS, + LIBERO_BASELINE_TRAINING_ITERATIONS, +) +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_2b import make_2b_experiment +from cosmos_framework.data.vfm.action.libero_dataset import LIBERODataset +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry + +cs = ConfigStore.instance() + +DATALOADER_SEED = 0 + +LIBERO_REPO_IDS = ["libero_10", "libero_90", "libero_object", "libero_spatial", "libero_goal"] + +# --------------------------------------------------------------------------- +# Dataset definition +# --------------------------------------------------------------------------- +LIBERO_TRAIN_DATASET = [ + L(dataset_entry)( + name="libero", + dataset=L(LIBERODataset)( + repo_id=LIBERO_REPO_IDS, + split="train", + camera_mode="image", + ), + ratio=1.0, + ), +] + +# --------------------------------------------------------------------------- +# Base experiment — 2B, 4k iters +# --------------------------------------------------------------------------- +libero_exp = make_2b_experiment( + exp_name="libero_exp", + datasets=LIBERO_TRAIN_DATASET, + batch_size=LIBERO_BASELINE_BATCH_SIZE, + num_workers=LIBERO_BASELINE_NUM_WORKERS, + training_iterations=LIBERO_BASELINE_TRAINING_ITERATIONS, +) + +# --- Experiment-specific overrides --- +libero_exp["job"]["group"] = "debugging" + +# Replace the callbacks entry in defaults +for i, d in enumerate(libero_exp["defaults"]): + if isinstance(d, dict) and "override /callbacks" in d: + libero_exp["defaults"][i] = { + "override /callbacks": [ + "basic", + "optimization", + "job_monitor", + "training_stats", + ] + } + break + +# Scheduler: LIBERO uses f_max=1.0 +libero_exp["scheduler"]["f_max"] = [1.0] +libero_exp["scheduler"]["warm_up_steps"] = [LIBERO_BASELINE_TRAINING_ITERATIONS // 20] + +cs.store(group="experiment", package="_global_", name="libero_exp", node=libero_exp) + +# Alias for backward compatibility +cs.store(group="experiment", package="_global_", name="libero_exp_streaming", node=libero_exp) diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/libero_fd_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/libero_fd_experiment.py new file mode 100644 index 0000000..58c1588 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/libero_fd_experiment.py @@ -0,0 +1,102 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# Forward Dynamics Experiment for LIBERO + +from datetime import datetime + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.configs.base.experiment.action._experiment_helpers import ( + LIBERO_BASELINE_BATCH_SIZE, + LIBERO_BASELINE_NUM_WORKERS, + LIBERO_BASELINE_TRAINING_ITERATIONS, + make_libero_dataset, +) +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_2b import make_2b_experiment + +cs = ConfigStore.instance() + + +def _build_libero_fd_base( + exp_name: str, + *, + training_iterations: int = LIBERO_BASELINE_TRAINING_ITERATIONS, + batch_size: int = LIBERO_BASELINE_BATCH_SIZE, + num_workers: int = LIBERO_BASELINE_NUM_WORKERS, + job_group: str = "debugging", + job_project: str | None = None, + **dataset_kwargs, +) -> dict: + dataset_kwargs.setdefault("mode", "forward_dynamics") + exp = make_2b_experiment( + exp_name=exp_name, + datasets=make_libero_dataset(**dataset_kwargs), + batch_size=batch_size, + num_workers=num_workers, + training_iterations=training_iterations, + ) + + # Match libero_experiment.libero_exp post-build mutations. + for i, d in enumerate(exp["defaults"]): + if isinstance(d, dict) and "override /callbacks" in d: + exp["defaults"][i] = { + "override /callbacks": [ + "basic", + "optimization", + "job_monitor", + "training_stats", + ] + } + break + exp["scheduler"]["f_max"] = [1.0] + exp["scheduler"]["warm_up_steps"] = [training_iterations // 20] + + exp["job"]["group"] = job_group + if job_project is not None: + exp["job"]["project"] = job_project + return exp + + +# --------------------------------------------------------------------------- +# libero_exp_fd — base FD experiment +# --------------------------------------------------------------------------- +libero_exp_fd = _build_libero_fd_base( + exp_name=f"libero_exp_fd_{datetime.now().strftime('%Y%m%d_%H%M%S')}", +) +cs.store( + group="experiment", + package="_global_", + name="libero_exp_fd", + node=libero_exp_fd, +) + + +# --------------------------------------------------------------------------- +# Sweep camera_mode × action_space × rotation_space (lr=2e-4, mode=forward_dynamics) +# --------------------------------------------------------------------------- +def action_camera_rotation_sweep(): + camera_modes = ["image", "wrist_image"] + action_spaces = ["relative", "frame_wise_relative"] + rotation_spaces = ["9d", "6d", "3d"] + + for camera_mode in camera_modes: + for action_space in action_spaces: + for rotation_space in rotation_spaces: + cam_short = "img" if camera_mode == "image" else "wrist" + act_short = "rel" if action_space == "relative" else "fwrel" + rot_short = rotation_space + name = f"libero_exp_fd_{cam_short}_{act_short}_{rot_short}_varlen" + + exp = _build_libero_fd_base( + exp_name=name, + job_group="action_libero", + camera_mode=camera_mode, + action_space=action_space, + rotation_space=rotation_space, + ) + exp["optimizer"]["lr"] = 2e-4 + cs.store(group="experiment", package="_global_", name=name, node=exp) + + +action_camera_rotation_sweep() diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/libero_joint_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/libero_joint_experiment.py new file mode 100644 index 0000000..ee3be4c --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/libero_joint_experiment.py @@ -0,0 +1,183 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# Joint Training Experiment for LIBERO - randomly samples from all modes (FD, ID, policy, video) + +from datetime import datetime + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.configs.base.experiment.action._experiment_helpers import ( + LIBERO_BASELINE_BATCH_SIZE, + LIBERO_BASELINE_NUM_WORKERS, + make_libero_dataset, +) +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_2b import make_2b_experiment + +cs = ConfigStore.instance() + +# Joint mode trains 2x longer than the libero baseline (4_000) — keep this local. +TRAINING_ITERATIONS = 8000 + + +def _build_libero_joint_base( + exp_name: str, + *, + training_iterations: int = TRAINING_ITERATIONS, + batch_size: int = LIBERO_BASELINE_BATCH_SIZE, + num_workers: int = LIBERO_BASELINE_NUM_WORKERS, + job_group: str = "action_libero", + job_project: str | None = None, + **dataset_kwargs, +) -> dict: + # mode="joint" is the defining trait of this experiment family. + dataset_kwargs.setdefault("mode", "joint") + exp = make_2b_experiment( + exp_name=exp_name, + datasets=make_libero_dataset(**dataset_kwargs), + batch_size=batch_size, + num_workers=num_workers, + training_iterations=training_iterations, + ) + + # Match libero_experiment.libero_exp post-build mutations. + for i, d in enumerate(exp["defaults"]): + if isinstance(d, dict) and "override /callbacks" in d: + exp["defaults"][i] = { + "override /callbacks": [ + "basic", + "optimization", + "job_monitor", + "training_stats", + ] + } + break + exp["scheduler"]["f_max"] = [1.0] + exp["scheduler"]["warm_up_steps"] = [training_iterations // 20] + + exp["job"]["group"] = job_group + if job_project is not None: + exp["job"]["project"] = job_project + return exp + + +# --------------------------------------------------------------------------- +# libero_exp_joint — base joint training (8k iters, lr=2e-4) +# --------------------------------------------------------------------------- +libero_exp_joint = _build_libero_joint_base( + exp_name=f"libero_exp_joint_{datetime.now().strftime('%Y%m%d_%H%M%S')}", +) +libero_exp_joint["optimizer"]["lr"] = 2e-4 + +cs.store( + group="experiment", + package="_global_", + name="libero_exp_joint", + node=libero_exp_joint, +) + + +# --------------------------------------------------------------------------- +# Chunk size sweep (lr=2e-4, mode=joint, varying chunk_length) +# --------------------------------------------------------------------------- +def chunk_size_sweep(): + for chunk_length in [8, 16, 24, 32, 40, 48]: + name = f"libero_exp_lr2e4_joint_chunk{chunk_length}" + exp = _build_libero_joint_base( + exp_name=name, + job_project="cosmos3_action_libero", + chunk_length=chunk_length, + ) + exp["optimizer"]["lr"] = 2e-4 + cs.store(group="experiment", package="_global_", name=name, node=exp) + + +# --------------------------------------------------------------------------- +# Sweep camera_mode × action_space × rotation_space (lr=2e-4, mode=joint) +# --------------------------------------------------------------------------- +def action_camera_rotation_sweep(): + camera_modes = ["image", "wrist_image"] + action_spaces = ["relative", "frame_wise_relative"] + rotation_spaces = ["9d", "6d", "3d"] + + for camera_mode in camera_modes: + for action_space in action_spaces: + for rotation_space in rotation_spaces: + cam_short = "img" if camera_mode == "image" else "wrist" + act_short = "rel" if action_space == "relative" else "fwrel" + rot_short = rotation_space + name = f"libero_exp_joint_{cam_short}_{act_short}_{rot_short}_varlen" + + exp = _build_libero_joint_base( + exp_name=name, + camera_mode=camera_mode, + action_space=action_space, + rotation_space=rotation_space, + ) + exp["optimizer"]["lr"] = 2e-4 + cs.store(group="experiment", package="_global_", name=name, node=exp) + + +# --------------------------------------------------------------------------- +# Single-view 60k iter joint experiment (and chunk8 variant) +# --------------------------------------------------------------------------- +def single_view_joint_exp(): + """ + CUDA_VISIBLE_DEVICES=0 PYTHONPATH=. torchrun --nproc_per_node=1 --master_port=12341 cosmos_framework/scripts/train.py --config=cosmos_framework/configs/base/config.py -- experiment=libero_exp_joint_single_view_lr2e4_60k + """ + name = "libero_exp_joint_single_view_lr2e4_60k" + exp = _build_libero_joint_base( + exp_name=name, + training_iterations=60_000, + batch_size=256, + job_project="cosmos3_action_libero", + ) + exp["optimizer"]["lr"] = 2e-4 + exp["scheduler"]["warm_up_steps"] = [100] + exp["scheduler"]["cycle_lengths"] = [60_000] + exp["checkpoint"]["load_from_object_store"] = dict(bucket="nv-00-10206-checkpoint") + + cs.store(group="experiment", package="_global_", name=name, node=exp) + + chunk8_name = "libero_exp_joint_single_view_lr2e4_60k_chunk8" + exp_chunk8 = _build_libero_joint_base( + exp_name=chunk8_name, + training_iterations=60_000, + batch_size=256, + job_project="cosmos3_action_libero", + chunk_length=8, + ) + exp_chunk8["optimizer"]["lr"] = 2e-4 + exp_chunk8["scheduler"]["warm_up_steps"] = [100] + exp_chunk8["scheduler"]["cycle_lengths"] = [60_000] + exp_chunk8["checkpoint"]["load_from_object_store"] = dict(bucket="nv-00-10206-checkpoint") + cs.store(group="experiment", package="_global_", name=chunk8_name, node=exp_chunk8) + + +# --------------------------------------------------------------------------- +# Wrist-view 60k iter joint experiment +# --------------------------------------------------------------------------- +def wrist_view_joint_exp(): + """ + CUDA_VISIBLE_DEVICES=0 PYTHONPATH=. torchrun --nproc_per_node=1 --master_port=12341 cosmos_framework/scripts/train.py --config=cosmos_framework/configs/base/config.py -- experiment=libero_exp_joint_wrist_view_lr2e4_60k + """ + name = "libero_exp_joint_wrist_view_lr2e4_60k" + exp = _build_libero_joint_base( + exp_name=name, + training_iterations=60_000, + batch_size=256, + job_project="cosmos3_action_libero", + camera_mode="wrist_image", + ) + exp["optimizer"]["lr"] = 2e-4 + exp["scheduler"]["warm_up_steps"] = [100] + exp["scheduler"]["cycle_lengths"] = [60_000] + exp["trainer"]["run_validation"] = False + exp["checkpoint"]["load_from_object_store"] = dict(bucket="nv-00-10206-checkpoint") + cs.store(group="experiment", package="_global_", name=name, node=exp) + + +chunk_size_sweep() +single_view_joint_exp() +wrist_view_joint_exp() +action_camera_rotation_sweep() diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/libero_policy_datapacker_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/libero_policy_datapacker_experiment.py new file mode 100644 index 0000000..aed13db --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/libero_policy_datapacker_experiment.py @@ -0,0 +1,287 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""LIBERO policy SFT — 8B, DataPackerDataLoader variant. + +Equivalent to ``action_policy_sft_8b`` but replaces the 3-layer + IterativeJointDataLoader → InfiniteDataLoader → ActionUnifiedIterableDataset +with the single-layer OSS-facing DataPackerDataLoader + ActionDataPacker. + +Batch semantics are preserved: ``max_batch_size=256`` caps the sample count; +``max_tokens=999_999`` is set high so the token budget never fires first. + +Usage (smoke test — same overrides as launch_action_from_vfm.sh):: + + LIBERO_LOCAL_DATA_ROOT=outputs/libero_datasets torchrun \\ + --nproc_per_node=4 --master_port=12342 -m cosmos_framework.scripts.train \\ + --config=cosmos_framework/configs/base/config.py -- \\ + experiment=action_policy_sft_8b_datapacker \\ + trainer.max_iter=10 trainer.logging_iter=1 \\ + job.group=debug job.wandb_mode=disabled \\ + upload_reproducible_setup=false \\ + checkpoint=local ckpt_type=dummy \\ + model.config.parallelism.data_parallel_shard_degree=-1 \\ + model.config.parallelism.use_torch_compile=false +""" + +from __future__ import annotations + +import copy +import math + +import torch +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.experiment.action.posttrain_config.libero_policy_experiment import ( + _LIBERO_REPO_IDS, + _LIBERO_ROOTS, + _build_libero_policy_base_8b, +) +from cosmos_framework.data.vfm.action.libero_dataset import LIBERODataset +from cosmos_framework.data.vfm.action.transforms import ActionTransformPipeline +from cosmos_framework.data.vfm.data_packer import DataPacker +from cosmos_framework.data.vfm.data_packer_dataloader import DataPackerDataLoader +from cosmos_framework.callbacks.dataloader_state import DataLoaderStateCallback + +cs = ConfigStore.instance() + + +# --------------------------------------------------------------------------- +# data_source factory +# +# DataPackerDataLoader accepts map-style Datasets directly — with shuffle=True +# it wraps LIBERODataset in _ShuffledMapIterableDataset internally, so no +# manual MapToIterableAdapter wrapping is needed. +# --------------------------------------------------------------------------- + + +def get_libero_map_dataset(**kwargs) -> LIBERODataset: + """Return LIBERODataset as a map-style Dataset for DataPackerDataLoader. + + DataPackerDataLoader(shuffle=True) wraps it in _ShuffledMapIterableDataset, + giving per-epoch randperm shuffle and stateful checkpoint/resume via + DataLoaderStateCallback(distributor_type="data_packer"). + """ + return LIBERODataset(**kwargs) + + +# --------------------------------------------------------------------------- +# ActionDataPacker +# +# OSS users writing a DataPacker for a custom robot dataset should follow +# this pattern: subclass DataPacker, implement three methods, and place the +# class next to the experiment config that uses it. +# --------------------------------------------------------------------------- + + +class ActionDataPacker(DataPacker): + """DataPacker adapter for LIBERODataset + ActionTransformPipeline. + + Bridges raw LIBERODataset items into the DataPackerDataLoader packing + engine with the same transform pipeline and batch format as the + IterativeJointDataLoader-based ``action_policy_sft_8b`` experiment. + + Three responsibilities: + - ``sft_process_sample``: run the full ActionTransformPipeline on a + raw LIBERODataset sample (resize, tokenize, pad action, build SequencePlan). + - ``compute_num_tokens``: count video + text tokens for packing budget. + - ``sft_collate_fn``: assemble the batch dict OmniMoTModel expects. + """ + + def __init__( + self, + tokenizer_spatial_compression_factor: int = 16, + tokenizer_temporal_compression_factor: int = 4, + patch_spatial: int = 2, + tokenizer_config=None, + cfg_dropout_rate: float = 0.1, + max_action_dim: int = 64, + action_channel_masking: bool = True, + ) -> None: + self._spatial = tokenizer_spatial_compression_factor + self._temporal = tokenizer_temporal_compression_factor + self._patch = patch_spatial + self._transform = ActionTransformPipeline( + pad_keys=["video"], + tokenizer_config=tokenizer_config, + cfg_dropout_rate=cfg_dropout_rate, + max_action_dim=max_action_dim, + action_channel_masking=action_channel_masking, + append_duration_fps_timestamps=True, + append_resolution_info=True, + ) + + def sft_process_sample(self, item: dict) -> dict: + """Apply ActionTransformPipeline to one raw LIBERODataset sample.""" + return self._transform(item, resolution=None) + + def compute_num_tokens(self, sample: dict) -> int: + """Count video latent tokens + text tokens for the packing budget.""" + tokens = 1 + len(sample.get("text_token_ids", [])) + v = sample.get("video") # [C,T,H,W] or None + if v is not None: + _, T, H, W = v.shape # [C,T,H,W] + latent_h = math.ceil(H / (self._spatial * self._patch)) + latent_w = math.ceil(W / (self._spatial * self._patch)) + latent_t = 1 + (T - 1) // self._temporal + tokens += latent_h * latent_w * latent_t + 2 + return tokens + + def sft_collate_fn(self, samples: list, max_len: int, ignore_label_id: int = -100) -> dict: + """Assemble a list of processed samples into the OmniMoTModel batch dict. + + Mirrors the format produced by InfiniteDataLoader + custom_collate_fn + in action_policy_sft_8b so OmniMoTModel.training_step() needs no changes. + Variable-length or custom-type fields are kept as lists; scalar numeric + fields are stacked into tensors. + """ + return { + # Nested list [[tensor]] matches PackingDataLoader's _MULTI_ITEM_KEYS format + "text_token_ids": [[s["text_token_ids"]] for s in samples], + "video": [s.get("video") for s in samples], + "action": [s.get("action") for s in samples], + "padding_mask": [s.get("padding_mask") for s in samples], + "image_size": [s.get("image_size") for s in samples], + "fps": torch.tensor([float(s.get("fps", 0.0)) for s in samples]), # [N] + "domain_id": [s.get("domain_id") for s in samples], + "sequence_plan": [s.get("sequence_plan") for s in samples], + "raw_action_dim": [s.get("raw_action_dim") for s in samples], + "ai_caption": [s.get("ai_caption", "") for s in samples], + } + + +# --------------------------------------------------------------------------- +# Experiment registration +# --------------------------------------------------------------------------- + + +def action_policy_sft_8b_datapacker_experiments() -> None: + """Register action_policy_sft_8b_datapacker in Hydra ConfigStore. + + Builds from the same base as action_policy_sft_8b (identical model, + optimizer, scheduler, checkpoint, dataset params), then replaces + dataloader_train with DataPackerDataLoader + ActionDataPacker. + """ + # Build from the same base as action_policy_sft_8b. + # batch_size/num_workers here configure the (discarded) IterativeJointDataLoader; + # DataPackerDataLoader's own num_workers=4 is set below when replacing dataloader_train. + exp = _build_libero_policy_base_8b( + "action_policy_sft_8b_datapacker", + training_iterations=16_000, + batch_size=256, + num_workers=4, + job_group="action_libero", + job_project="cosmos3_action_libero", + mode="policy", + keys_to_skip_loading=["net_ema.", "action2llm", "llm2action", "action_modality_embed", "action_pos_embed"], + repo_id=copy.deepcopy(_LIBERO_REPO_IDS), + root=copy.deepcopy(_LIBERO_ROOTS), + fps=20, + camera_mode="concat_view", + action_space="frame_wise_relative", + rotation_space="6d", + chunk_length=16, + seed=0, + val_ratio=0.01, + ) + + # Mirror cosmos-inference action_policy_sft_nano.yaml: replace the + # "training_stats" callbacks group (set by _build_libero_policy_base_8b) + # with "generation". every_n_sample_{ema,reg} are pinned below to + # every_n=999_999 so they never fire during the 16k-iter run, matching the + # YAML's effectively-disabled stance. + for _i, _d in enumerate(exp["defaults"]): + if isinstance(_d, dict) and "override /callbacks" in _d: + exp["defaults"][_i] = { + "override /callbacks": [ + "basic", + "optimization", + "job_monitor", + "generation", + ] + } + break + + # Post-build overrides — identical to action_policy_sft_8b + exp["scheduler"]["f_max"] = [1.0] + exp["scheduler"]["f_min"] = [0.0] + exp["scheduler"]["warm_up_steps"] = [500] + exp["scheduler"]["cycle_lengths"] = [20_000] + exp["optimizer"]["lr"] = 5e-5 + exp["trainer"]["max_iter"] = 16_000 + exp["trainer"]["logging_iter"] = 100 + exp["trainer"]["run_validation"] = False + exp["trainer"]["run_validation_on_start"] = False + exp["checkpoint"]["load_path"] = "outputs/checkpoints/action_policy_sft_8b" + exp["checkpoint"]["load_training_state"] = False + exp["checkpoint"]["save_iter"] = 100 + exp["trainer"]["callbacks"]["compile_tokenizer"]["warmup_resolutions"] = ["256", "480", "720"] + exp["trainer"]["callbacks"]["every_n_sample_ema"] = dict(every_n=999_999, save_s3=False) + exp["trainer"]["callbacks"]["every_n_sample_reg"] = dict(every_n=999_999, save_s3=False) + exp["model"]["config"]["rectified_flow_training_config"]["action_loss_weight"] = 10.0 + exp["model"]["config"]["num_embodiment_domains"] = 32 + + # Local-only mode: bypass internal GCP object-store paths in favor of the + # local checkpoint_path / wan_vae_path declared in + # examples/toml/launch_action_libero.toml and the HF tokenizer/weights + # stack. Without these, defaults pull the GCP bucket + + # qwen3_vl_mot_vlm_8b_instruct_gcp variant. + exp["model"]["config"]["tokenizer"]["bucket_name"] = "" + exp["model"]["config"]["tokenizer"]["object_store_credential_path_pretrained"] = "" + exp["checkpoint"]["load_from_object_store"] = dict(enabled=False) + exp["checkpoint"]["save_to_object_store"] = dict(enabled=False) + exp["model"]["config"]["vlm_config"]["tokenizer"] = dict(config_variant="hf") + exp["model"]["config"]["vlm_config"]["pretrained_weights"]["enabled"] = False + + # DataLoaderStateCallback tracks per-worker (epoch, position) when shuffle=True + # is passed to DataPackerDataLoader. It is a no-op when the batch carries no + # position metadata (shuffle=False or iterable data_source). + exp["trainer"]["callbacks"]["dataloader_state"] = L(DataLoaderStateCallback)( + distributor_type="data_packer" + ) + + # Replace the 3-layer IterativeJointDataLoader stack with DataPackerDataLoader. + # Pass LIBERODataset directly as a map-style Dataset — DataPackerDataLoader + # wraps it in _ShuffledMapIterableDataset internally (shuffle=True), so + # MapToIterableAdapter is no longer needed. + # max_tokens=999_999 ensures the token budget never triggers; + # max_batch_size=128 mirrors action_policy_sft_nano.yaml's per-step batch + # (inner InfiniteDataLoader.batch_size=128 under IterativeJointDataLoader). + exp["dataloader_train"] = L(DataPackerDataLoader)( + data_source=L(get_libero_map_dataset)( + repo_id=copy.deepcopy(_LIBERO_REPO_IDS), + root=copy.deepcopy(_LIBERO_ROOTS), + fps=20, + camera_mode="concat_view", + action_space="frame_wise_relative", + rotation_space="6d", + chunk_length=16, + seed=0, + val_ratio=0.01, + mode="policy", + ), + data_packer=L(ActionDataPacker)( + tokenizer_spatial_compression_factor="${model.config.tokenizer.spatial_compression_factor}", + tokenizer_temporal_compression_factor="${model.config.tokenizer.temporal_compression_factor}", + patch_spatial="${model.config.diffusion_expert_config.patch_spatial}", + tokenizer_config="${model.config.vlm_config.tokenizer}", + cfg_dropout_rate=0.1, + max_action_dim="${model.config.max_action_dim}", + action_channel_masking=True, + ), + max_tokens=999_999, + max_batch_size=128, + pool_size=16, + shuffle=True, + seed=0, + num_workers=4, + prefetch_factor=4, + persistent_workers=True, + pin_memory=True, + ) + + cs.store(group="experiment", package="_global_", name="action_policy_sft_8b_datapacker", node=exp) + + +action_policy_sft_8b_datapacker_experiments() diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/libero_policy_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/libero_policy_experiment.py new file mode 100644 index 0000000..524f1a2 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/libero_policy_experiment.py @@ -0,0 +1,1063 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# Policy Experiment for LIBERO - training a policy network to predict video and action jointly + +import copy +import os +from datetime import datetime + +from hydra.core.config_store import ConfigStore +from loguru import logger as log + +from cosmos_framework.configs.base.experiment.action._experiment_helpers import ( + LIBERO_BASELINE_BATCH_SIZE, + LIBERO_BASELINE_NUM_WORKERS, + LIBERO_BASELINE_TRAINING_ITERATIONS, + LIBERO_LOCAL_ROOT_ENV, + make_libero_dataset, +) +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_2b import make_2b_experiment +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_8b import make_8b_experiment + +ACTION_STATS_PATH = "cosmos_framework/data/vfm/action/libero_action_stats_10k.json" +NATIVE_FRAME_WISE_ROT6D_ACTION_STATS_PATH = ( + "cosmos_framework/data/vfm/action/normalizers/libero_native_frame_wise_relative_rot6d.json" +) +cs = ConfigStore.instance() + +_LIBERO_REPO_IDS = ["libero_10", "libero_object", "libero_spatial", "libero_goal"] + +# Cluster layout (default). Override per-suite paths to point at your data. +_LIBERO_ROOTS_LUSTRE = [ + "", + "", + "", + "", +] + +# Local-mount layout (matches gs://nv-00-10206-robot/lerobot_v30/). Order matches +# _LIBERO_REPO_IDS. Mirrors LIBERO_LOCAL_SUITE_DIRS in _experiment_helpers.py. +_LIBERO_LOCAL_SUITE_DIRS = [ + "libero_10_no_noops_1.0.0_lerobot_aligned_20260124", + "libero_object_no_noops_1.0.0_lerobot_aligned_20260124", + "libero_spatial_no_noops_1.0.0_lerobot_20260124", + "libero_goal_no_noops_1.0.0_lerobot_20260124", +] + + +def _resolve_libero_roots() -> list[str]: + """Resolve the libero ``root`` list, honoring ``$LIBERO_LOCAL_DATA_ROOT``. + + Mirrors ``_resolve_libero_default_roots`` in ``_experiment_helpers.py`` so + debug machines without the shared cluster mount can opt into a local mount + via the same env var that the other libero experiments already respect. + """ + base = os.environ.get(LIBERO_LOCAL_ROOT_ENV) + if not base: + return list(_LIBERO_ROOTS_LUSTRE) + + candidates = [os.path.join(base, suite) for suite in _LIBERO_LOCAL_SUITE_DIRS] + missing = [p for p in candidates if not os.path.isdir(p)] + if missing: + raise FileNotFoundError( + f"${LIBERO_LOCAL_ROOT_ENV}={base} is set but the following LIBERO suite directories are missing: {missing}" + ) + log.info(f"[libero] using local LIBERO mount at {base} (via ${LIBERO_LOCAL_ROOT_ENV})") + return candidates + + +_LIBERO_ROOTS = _resolve_libero_roots() + + +def _build_libero_policy_base( + exp_name: str, + *, + training_iterations: int = LIBERO_BASELINE_TRAINING_ITERATIONS, + batch_size: int = LIBERO_BASELINE_BATCH_SIZE, + num_workers: int = LIBERO_BASELINE_NUM_WORKERS, + job_group: str = "debugging", + job_project: str | None = None, + mode: str = "policy", + **dataset_kwargs, +) -> dict: + """Build a libero policy-mode experiment. + + Mirrors the original ``libero_exp`` setup (callbacks override, scheduler + f_max=1.0, warm_up = iter//20). ``mode`` defaults to ``"policy"`` but can + be overridden (e.g., the ``normalized_action_joint_exp`` chain uses other + modes despite living in this file). + """ + dataset_kwargs.setdefault("mode", mode) + exp = make_2b_experiment( + exp_name=exp_name, + datasets=make_libero_dataset(**dataset_kwargs), + batch_size=batch_size, + num_workers=num_workers, + training_iterations=training_iterations, + ) + + # Match libero_experiment.libero_exp post-build mutations. + for i, d in enumerate(exp["defaults"]): + if isinstance(d, dict) and "override /callbacks" in d: + exp["defaults"][i] = { + "override /callbacks": [ + "basic", + "optimization", + "job_monitor", + "training_stats", + ] + } + break + exp["scheduler"]["f_max"] = [1.0] + exp["scheduler"]["warm_up_steps"] = [training_iterations // 20] + + exp["job"]["group"] = job_group + if job_project is not None: + exp["job"]["project"] = job_project + return exp + + +def _build_libero_policy_base_8b( + exp_name: str, + *, + training_iterations: int = LIBERO_BASELINE_TRAINING_ITERATIONS, + batch_size: int = LIBERO_BASELINE_BATCH_SIZE, + num_workers: int = LIBERO_BASELINE_NUM_WORKERS, + job_group: str = "debugging", + job_project: str | None = None, + mode: str = "policy", + keys_to_skip_loading: list[str] | None = None, + use_deterministic_seed: bool = False, + **dataset_kwargs, +) -> dict: + """8B counterpart to ``_build_libero_policy_base``. + + Mirrors the libero post-build mutations (callbacks override, scheduler + f_max=1.0, warm_up = iter//20) but builds from ``make_8b_experiment`` + instead of the 2B base. + """ + dataset_kwargs.setdefault("mode", mode) + extra_kwargs: dict = {} + if keys_to_skip_loading is not None: + extra_kwargs["keys_to_skip_loading"] = keys_to_skip_loading + exp = make_8b_experiment( + exp_name=exp_name, + datasets=make_libero_dataset(**dataset_kwargs), + batch_size=batch_size, + num_workers=num_workers, + training_iterations=training_iterations, + use_deterministic_seed=use_deterministic_seed, + **extra_kwargs, + ) + + for i, d in enumerate(exp["defaults"]): + if isinstance(d, dict) and "override /callbacks" in d: + exp["defaults"][i] = { + "override /callbacks": [ + "basic", + "optimization", + "job_monitor", + "training_stats", + ] + } + break + exp["scheduler"]["f_max"] = [1.0] + exp["scheduler"]["warm_up_steps"] = [training_iterations // 20] + + exp["trainer"]["callbacks"]["compile_tokenizer"]["enabled"] = True + exp["trainer"]["callbacks"]["compile_tokenizer"]["warmup_resolutions"] = ["256", "480", "720"] + + exp["job"]["group"] = job_group + if job_project is not None: + exp["job"]["project"] = job_project + return exp + + +# --------------------------------------------------------------------------- +# libero_exp_policy — base policy experiment +# --------------------------------------------------------------------------- +libero_exp_policy = _build_libero_policy_base( + exp_name=f"libero_exp_policy_{datetime.now().strftime('%Y%m%d_%H%M%S')}", +) +cs.store( + group="experiment", + package="_global_", + name="libero_exp_policy", + node=libero_exp_policy, +) + + +# --------------------------------------------------------------------------- +# LR sweep on libero_exp_policy +# --------------------------------------------------------------------------- +def lr_sweep(): + for lr_value, suffix in [(4e-4, "lr4e4"), (2e-4, "lr2e4"), (8e-4, "lr8e4")]: + name = f"libero_exp_policy_{suffix}" + exp = _build_libero_policy_base(exp_name=name, job_group="action_libero") + exp["optimizer"]["lr"] = lr_value + cs.store(group="experiment", package="_global_", name=name, node=exp) + + +# --------------------------------------------------------------------------- +# Multi-view policy experiment (camera_mode=concat_view) +# --------------------------------------------------------------------------- +def multi_view_policy_exp(): + """ + CUDA_VISIBLE_DEVICES=0 PYTHONPATH=. torchrun --nproc_per_node=1 --master_port=12341 cosmos_framework/scripts/train.py --config=cosmos_framework/configs/base/config.py -- experiment=libero_exp_policy_multi_view_lr2e4 + """ + name = "libero_exp_policy_multi_view_lr2e4" + exp = _build_libero_policy_base( + exp_name=name, + job_group="action_libero", + job_project="cosmos3_action_libero", + camera_mode="concat_view", + ) + cs.store(group="experiment", package="_global_", name=name, node=exp) + + """ + CUDA_VISIBLE_DEVICES=0 PYTHONPATH=. torchrun --nproc_per_node=1 --master_port=12341 cosmos_framework/scripts/train.py --config=cosmos_framework/configs/base/config.py -- experiment=libero_exp_policy_multi_view_lr2e4_20k + """ + name_20k = "libero_exp_policy_multi_view_lr2e4_20k" + exp_20k = _build_libero_policy_base( + exp_name=name_20k, + training_iterations=20_000, + job_group="action_libero", + job_project="cosmos3_action_libero", + camera_mode="concat_view", + ) + cs.store(group="experiment", package="_global_", name=name_20k, node=exp_20k) + + +# --------------------------------------------------------------------------- +# Single-view 60k iter policy experiment +# --------------------------------------------------------------------------- +def single_view_policy_exp(): + """ + CUDA_VISIBLE_DEVICES=0 PYTHONPATH=. torchrun --nproc_per_node=1 --master_port=12341 cosmos_framework/scripts/train.py --config=cosmos_framework/configs/base/config.py -- experiment=libero_exp_policy_single_view_lr2e4_60k + """ + name = "libero_exp_policy_single_view_lr2e4_60k" + exp = _build_libero_policy_base( + exp_name=name, + training_iterations=60_000, + batch_size=256, + job_group="action_libero", + job_project="cosmos3_action_libero", + ) + exp["optimizer"]["lr"] = 2e-4 + exp["scheduler"]["warm_up_steps"] = [100] + exp["scheduler"]["cycle_lengths"] = [60_000] + cs.store(group="experiment", package="_global_", name=name, node=exp) + + +# --------------------------------------------------------------------------- +# Normalized-action policy experiment (with image / wrist_image variants) +# --------------------------------------------------------------------------- +def normalized_action_policy_exp(): + """ + CUDA_VISIBLE_DEVICES=0 PYTHONPATH=. torchrun --nproc_per_node=1 --master_port=12341 cosmos_framework/scripts/train.py --config=cosmos_framework/configs/base/config.py -- experiment=libero_exp_policy_norm_action_lr2e4_60k + """ + # Use the shared action stats JSON through LIBERODataset's new normalization API. + name = "libero_exp_policy_norm_action_lr2e4_60k" + exp = _build_libero_policy_base( + exp_name=name, + training_iterations=60_000, + batch_size=256, + job_group="action_libero", + job_project="cosmos3_action_libero", + action_normalization="minmax", + action_stats_path=ACTION_STATS_PATH, + ) + exp["optimizer"]["lr"] = 2e-4 + exp["scheduler"]["warm_up_steps"] = [100] + exp["scheduler"]["cycle_lengths"] = [60_000] + exp["checkpoint"]["load_from_object_store"] = dict(bucket="nv-00-10206-checkpoint") + cs.store(group="experiment", package="_global_", name=name, node=exp) + + """ + CUDA_VISIBLE_DEVICES=0 PYTHONPATH=. torchrun --nproc_per_node=1 --master_port=12341 cosmos_framework/scripts/train.py --config=cosmos_framework/configs/base/config.py -- experiment=libero_exp_policy_norm_action_wrist_lr2e4_60k + """ + wrist_name = "libero_exp_policy_norm_action_wrist_lr2e4_60k" + wrist_exp = _build_libero_policy_base( + exp_name=wrist_name, + training_iterations=60_000, + batch_size=256, + job_group="action_libero", + job_project="cosmos3_action_libero", + action_normalization="minmax", + action_stats_path=ACTION_STATS_PATH, + camera_mode="wrist_image", + ) + wrist_exp["optimizer"]["lr"] = 2e-4 + wrist_exp["scheduler"]["warm_up_steps"] = [100] + wrist_exp["scheduler"]["cycle_lengths"] = [60_000] + wrist_exp["checkpoint"]["load_from_object_store"] = dict(bucket="nv-00-10206-checkpoint") + cs.store(group="experiment", package="_global_", name=wrist_name, node=wrist_exp) + + +# --------------------------------------------------------------------------- +# Wrist-view 60k iter policy experiment +# --------------------------------------------------------------------------- +def wrist_view_policy_exp(): + """ + CUDA_VISIBLE_DEVICES=0 PYTHONPATH=. torchrun --nproc_per_node=1 --master_port=12341 cosmos_framework/scripts/train.py --config=cosmos_framework/configs/base/config.py -- experiment=libero_exp_policy_wrist_view_lr2e4_60k + """ + name = "libero_exp_policy_wrist_view_lr2e4_60k" + exp = _build_libero_policy_base( + exp_name=name, + training_iterations=60_000, + batch_size=256, + job_group="action_libero", + job_project="cosmos3_action_libero", + camera_mode="wrist_image", + ) + exp["optimizer"]["lr"] = 2e-4 + exp["scheduler"]["warm_up_steps"] = [100] + exp["scheduler"]["cycle_lengths"] = [60_000] + exp["checkpoint"]["load_from_object_store"] = dict(bucket="nv-00-10206-checkpoint") + cs.store(group="experiment", package="_global_", name=name, node=exp) + + +# --------------------------------------------------------------------------- +# Sweep camera_mode × action_space × rotation_space (lr=2e-4) +# --------------------------------------------------------------------------- +def action_camera_rotation_sweep(): + camera_modes = ["image", "wrist_image"] + action_spaces = ["relative", "frame_wise_relative"] + rotation_spaces = ["9d", "6d", "3d"] + + for camera_mode in camera_modes: + for action_space in action_spaces: + for rotation_space in rotation_spaces: + cam_short = "img" if camera_mode == "image" else "wrist" + act_short = "rel" if action_space == "relative" else "fwrel" + rot_short = rotation_space + name = f"libero_exp_policy_{cam_short}_{act_short}_{rot_short}_varlen" + + exp = _build_libero_policy_base( + exp_name=name, + job_group="action_libero", + camera_mode=camera_mode, + action_space=action_space, + rotation_space=rotation_space, + ) + exp["optimizer"]["lr"] = 2e-4 + cs.store(group="experiment", package="_global_", name=name, node=exp) + + +# --------------------------------------------------------------------------- +# Normalized-action joint experiments (libero-90 removed for faster convergence). +# These use mode="policy" or mode="forward_dynamics" despite the name. +# Build the libero dataset then layer further variants. +# --------------------------------------------------------------------------- +def normalized_action_joint_exp(): + # Common dataset-kwargs base for the libero wrist, normalized variants. + wrist_norm_kwargs = dict( + repo_id=copy.deepcopy(_LIBERO_REPO_IDS), + root=copy.deepcopy(_LIBERO_ROOTS), + camera_mode="wrist_image", + action_normalization="minmax", + action_stats_path=ACTION_STATS_PATH, + ) + + _UNSET = object() + + def _make( + exp_name: str, + *, + mode: str = "policy", + max_iter: int = 20_000, + cycle_lengths: list | None = None, + warm_up_steps: list | None = None, + f_min: list | None = None, + lr: float = 2e-4, + batch_size: int = 256, + num_workers: int = 24, + cfg_dropout_rate: float | None = None, + action_loss_weight: float | None = None, + max_samples_per_batch: int | None = None, + max_sequence_length=_UNSET, # sentinel: pass _UNSET to keep parent default + run_validation_on_start: bool | None = None, + logging_iter: int | None = None, + validation_iter: int | None = None, + max_val_iter: int | None = None, + load_path: str | None = None, + action_channel_masking: bool | None = None, + fps: int | None = None, + ) -> dict: + ds_kwargs = dict(wrist_norm_kwargs) + ds_kwargs["mode"] = mode + if fps is not None: + ds_kwargs["fps"] = fps + + exp = _build_libero_policy_base( + exp_name=exp_name, + training_iterations=max_iter, + batch_size=batch_size, + num_workers=num_workers, + job_group="action_libero", + job_project="cosmos3_action_libero", + mode=mode, + **{k: v for k, v in ds_kwargs.items() if k != "mode"}, + ) + exp["optimizer"]["lr"] = lr + exp["scheduler"]["f_min"] = f_min if f_min is not None else [0.0] + exp["scheduler"]["warm_up_steps"] = warm_up_steps if warm_up_steps is not None else [2000] + exp["scheduler"]["cycle_lengths"] = cycle_lengths if cycle_lengths is not None else [max_iter] + exp["trainer"]["max_iter"] = max_iter + exp["trainer"]["run_validation"] = False + if logging_iter is not None: + exp["trainer"]["logging_iter"] = logging_iter + if run_validation_on_start is not None: + exp["trainer"]["run_validation_on_start"] = run_validation_on_start + if validation_iter is not None: + exp["trainer"]["validation_iter"] = validation_iter + if max_val_iter is not None: + exp["trainer"]["max_val_iter"] = max_val_iter + exp["checkpoint"]["load_from_object_store"] = dict(bucket="nv-00-10206-checkpoint-experiments") + exp["checkpoint"]["save_to_object_store"] = dict(bucket="nv-00-10206-checkpoint-experiments") + if load_path is not None: + exp["checkpoint"]["load_path"] = load_path + + # dataloader-level overrides (action_channel_masking, cfg_dropout_rate, max_samples_per_batch). + ds_node = exp["dataloader_train"]["dataloaders"]["action_data"]["dataloader"]["dataset"] + if cfg_dropout_rate is not None: + ds_node["cfg_dropout_rate"] = cfg_dropout_rate + if action_channel_masking is not None: + ds_node["action_channel_masking"] = action_channel_masking + if max_samples_per_batch is not None: + exp["dataloader_train"]["max_samples_per_batch"] = max_samples_per_batch + if max_sequence_length is not _UNSET: + # Setting to None disables sequence packing (each sample its own batch slot). + # Default from make_2b_experiment is the ${model.config.max_num_tokens_after_packing} + # interpolation; pass _UNSET to keep that. + exp["dataloader_train"]["max_sequence_length"] = max_sequence_length + + if action_loss_weight is not None: + exp["model"]["config"]["rectified_flow_training_config"]["action_loss_weight"] = action_loss_weight + + return exp + + # libero_exp_policy_norm_action_wrist_lr2e4_20k — the root of this chain. + name = "libero_exp_policy_norm_action_wrist_lr2e4_20k" + exp = _make(name, mode="policy", lr=2e-4, max_iter=20_000) + cs.store(group="experiment", package="_global_", name=name, node=exp) + + # _no_cfg_20k = parent + cfg_dropout_rate=0.0 + name = "libero_exp_policy_norm_action_wrist_lr2e4_no_cfg_20k" + exp = _make(name, mode="policy", lr=2e-4, max_iter=20_000, cfg_dropout_rate=0.0) + cs.store(group="experiment", package="_global_", name=name, node=exp) + + # _no_cfg_bs64_20k = parent + batch_size=64 + name = "libero_exp_policy_norm_action_wrist_lr2e4_no_cfg_bs64_20k" + exp = _make(name, mode="policy", lr=2e-4, max_iter=20_000, cfg_dropout_rate=0.0, batch_size=64) + cs.store(group="experiment", package="_global_", name=name, node=exp) + + # _lr1e3_no_cfg_bs64_20k = bs64_20k + lr=1e-3 + name = "libero_exp_policy_norm_action_wrist_lr1e3_no_cfg_bs64_20k" + exp = _make(name, mode="policy", lr=1e-3, max_iter=20_000, cfg_dropout_rate=0.0, batch_size=64) + cs.store(group="experiment", package="_global_", name=name, node=exp) + + # _lr1e3_no_cfg_bs256_20k = no_cfg_20k + lr=1e-3 (bs=256 inherited) + name = "libero_exp_policy_norm_action_wrist_lr1e3_no_cfg_bs256_20k" + exp = _make(name, mode="policy", lr=1e-3, max_iter=20_000, cfg_dropout_rate=0.0) + cs.store(group="experiment", package="_global_", name=name, node=exp) + + # action10 variants — action_loss_weight=100.0 + various LR + for lr_v, suffix in [(4e-5, "lr4e5"), (2e-5, "lr2e5"), (1e-4, "lr1e4"), (3e-4, "lr3e4")]: + name = f"libero_exp_policy_norm_action_wrist_{suffix}_no_cfg_bs256_action10_20k" + exp = _make( + name, + mode="policy", + lr=lr_v, + max_iter=20_000, + cfg_dropout_rate=0.0, + action_loss_weight=100.0, + ) + cs.store(group="experiment", package="_global_", name=name, node=exp) + + # _lr5e4_no_cfg_30k — adds validation overrides + lr=5e-4 + cfg=0 + name = "libero_exp_policy_norm_action_wrist_lr5e4_no_cfg_30k" + exp = _make( + name, + mode="policy", + lr=5e-4, + max_iter=20_000, + cfg_dropout_rate=0.0, + run_validation_on_start=False, + validation_iter=1000, + max_val_iter=40, + ) + cs.store(group="experiment", package="_global_", name=name, node=exp) + + # _lr5e4_no_cfg_30k_resume — same + load_path override + name = "libero_exp_policy_norm_action_wrist_lr5e4_no_cfg_30k_resume" + exp = _make( + name, + mode="policy", + lr=5e-4, + max_iter=20_000, + cfg_dropout_rate=0.0, + run_validation_on_start=False, + validation_iter=1000, + max_val_iter=40, + load_path="cosmos3_uva_libero/uva_libero/libero_exp_policy_norm_action_wrist_lr2e4_no_cfg_bs64_20k/checkpoints/iter_000018500", + ) + cs.store(group="experiment", package="_global_", name=name, node=exp) + + # _lr5e4_no_cfg_40k — 40k iters, max_samples_per_batch=256, max_sequence_length=None, + # logging_iter=10, run_validation_on_start=False, lr=1e-4 (override at the end) + name = "libero_exp_policy_norm_action_wrist_lr5e4_no_cfg_40k" + exp = _make( + name, + mode="policy", + lr=1e-4, + max_iter=40_000, + cycle_lengths=[40_000], + cfg_dropout_rate=0.0, + run_validation_on_start=False, + logging_iter=10, + max_samples_per_batch=256, + max_sequence_length=None, + ) + cs.store(group="experiment", package="_global_", name=name, node=exp) + + # _lr5e4_no_cfg_40k_20fps_re — 40k iters, fps=20, lr=5e-5 + name = "libero_exp_policy_norm_action_wrist_lr5e4_no_cfg_40k_20fps_re" + exp_40k_20fps = _make( + name, + mode="policy", + lr=5e-5, + max_iter=40_000, + cycle_lengths=[40_000], + cfg_dropout_rate=0.0, + run_validation_on_start=False, + logging_iter=100, + max_samples_per_batch=256, + max_sequence_length=None, + fps=20, + ) + cs.store(group="experiment", package="_global_", name=name, node=exp_40k_20fps) + + # _fd_norm_action_wrist_lr5e4_no_cfg_20k_20fps — same chain, 20k iters, mode=forward_dynamics + name = "libero_exp_fd_norm_action_wrist_lr5e4_no_cfg_20k_20fps" + exp = _make( + name, + mode="forward_dynamics", + lr=5e-5, + max_iter=20_000, + cycle_lengths=[20_000], + cfg_dropout_rate=0.0, + run_validation_on_start=False, + logging_iter=100, + max_samples_per_batch=256, + max_sequence_length=None, + fps=20, + ) + cs.store(group="experiment", package="_global_", name=name, node=exp) + + +# --------------------------------------------------------------------------- +# Action-design ablation study — joint and policy mode variants. +# All inherit settings from libero_exp_policy_norm_action_wrist_lr5e4_no_cfg_40k_20fps_re +# (libero wrist, action_normalization="minmax", fps=20, max_samples_per_batch=256). +# --------------------------------------------------------------------------- +def action_design_ablation(): + wrist_norm_kwargs = dict( + repo_id=copy.deepcopy(_LIBERO_REPO_IDS), + root=copy.deepcopy(_LIBERO_ROOTS), + camera_mode="wrist_image", + action_normalization="minmax", + action_stats_path=ACTION_STATS_PATH, + fps=20, + ) + + def _make_ablation_2b( + exp_name: str, + *, + mode: str, + action_loss_weight: float = 100.0, + action_channel_masking: bool = True, + lr_multipliers: dict | None = None, + cfg_dropout_rate: float = 0.1, + dataset_overrides: dict | None = None, + ) -> dict: + ds_kwargs = dict(wrist_norm_kwargs) + if dataset_overrides is not None: + ds_kwargs.update(dataset_overrides) + exp = _build_libero_policy_base( + exp_name=exp_name, + training_iterations=20_000, + batch_size=256, + num_workers=24, + job_group="action_libero", + job_project="cosmos3_action_libero", + mode=mode, + **ds_kwargs, + ) + exp["optimizer"]["lr"] = 1e-4 + if lr_multipliers is not None: + exp["optimizer"]["lr_multipliers"] = lr_multipliers + exp["scheduler"]["f_min"] = [0.0] + exp["scheduler"]["warm_up_steps"] = [2000] + exp["scheduler"]["cycle_lengths"] = [20_000] + exp["trainer"]["max_iter"] = 20_000 + exp["trainer"]["logging_iter"] = 100 + exp["trainer"]["run_validation"] = False + exp["trainer"]["run_validation_on_start"] = False + exp["model"]["config"]["rectified_flow_training_config"]["action_loss_weight"] = action_loss_weight + # cfg_dropout_rate + action_channel_masking are inside dataloader_train. + ds_node = exp["dataloader_train"]["dataloaders"]["action_data"]["dataloader"]["dataset"] + ds_node["cfg_dropout_rate"] = cfg_dropout_rate + ds_node["action_channel_masking"] = action_channel_masking + exp["dataloader_train"]["max_samples_per_batch"] = 256 + exp["dataloader_train"]["max_sequence_length"] = None + exp["checkpoint"]["load_from_object_store"] = dict(bucket="nv-00-10206-checkpoint-experiments") + exp["checkpoint"]["save_to_object_store"] = dict(bucket="nv-00-10206-checkpoint-experiments") + return exp + + # First register the lr_sweep_lr1e4 anchor used by ablations as their parent + # (mode=joint, cfg_dropout_rate=0.1, action_loss_weight=10.0 inherited). + name = "libero_exp_policy_norm_action_wrist_no_cfg_40k_20fps_lr_sweep_lr1e4" + exp = _make_ablation_2b( + name, + mode="joint", + action_loss_weight=10.0, # inherited from base 2B make_2b_experiment + ) + # _lr_sweep_lr1e4 keeps the 40k schedule, not 20k. + exp["trainer"]["max_iter"] = 40_000 + exp["scheduler"]["cycle_lengths"] = [40_000] + cs.store(group="experiment", package="_global_", name=name, node=exp) + + # Joint ablations — all 20k, action_loss_weight=100 except *_wo_action_loss_reweighting + cs.store( + group="experiment", + package="_global_", + name="libero_exp_ablation_study_joint_baseline", + node=_make_ablation_2b( + "libero_exp_ablation_study_joint_baseline", + mode="joint", + action_loss_weight=100.0, + ), + ) + + cs.store( + group="experiment", + package="_global_", + name="libero_exp_ablation_study_joint_wo_action_masking", + node=_make_ablation_2b( + "libero_exp_ablation_study_joint_wo_action_masking", + mode="joint", + action_loss_weight=100.0, + action_channel_masking=False, + ), + ) + + cs.store( + group="experiment", + package="_global_", + name="libero_exp_ablation_study_joint_wo_action_loss_reweighting", + node=_make_ablation_2b( + "libero_exp_ablation_study_joint_wo_action_loss_reweighting", + mode="joint", + action_loss_weight=10.0, + ), + ) + + cs.store( + group="experiment", + package="_global_", + name="libero_exp_ablation_study_joint_w_action_lr_scaling", + node=_make_ablation_2b( + "libero_exp_ablation_study_joint_w_action_lr_scaling", + mode="joint", + action_loss_weight=10.0, + lr_multipliers={ + "action2llm": 5.0, + "llm2action": 5.0, + "action_modality_embed": 5.0, + }, + ), + ) + + # Policy ablations — same as joint but mode=policy + cs.store( + group="experiment", + package="_global_", + name="libero_exp_ablation_study_policy_baseline", + node=_make_ablation_2b( + "libero_exp_ablation_study_policy_baseline", + mode="policy", + action_loss_weight=100.0, + ), + ) + + cs.store( + group="experiment", + package="_global_", + name="libero_exp_ablation_study_policy_wo_action_masking", + node=_make_ablation_2b( + "libero_exp_ablation_study_policy_wo_action_masking", + mode="policy", + action_loss_weight=100.0, + action_channel_masking=False, + ), + ) + + cs.store( + group="experiment", + package="_global_", + name="libero_exp_ablation_study_policy_wo_action_loss_reweighting", + node=_make_ablation_2b( + "libero_exp_ablation_study_policy_wo_action_loss_reweighting", + mode="policy", + action_loss_weight=10.0, + ), + ) + + cs.store( + group="experiment", + package="_global_", + name="libero_exp_ablation_study_policy_w_action_lr_scaling", + node=_make_ablation_2b( + "libero_exp_ablation_study_policy_w_action_lr_scaling", + mode="policy", + action_loss_weight=10.0, + lr_multipliers={ + "action2llm": 5.0, + "llm2action": 5.0, + "action_modality_embed": 5.0, + }, + ), + ) + + +# --------------------------------------------------------------------------- +# 8B variants of libero_exp_ablation_study_policy_w_action_lr_scaling. +# * _8b — built on the default 8B pretrained checkpoint from +# ``make_8b_experiment`` (cosmos3 8B multires recipe). +# * _8b_midtrain_v1p1 — built on the action mid-training checkpoint +# ``cosmos3_action/midtrain_v1p1/action_midtrain_exp005_v1p1_equalTokens +# /checkpoints/iter_000013000/`` (action layers loaded too). +# * _8b_midtrain_v1p1_concat_view — same mid-training checkpoint, but with +# ``camera_mode="concat_view"`` for concatenated third-person +# and wrist views. +# * _8b_midtrain_v1p1_concat_view_rot6d_native — same as concat_view, but +# with native-frame 6D rotations and no OpenCV pose +# conversion. +# * _8b_midtrain_v1p1_concat_view_rot6d_native_no_rot_norm_quantile — same as +# rot6d_native, but uses quantile action normalization +# while leaving 6D rotation dims unnormalized. +# * _8b_midtrain_v1p1_concat_view_rot6d_native_rot_norm_quantile — same as +# rot6d_native, but normalizes 6D rotation dims too +# with quantile action normalization. +# * *_det_seed — deterministic-dataloader-seed variants of the two +# native quantile experiments. +# * *_det_seed_iter40k — same as the rot_norm_quantile deterministic-seed +# variant, but warm-starts from the 40k action-midtrain +# checkpoint. +# * _8b_exp506_midtraining_v1_*_iter40k — same deterministic native normalization +# setups, but warm-start from the exp506 VFM base +# checkpoint and initialize action layers fresh. +# * _8b_concat_view — same as _8b but with ``camera_mode="concat_view"`` so the +# third-person and wrist views are concatenated along +# width into a single (H, 2*W) frame fed to the model. +# All other hparams (lr, schedule, dataloader settings, action_loss_weight, +# lr_multipliers) are kept identical to the 2B parent. +# --------------------------------------------------------------------------- +def action_design_ablation_8b(): + def _wrist_norm_kwargs(camera_mode: str) -> dict: + return dict( + repo_id=copy.deepcopy(_LIBERO_REPO_IDS), + root=copy.deepcopy(_LIBERO_ROOTS), + camera_mode=camera_mode, + action_normalization="minmax", + action_stats_path=ACTION_STATS_PATH, + fps=20, + ) + + def _make_ablation_8b( + exp_name: str, + *, + camera_mode: str = "wrist_image", + load_path: str | None = None, + keys_to_skip_loading: list[str] | None = None, + dataset_overrides: dict | None = None, + use_deterministic_seed: bool = False, + batch_size: int = 256, + num_workers: int = 24, + lr: float = 1e-4, + training_iterations: int = 20_000, + warm_up_steps: int = 2000, + grad_accum_iter: int = 1, + max_samples_per_batch: int | None = 256, + cfg_dropout_rate: float = 0.1, + action_channel_masking: bool = True, + action_loss_weight: float = 10.0, + action_param_lr_multiplier: float = 5.0, + shard_across_workers: bool | None = None, + ) -> dict: + ds_kwargs = _wrist_norm_kwargs(camera_mode) + if dataset_overrides is not None: + ds_kwargs.update(dataset_overrides) + exp = _build_libero_policy_base_8b( + exp_name=exp_name, + training_iterations=training_iterations, + batch_size=batch_size, + num_workers=num_workers, + job_group="action_libero", + job_project="cosmos3_action_libero", + mode="policy", + keys_to_skip_loading=keys_to_skip_loading, + use_deterministic_seed=use_deterministic_seed, + **ds_kwargs, + ) + exp["optimizer"]["lr"] = lr + exp["optimizer"]["lr_multipliers"] = { + "action2llm": action_param_lr_multiplier, + "llm2action": action_param_lr_multiplier, + "action_modality_embed": action_param_lr_multiplier, + } + exp["scheduler"]["f_min"] = [0.0] + exp["scheduler"]["warm_up_steps"] = [warm_up_steps] + exp["scheduler"]["cycle_lengths"] = [training_iterations] + exp["trainer"]["max_iter"] = training_iterations + exp["trainer"]["logging_iter"] = 100 + exp["trainer"]["run_validation"] = False + exp["trainer"]["run_validation_on_start"] = False + exp["trainer"]["grad_accum_iter"] = grad_accum_iter + exp["model"]["config"]["rectified_flow_training_config"]["action_loss_weight"] = action_loss_weight + ds_node = exp["dataloader_train"]["dataloaders"]["action_data"]["dataloader"]["dataset"] + ds_node["cfg_dropout_rate"] = cfg_dropout_rate + ds_node["action_channel_masking"] = action_channel_masking + if shard_across_workers is not None: + ds_node["shard_across_workers"] = shard_across_workers + exp["dataloader_train"]["max_samples_per_batch"] = max_samples_per_batch + exp["dataloader_train"]["max_sequence_length"] = None + exp["checkpoint"]["load_from_object_store"] = dict(bucket="nv-00-10206-checkpoint-experiments") + exp["checkpoint"]["save_to_object_store"] = dict(bucket="nv-00-10206-checkpoint-experiments") + if load_path is not None: + exp["checkpoint"]["load_path"] = load_path + return exp + + cs.store( + group="experiment", + package="_global_", + name="libero_exp_ablation_study_policy_w_action_lr_scaling_8b", + node=_make_ablation_8b("libero_exp_ablation_study_policy_w_action_lr_scaling_8b"), + ) + + # bucket nv-00-10206-checkpoint-experiments matches the gcp default, so we + # only need to override load_path. keys_to_skip_loading=[] preserves the + # action layers that are already trained in the mid-training checkpoint. + cs.store( + group="experiment", + package="_global_", + name="libero_exp_ablation_study_policy_w_action_lr_scaling_8b_midtrain_v1p1", + node=_make_ablation_8b( + "libero_exp_ablation_study_policy_w_action_lr_scaling_8b_midtrain_v1p1", + load_path=( + "cosmos3_action/midtrain_v1p1/action_midtrain_exp005_v1p1_equalTokens/checkpoints/iter_000013000/" + ), + keys_to_skip_loading=[], + ), + ) + + cs.store( + group="experiment", + package="_global_", + name="libero_exp_ablation_study_policy_w_action_lr_scaling_8b_midtrain_v1p1_concat_view", + node=_make_ablation_8b( + "libero_exp_ablation_study_policy_w_action_lr_scaling_8b_midtrain_v1p1_concat_view", + camera_mode="concat_view", + load_path=( + "cosmos3_action/midtrain_v1p1/action_midtrain_exp005_v1p1_equalTokens/checkpoints/iter_000013000/" + ), + keys_to_skip_loading=[], + ), + ) + + cs.store( + group="experiment", + package="_global_", + name="libero_exp_ablation_study_policy_w_action_lr_scaling_8b_exp506_midtraining_v1_concat_view_rot6d_native_no_rot_norm_quantile_det_seed_iter40k", + node=_make_ablation_8b( + "libero_exp_ablation_study_policy_w_action_lr_scaling_8b_exp506_midtraining_v1_concat_view_rot6d_native_no_rot_norm_quantile_det_seed_iter40k", + camera_mode="concat_view", + load_path=( + "cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/" + "t2w_mot_exp506_000_qwen3_vl_8b_multires_recipe_midtraining_v1/checkpoints/iter_000040000/" + ), + use_deterministic_seed=True, + dataset_overrides={ + "action_space": "frame_wise_relative", + "rotation_space": "6d", + "pose_coordinate_frame": "native", + "action_normalization": "quantile", + "action_stats_path": NATIVE_FRAME_WISE_ROT6D_ACTION_STATS_PATH, + }, + ), + ) + + cs.store( + group="experiment", + package="_global_", + name="libero_exp_ablation_study_policy_w_action_lr_scaling_8b_exp506_midtraining_v1_concat_view_rot6d_native_rot_norm_quantile_det_seed_iter40k", + node=_make_ablation_8b( + "libero_exp_ablation_study_policy_w_action_lr_scaling_8b_exp506_midtraining_v1_concat_view_rot6d_native_rot_norm_quantile_det_seed_iter40k", + camera_mode="concat_view", + load_path=( + "cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/" + "t2w_mot_exp506_000_qwen3_vl_8b_multires_recipe_midtraining_v1/checkpoints/iter_000040000/" + ), + use_deterministic_seed=True, + dataset_overrides={ + "action_space": "frame_wise_relative", + "rotation_space": "6d", + "pose_coordinate_frame": "native", + "action_normalization": "quantile_rot", + "action_stats_path": NATIVE_FRAME_WISE_ROT6D_ACTION_STATS_PATH, + }, + ), + ) + + cs.store( + group="experiment", + package="_global_", + name="libero_exp_ablation_study_policy_w_action_lr_scaling_8b_exp506_midtraining_v1_concat_view_rot6d_native_no_rot_norm_minmax_det_seed_iter40k", + node=_make_ablation_8b( + "libero_exp_ablation_study_policy_w_action_lr_scaling_8b_exp506_midtraining_v1_concat_view_rot6d_native_no_rot_norm_minmax_det_seed_iter40k", + camera_mode="concat_view", + load_path=( + "cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/" + "t2w_mot_exp506_000_qwen3_vl_8b_multires_recipe_midtraining_v1/checkpoints/iter_000040000/" + ), + use_deterministic_seed=True, + dataset_overrides={ + "action_space": "frame_wise_relative", + "rotation_space": "6d", + "pose_coordinate_frame": "native", + "action_normalization": "minmax", + "action_stats_path": NATIVE_FRAME_WISE_ROT6D_ACTION_STATS_PATH, + }, + ), + ) + + cs.store( + group="experiment", + package="_global_", + name="libero_exp_ablation_study_policy_w_action_lr_scaling_8b_exp506_midtraining_v1_concat_view_rot6d_native_rot_norm_minmax_det_seed_iter40k", + node=_make_ablation_8b( + "libero_exp_ablation_study_policy_w_action_lr_scaling_8b_exp506_midtraining_v1_concat_view_rot6d_native_rot_norm_minmax_det_seed_iter40k", + camera_mode="concat_view", + load_path=( + "cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/" + "t2w_mot_exp506_000_qwen3_vl_8b_multires_recipe_midtraining_v1/checkpoints/iter_000040000/" + ), + use_deterministic_seed=True, + dataset_overrides={ + "action_space": "frame_wise_relative", + "rotation_space": "6d", + "pose_coordinate_frame": "native", + "action_normalization": "minmax", + "action_stats_path": NATIVE_FRAME_WISE_ROT6D_ACTION_STATS_PATH, + }, + ), + ) + + # Concatenated third-person + wrist view (camera_mode="concat_view" stitches the + # two cameras horizontally inside LIBERODataset, producing a (H, 2*W) frame + # tagged with viewpoint="concat_view"). + cs.store( + group="experiment", + package="_global_", + name="libero_exp_ablation_study_policy_w_action_lr_scaling_8b_concat_view", + node=_make_ablation_8b( + "libero_exp_ablation_study_policy_w_action_lr_scaling_8b_concat_view", + camera_mode="concat_view", + ), + ) + + +def action_policy_sft_8b_experiments(): + """8B policy SFT on LIBERO-4 with frame-wise relative actions, 6D rotation, concat_view. + + Converted from cosmos3-internal configs/experiment/action_policy_sft_8b.yaml. + + API caveats vs the original YAML: + - action_normalization='quantile_rot' is not in i4 (i4 only has normalize_action: bool). + To restore full fidelity: add normalizers/libero_native_frame_wise_relative_rot6d.json, + then set normalize_action=True + action_stats_path pointing to that file. + - format_prompt_as_json and pose_coordinate_frame are cosmos3-internal-only; omitted. + - load_path is a local outputs/ path from the original; override for cluster/S3 runs. + """ + exp = _build_libero_policy_base_8b( + "action_policy_sft_8b", + training_iterations=16_000, + batch_size=256, + num_workers=4, + job_group="action_libero", + job_project="cosmos3_action_libero", + mode="policy", + keys_to_skip_loading=["net_ema.", "action2llm", "llm2action", "action_modality_embed", "action_pos_embed"], + # LIBERODataset kwargs forwarded via **dataset_kwargs: + repo_id=copy.deepcopy(_LIBERO_REPO_IDS), + root=copy.deepcopy(_LIBERO_ROOTS), + fps=20, + camera_mode="concat_view", + action_space="frame_wise_relative", + rotation_space="6d", + chunk_length=16, + seed=0, + val_ratio=0.01, + ) + + # Scheduler: f_max=1.0, warm_up=500; cycle_lengths=20000 intentionally longer than max_iter + exp["scheduler"]["f_max"] = [1.0] + exp["scheduler"]["f_min"] = [0.0] + exp["scheduler"]["warm_up_steps"] = [500] + exp["scheduler"]["cycle_lengths"] = [20_000] + + # Optimizer: lr=5e-5 + exp["optimizer"]["lr"] = 5e-5 + + # Trainer + exp["trainer"]["max_iter"] = 16_000 + exp["trainer"]["logging_iter"] = 100 + exp["trainer"]["run_validation"] = False + exp["trainer"]["run_validation_on_start"] = False + + # Checkpoint: set to None by default; override via checkpoint.load_path= in the launch script + # or set LIBERO_ACTION_CHECKPOINT_PATH env var to the pretrained checkpoint directory. + exp["checkpoint"]["load_path"] = "outputs/checkpoints/action_policy_sft_8b" + exp["checkpoint"]["load_training_state"] = False + exp["checkpoint"]["save_iter"] = 500 + + # compile_tokenizer: make_8b_experiment sets enabled=True but drops warmup_resolutions + exp["trainer"]["callbacks"]["compile_tokenizer"]["warmup_resolutions"] = ["256", "480", "720"] + + # Model overrides + exp["model"]["config"]["rectified_flow_training_config"]["action_loss_weight"] = 10.0 + exp["model"]["config"]["num_embodiment_domains"] = 32 + + # wrap_dataset overrides + ds_node = exp["dataloader_train"]["dataloaders"]["action_data"]["dataloader"]["dataset"] + ds_node["action_channel_masking"] = True + + # Dataloader: disable sequence packing (max_sequence_length=None), cap batch budget + exp["dataloader_train"]["max_samples_per_batch"] = 256 + exp["dataloader_train"]["max_sequence_length"] = None + + cs.store(group="experiment", package="_global_", name="action_policy_sft_8b", node=exp) + + +lr_sweep() +multi_view_policy_exp() +single_view_policy_exp() +action_camera_rotation_sweep() +normalized_action_policy_exp() +wrist_view_policy_exp() +normalized_action_joint_exp() +action_design_ablation() +action_design_ablation_8b() +action_policy_sft_8b_experiments() diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/pusht_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/pusht_experiment.py new file mode 100644 index 0000000..f9fe0b7 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/pusht_experiment.py @@ -0,0 +1,86 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# PushT experiment — Cosmos3 2B pretrained and 8B GA midtrain bases +# +# CUDA_VISIBLE_DEVICES=0 PYTHONPATH=. torchrun --nproc_per_node=1 \ +# --master_port=12341 cosmos_framework/scripts/train.py \ +# --config=cosmos_framework/configs/base/config.py \ +# -- experiment=pusht_exp + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.experiment.action._experiment_helpers import register_modes +from cosmos_framework.configs.base.experiment.action.midtrain_ga_config.cosmos3_8B import make_8b_experiment +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_2b import make_2b_experiment +from cosmos_framework.data.vfm.action.pusht_dataset import PushTDataset +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry + +cs = ConfigStore.instance() + +TRAINING_ITERATIONS = 4_000 +DATALOADER_SEED = 0 +MAX_SAMPLES_PER_BATCH = 256 + +# --------------------------------------------------------------------------- +# Dataset definition +# --------------------------------------------------------------------------- +PUSHT_TRAIN_DATASET = [ + L(dataset_entry)( + name="pusht", + dataset=L(PushTDataset)( + repo_id="lerobot/pusht_image", + split="train", + split_seed=DATALOADER_SEED, + split_val_ratio=0.05, + mode="forward_dynamics", + ), + ratio=1.0, + ), +] + +# --------------------------------------------------------------------------- +# Base experiment — 2B, 4k iters +# --------------------------------------------------------------------------- +pusht_exp = make_2b_experiment( + exp_name="pusht_exp", + datasets=PUSHT_TRAIN_DATASET, + training_iterations=TRAINING_ITERATIONS, +) + +# Checkpoint save interval +pusht_exp["checkpoint"]["save_iter"] = 1000 +pusht_exp["dataloader_train"]["max_sequence_length"] = None +pusht_exp["dataloader_train"]["max_samples_per_batch"] = MAX_SAMPLES_PER_BATCH + +cs.store( + group="experiment", + package="_global_", + name="pusht_exp", + node=pusht_exp, +) +register_modes(cs, "pusht_exp", pusht_exp, dataloader_key="action_data") + +# --------------------------------------------------------------------------- +# Base experiment — 8B GA midtrain, 4k iters +# --------------------------------------------------------------------------- +pusht_8b_exp_fd = make_8b_experiment( + exp_name="pusht_8b_exp", + datasets=PUSHT_TRAIN_DATASET, + training_iterations=TRAINING_ITERATIONS, + batch_size=32, + num_workers=16, + max_samples_per_batch=MAX_SAMPLES_PER_BATCH, +) + +# Checkpoint save interval +pusht_8b_exp_fd["checkpoint"]["save_iter"] = 1000 + +cs.store( + group="experiment", + package="_global_", + name="pusht_8b_exp_fd", + node=pusht_8b_exp_fd, +) +register_modes(cs, "pusht_8b_exp_fd", pusht_8b_exp_fd, dataloader_key="action_data") diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/robomind_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/robomind_experiment.py new file mode 100644 index 0000000..35458e9 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/robomind_experiment.py @@ -0,0 +1,82 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# RoboMIND experiment — Cosmos3 2B pretrained base +# +# Base experiment (policy mode) + mode variants (fd, id, policy, i2v, joint). +# Plus embodiment-type variants and keep-aspect-ratio variants. + + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.experiment.action._experiment_helpers import ( + register_embodiment_type, + register_modes, +) +from cosmos_framework.configs.base.experiment.action.midtrain_config.action_datasets import ( + DATASET_ROBOMIND_FRANKA_480, + DATASET_ROBOMIND_FRANKA_DUAL_480, +) +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_2b import make_2b_experiment +from cosmos_framework.data.vfm.action.robomind_ur_dataset import RoboMINDURDataset +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry + +cs = ConfigStore.instance() + +# --------------------------------------------------------------------------- +# Dataset definition (shared across all modes) +# --------------------------------------------------------------------------- +ROBOMIND_DATASET = [ + L(dataset_entry)( + name="robomind", + dataset=L(RoboMINDURDataset)(chunk_length=16, split="train"), + ratio=1.0, + ), +] + +ROBOMIND_FRANKA_DATASET = [ + DATASET_ROBOMIND_FRANKA_480, +] + +ROBOMIND_FRANKA_DUAL_DATASET = [ + DATASET_ROBOMIND_FRANKA_DUAL_480, +] + +# --------------------------------------------------------------------------- +# Base experiment — 2B, 4k iters (policy mode default) +# --------------------------------------------------------------------------- +robomind = make_2b_experiment( + exp_name="robomind", + datasets=ROBOMIND_DATASET, + training_iterations=4_000, +) +robomind["job"]["group"] = "robomind" + +cs.store("robomind", robomind, group="experiment", package="_global_") +register_modes(cs, "robomind", robomind, dataloader_key="action_data") + + +# --------------------------------------------------------------------------- +# Embodiment-type variants +# --------------------------------------------------------------------------- +robomind_franka = make_2b_experiment( + exp_name="robomind_franka", + datasets=ROBOMIND_FRANKA_DATASET, + training_iterations=4_000, +) +robomind_franka["job"]["group"] = "robomind" +cs.store("robomind_franka", robomind_franka, group="experiment", package="_global_") +register_modes(cs, "robomind_franka", robomind_franka, dataloader_key="action_data") + +robomind_franka_dual = make_2b_experiment( + exp_name="robomind_franka_dual", + datasets=ROBOMIND_FRANKA_DUAL_DATASET, + training_iterations=4_000, +) +robomind_franka_dual["job"]["group"] = "robomind" +cs.store("robomind_franka_dual", robomind_franka_dual, group="experiment", package="_global_") +register_modes(cs, "robomind_franka_dual", robomind_franka_dual, dataloader_key="action_data") + +robomind_ur = register_embodiment_type(cs, "robomind", "ur", "robomind-ur", robomind, dataloader_key="action_data") +register_modes(cs, "robomind_ur", robomind_ur, dataloader_key="action_data") diff --git a/cosmos_framework/configs/base/experiment/action/posttrain_config/umi_experiment.py b/cosmos_framework/configs/base/experiment/action/posttrain_config/umi_experiment.py new file mode 100644 index 0000000..ff0ef5b --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/posttrain_config/umi_experiment.py @@ -0,0 +1,272 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# UMI experiment — 8B multires pretrained base (modality-offset mRoPE) +# +# Uses make_8b_experiment() with multires overrides matching the +# _make_ablation_experiment pattern from mixed_res_dataset_ablations.py. +# +# Experiments: +# joint_multires_8b_exp_umi4tasks_mrope — base (25K iters) +# joint_multires_8b_exp_umi4tasks_mrope_overfit4K — 4K iter overfit +# ..._overfit4K_for_eval — 4K + val dataloader +# joint_multires_8b_exp_umi4tasks_add3drope_overfit4K — 3D RoPE, 4K +# ..._add3drope_overfit4K_for_eval — 3D RoPE + val + +import copy + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_8b import make_8b_experiment +from cosmos_framework.data.vfm.action.dataloaders import InfiniteDataLoader +from cosmos_framework.data.vfm.action.umi_dataset import get_umi_dataset +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry, wrap_dataset +from cosmos_framework.data.vfm.joint_dataloader import IterativeJointDataLoader + +cs = ConfigStore.instance() + +# --------------------------------------------------------------------------- +# Constants +# --------------------------------------------------------------------------- +CHECKPOINT_MULTIRES_302_003 = ( + "cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/" + "t2w_mot_exp302_003_qwen3_vl_8b_multires_modality_offset/" + "checkpoints/iter_000006750/" +) + +UMI_DATASET_NAMES = ( + "umi/cup_arrangement_0," + "data_scaling_law/towel_folding_0," + "data_scaling_law/mouse_arrangement_0," + "data_scaling_law/water_pouring_1" +) + +# --------------------------------------------------------------------------- +# Dataset definition (shared across all experiments) +# --------------------------------------------------------------------------- +DATASET_UMI4TASKS = [ + L(dataset_entry)( + name="umi", + dataset=L(get_umi_dataset)( + dataset_name=UMI_DATASET_NAMES, + dataset_type="umi_multi_task", + is_val=False, + mode="joint", + ), + ratio=1, + ), +] + + +# --------------------------------------------------------------------------- +# Multires overrides — transforms 480p single-res base into the 720p multires +# modality-offset configuration matching _make_ablation_experiment. +# --------------------------------------------------------------------------- +def _apply_multires_overrides(exp: dict) -> dict: + """Apply 720p multires modality-offset overrides to a make_8b_experiment() config.""" + # Model: 720p multires with per-resolution shift + exp["model"]["config"]["resolution"] = "720" + exp["model"]["config"]["max_num_tokens_after_packing"] = 45056 + exp["model"]["config"]["rectified_flow_training_config"]["shift"] = { + "256": 1, + "480": 2, + "720": 3, + } + + # Diffusion expert: modality margin; drop single-res rope extrapolation fields + diff_cfg = exp["model"]["config"]["diffusion_expert_config"] + diff_cfg["unified_3d_mrope_temporal_modality_margin"] = 15000 + for key in [ + "rope_h_extrapolation_ratio", + "rope_w_extrapolation_ratio", + "rope_t_extrapolation_ratio", + "max_vae_latent_side_after_patchify", + ]: + diff_cfg.pop(key, None) + + # Tokenizer: multires chunk/duration settings + tok_cfg = exp["model"]["config"]["tokenizer"] + tok_cfg["encode_chunk_frames"] = {"256": 68, "480": 24, "720": 12} + tok_cfg["encode_exact_durations"] = [17, 73] + + # Optimizer: lower LR for multires + exp["optimizer"]["lr"] = 1e-4 + + # Scheduler: no warmup + exp["scheduler"]["warm_up_steps"] = [0] + + # Checkpoint: exp302_003 multires pretrained, save every 1K + exp["checkpoint"]["save_iter"] = 1000 + exp["checkpoint"]["load_path"] = CHECKPOINT_MULTIRES_302_003 + + # Trainer: tokenizer compile warmup for 3 resolutions, higher recompile limit + exp["trainer"]["callbacks"]["compile_tokenizer"] = dict( + enabled=True, + warmup_resolutions=["256", "480", "720"], + ) + exp["trainer"]["compile_config"]["recompile_limit"] = 100 + + return exp + + +# --------------------------------------------------------------------------- +# Val dataloader (used by *_for_eval variants) +# --------------------------------------------------------------------------- +def _make_val_dataloader() -> L: + """Build the UMI val dataloader matching the original eval config.""" + return L(IterativeJointDataLoader)( + dataloaders={ + "umi_data": dict( + dataloader=L(InfiniteDataLoader)( + dataset=L(wrap_dataset)( + list_of_datasets=[ + L(dataset_entry)( + name="umi", + dataset=L(get_umi_dataset)( + dataset_name=UMI_DATASET_NAMES, + dataset_type="umi_multi_task", + is_val=True, + mode="joint", + ), + ratio=1.0, + ), + ], + tokenizer_config="${model.config.vlm_config.tokenizer}", + cfg_dropout_rate=0.0, + max_action_dim="${model.config.max_action_dim}", + ), + batch_size=2, + num_workers=0, + pin_memory=True, + drop_last=True, + seed=0, + ), + ratio=1, + ), + }, + tokenizer_spatial_compression_factor="${model.config.tokenizer.spatial_compression_factor}", + tokenizer_temporal_compression_factor="${model.config.tokenizer.temporal_compression_factor}", + patch_spatial="${model.config.diffusion_expert_config.patch_spatial}", + max_sequence_length="${model.config.max_num_tokens_after_packing}", + ) + + +# --------------------------------------------------------------------------- +# Base experiment — multires 8B UMI 4 tasks (mRoPE), 25K iters +# --------------------------------------------------------------------------- +joint_multires_8b_exp_umi4tasks_mrope = _apply_multires_overrides( + make_8b_experiment( + exp_name="joint_multires_8b_exp_umi4tasks_mrope", + datasets=copy.deepcopy(DATASET_UMI4TASKS), + ), +) + +cs.store( + group="experiment", + package="_global_", + name="joint_multires_8b_exp_umi4tasks_mrope", + node=joint_multires_8b_exp_umi4tasks_mrope, +) + + +# --------------------------------------------------------------------------- +# Overfit 4K — mRoPE +# --------------------------------------------------------------------------- +joint_multires_8b_exp_umi4tasks_mrope_overfit4K = dict( + defaults=[ + "/experiment/joint_multires_8b_exp_umi4tasks_mrope", + "_self_", + ], + scheduler=dict( + cycle_lengths=[4000], + ), + trainer=dict( + max_iter=4000, + ), +) + +cs.store( + group="experiment", + package="_global_", + name="joint_multires_8b_exp_umi4tasks_mrope_overfit4K", + node=joint_multires_8b_exp_umi4tasks_mrope_overfit4K, +) + + +# --------------------------------------------------------------------------- +# Overfit 4K for eval — mRoPE + val dataloader +# --------------------------------------------------------------------------- +joint_multires_8b_exp_umi4tasks_mrope_overfit4K_for_eval = dict( + defaults=[ + "/experiment/joint_multires_8b_exp_umi4tasks_mrope_overfit4K", + "_self_", + ], + dataloader_val=_make_val_dataloader(), +) + +cs.store( + group="experiment", + package="_global_", + name="joint_multires_8b_exp_umi4tasks_mrope_overfit4K_for_eval", + node=joint_multires_8b_exp_umi4tasks_mrope_overfit4K_for_eval, +) + + +# --------------------------------------------------------------------------- +# Overfit 4K — additive 3D RoPE (switches position embedding from mRoPE) +# --------------------------------------------------------------------------- +joint_multires_8b_exp_umi4tasks_add3drope_overfit4K = dict( + defaults=[ + "/experiment/joint_multires_8b_exp_umi4tasks_mrope", + "_self_", + ], + scheduler=dict( + cycle_lengths=[4000], + ), + trainer=dict( + max_iter=4000, + ), + model=dict( + config=dict( + diffusion_expert_config=dict( + position_embedding_type="3d_rope", + ), + ), + ), + checkpoint=dict( + keys_to_skip_loading=[ + "action2llm", + "llm2action", + "action_modality_embed", + "action_pos_embed", + "latent_pos_embed", + ], + ), +) + +cs.store( + group="experiment", + package="_global_", + name="joint_multires_8b_exp_umi4tasks_add3drope_overfit4K", + node=joint_multires_8b_exp_umi4tasks_add3drope_overfit4K, +) + + +# --------------------------------------------------------------------------- +# Overfit 4K for eval — additive 3D RoPE + val dataloader +# --------------------------------------------------------------------------- +joint_multires_8b_exp_umi4tasks_add3drope_overfit4K_for_eval = dict( + defaults=[ + "/experiment/joint_multires_8b_exp_umi4tasks_add3drope_overfit4K", + "_self_", + ], + dataloader_val=_make_val_dataloader(), +) + +cs.store( + group="experiment", + package="_global_", + name="joint_multires_8b_exp_umi4tasks_add3drope_overfit4K_for_eval", + node=joint_multires_8b_exp_umi4tasks_add3drope_overfit4K_for_eval, +) diff --git a/cosmos_framework/configs/base/experiment/action/pretrained_config/__init__.py b/cosmos_framework/configs/base/experiment/action/pretrained_config/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/pretrained_config/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/configs/base/experiment/action/pretrained_config/cosmos3_2b.py b/cosmos_framework/configs/base/experiment/action/pretrained_config/cosmos3_2b.py new file mode 100644 index 0000000..f3566ee --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/pretrained_config/cosmos3_2b.py @@ -0,0 +1,222 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Centralized Cosmos3 VFM 2B pretrained model config for Action experiments. + +Provides make_2b_experiment(), a factory that builds complete experiment configs +from the 2B mrope pretrained base (480p single-res). + +Usage — downstream experiment files just pass datasets:: + + from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_2b import ( + make_2b_experiment, + ) + + exp = make_2b_experiment("my_exp", [DATASET_BRIDGE, DATASET_UMI]) + cs.store(group="experiment", package="_global_", name="my_exp", node=exp) + + # Override specific fields after creation: + exp["scheduler"]["cycle_lengths"] = [4000] + exp["trainer"]["max_iter"] = 4000 + +The raw ``cosmos3_pretrained_2b`` dict is also registered in ConfigStore for +Hydra defaults inheritance (``defaults=["/experiment/cosmos3_pretrained_2b", "_self_"]``). +""" + +import copy + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.dataloaders import InfiniteDataLoader +from cosmos_framework.data.vfm.action.unified_dataset import wrap_dataset +from cosmos_framework.data.vfm.joint_dataloader import IterativeJointDataLoader + +PRETRAINED_CHECKPOINT = ( + "cosmos3_vfm/t2w_mot_2b_qwen3_vl_runs/" + "t2w_mot_dryrun_exp202_001_qwen3_vl_2b_480res_qwen3_captions_v4_300_frames_mrope/" + "checkpoints/iter_000026000/" +) + + +_DEFAULT_KEYS_TO_SKIP = ["action2llm", "llm2action", "action_modality_embed", "action_pos_embed"] + + +def make_2b_experiment( + exp_name: str, + datasets: list, + *, + batch_size: int = 4, + num_workers: int = 16, + training_iterations: int = 25_000, + use_deterministic_seed: bool = False, + action_data_ratio: int = 1, + max_action_dim: int = 64, + action_param_lr_multipliers: float = 5.0, + keys_to_skip_loading: list[str] = _DEFAULT_KEYS_TO_SKIP, +) -> dict: + """Build a Cosmos3 2B Action experiment config with the given dataset subset.""" + if keys_to_skip_loading: + log.warning( + f"keys_to_skip_loading={keys_to_skip_loading} — action layers will NOT be loaded from the " + "checkpoint. Set keys_to_skip_loading=[] when resuming from an action " + "checkpoint to avoid dropping trained action layers." + ) + return dict( + defaults=[ + {"override /data_train": None}, + {"override /data_val": None}, + {"override /model": "mot_fsdp"}, + {"override /optimizer": "fusedadamw"}, + {"override /scheduler": "lambdalinear"}, + {"override /tokenizer": "wan2pt2_tokenizer"}, + {"override /cluster": "gcp_iad_gb200"}, + {"override /vlm_config": "qwen3_vl_mot_vlm_2b_instruct_gcp"}, + {"override /checkpoint": "gcp"}, + { + "override /callbacks": [ + "basic", + "optimization", + "job_monitor", + # "training_stats", + ] + }, + "_self_", + ], + job=dict( + project="cosmos3_action", + group="cosmos3_action_2b_pretrained", + name=exp_name, + ), + model=dict( + config=dict( + action_gen=True, + max_action_dim=max_action_dim, + joint_attn_implementation="two_way", + resolution="480", + state_ch=48, + latent_downsample_factor=16, + state_t=300, + max_num_tokens_after_packing=30000, + rectified_flow_training_config=dict( + train_time_weight="uniform", + loss_scale=10.0, + use_discrete_rf=False, + shift=2, + ), + diffusion_expert_config=dict( + patch_spatial=2, + position_embedding_type="unified_3d_mrope", + unified_3d_mrope_reset_spatial_ids=True, + rope_h_extrapolation_ratio=2.0, + rope_w_extrapolation_ratio=2.0, + rope_t_extrapolation_ratio=1.0, + max_vae_latent_side_after_patchify=52, + enable_fps_modulation=True, + base_fps=24, + ), + tokenizer=dict( + bucket_name="${job.cluster.object_store_bucket_pretrained}", + object_store_credential_path_pretrained="${job.cluster.object_store_credential_pretrained}", + encode_chunk_frames={"256": 68, "480": 24, "720": 12}, + encode_exact_durations=[17, 61, 73, 93], + ), + parallelism=dict( + use_torch_compile=True, + data_parallel_shard_degree=8, + ), + vlm_config=dict( + use_system_prompt=False, + pretrained_weights=dict(enabled=True), + ), + ), + ), + optimizer=dict( + lr=2e-4, + betas=[0.9, 0.99], + weight_decay=0.05, + keys_to_select=[ + "moe_gen", + "time_embedder", + "vae2llm", + "llm2vae", + "action2llm", + "llm2action", + "action_modality_embed", + ], + lr_multipliers={ + "action2llm": action_param_lr_multipliers, + "llm2action": action_param_lr_multipliers, + "action_modality_embed": action_param_lr_multipliers, + }, + ), + scheduler=dict( + f_max=[0.5], + f_min=[0.2], + warm_up_steps=[2_000], + cycle_lengths=[training_iterations], + ), + trainer=dict( + max_iter=training_iterations, + logging_iter=200, + callbacks=dict( + grad_clip=dict(clip_norm=1.0), + manual_gc=dict(every_n=200), + compile_tokenizer=dict(enabled=True, warmup_resolutions=["256", "480", "720"]), + straggler_detection=dict(enabled=True, report_freq=50), + ), + compile_config=dict(recompile_limit=32, use_duck_shape=False), + ), + checkpoint=dict( + save_iter=500, + load_path=PRETRAINED_CHECKPOINT, + load_training_state=False, + strict_resume=True, + keys_to_skip_loading=keys_to_skip_loading, + ), + dataloader_train=L(IterativeJointDataLoader)( + dataloaders={ + "action_data": dict( + dataloader=L(InfiniteDataLoader)( + dataset=L(wrap_dataset)( + list_of_datasets=copy.deepcopy(datasets), + tokenizer_config="${model.config.vlm_config.tokenizer}", + cfg_dropout_rate=0.1, + max_action_dim="${model.config.max_action_dim}", + shard_across_workers=True, + append_duration_fps_timestamps=True, + append_resolution_info=True, + ), + batch_size=batch_size, + num_workers=num_workers, + pin_memory=True, + use_deterministic_seed=use_deterministic_seed, + in_order=False, + multiprocessing_context="spawn", + ), + ratio=action_data_ratio, + ), + }, + tokenizer_spatial_compression_factor="${model.config.tokenizer.spatial_compression_factor}", + tokenizer_temporal_compression_factor="${model.config.tokenizer.temporal_compression_factor}", + patch_spatial="${model.config.diffusion_expert_config.patch_spatial}", + max_sequence_length="${model.config.max_num_tokens_after_packing}", + ), + ) + + +# --------------------------------------------------------------------------- +# Hydra defaults inheritance config generated from make_2b_experiment(). +# --------------------------------------------------------------------------- +cosmos3_pretrained_2b = make_2b_experiment("cosmos3_pretrained_2b", datasets=[]) +cosmos3_pretrained_2b["job"]["group"] = "pretrained_config" +cosmos3_pretrained_2b["job"]["name"] = "cosmos3_pretrained_2b" + +cs = ConfigStore.instance() +cs.store( + group="experiment", + package="_global_", + name="cosmos3_pretrained_2b", + node=cosmos3_pretrained_2b, +) diff --git a/cosmos_framework/configs/base/experiment/action/pretrained_config/cosmos3_32b.py b/cosmos_framework/configs/base/experiment/action/pretrained_config/cosmos3_32b.py new file mode 100644 index 0000000..af62f68 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/pretrained_config/cosmos3_32b.py @@ -0,0 +1,200 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import copy + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.dataloaders import InfiniteDataLoader +from cosmos_framework.data.vfm.action.unified_dataset import wrap_dataset +from cosmos_framework.data.vfm.joint_dataloader import IterativeJointDataLoader + +PRETRAINED_CHECKPOINT = ( + "cosmos3_vfm/t2w_mot_32b_qwen3_vl_runs/t2w_mot_exp305_000_qwen3_vl_32b_multires_v7/checkpoints/iter_000085000/" +) + + +_DEFAULT_KEYS_TO_SKIP = ["action2llm", "llm2action", "action_modality_embed", "action_pos_embed"] + + +def make_32b_experiment( + exp_name: str, + datasets: list, + *, + batch_size: int = 4, + num_workers: int = 16, + training_iterations: int = 25_000, + use_deterministic_seed: bool = False, + action_data_ratio: int = 1, + max_action_dim: int = 64, + action_param_lr_multipliers: float = 5.0, + extra_dataloaders: dict[str, dict] | None = None, + keys_to_skip_loading: list[str] = _DEFAULT_KEYS_TO_SKIP, +) -> dict: + """ """ + if keys_to_skip_loading: + log.warning( + f"keys_to_skip_loading={keys_to_skip_loading} — action layers will NOT be loaded from the " + "checkpoint. Set keys_to_skip_loading=[] when resuming from an action " + "checkpoint to avoid dropping trained action layers." + ) + + dataloaders: dict[str, dict] = {} + if extra_dataloaders is not None: + dataloaders.update(extra_dataloaders) + dataloaders["action_data"] = dict( + dataloader=L(InfiniteDataLoader)( + dataset=L(wrap_dataset)( + list_of_datasets=copy.deepcopy(datasets), + tokenizer_config="${model.config.vlm_config.tokenizer}", + cfg_dropout_rate=0.1, + max_action_dim="${model.config.max_action_dim}", + shard_across_workers=True, + append_duration_fps_timestamps=True, + append_resolution_info=True, + ), + batch_size=batch_size, + num_workers=num_workers, + pin_memory=True, + use_deterministic_seed=use_deterministic_seed, + in_order=False, + multiprocessing_context="spawn", + ), + ratio=action_data_ratio, + ) + + return dict( + defaults=[ + {"override /data_train": None}, + {"override /data_val": None}, + {"override /model": "mot_fsdp"}, + {"override /optimizer": "fusedadamw"}, + {"override /scheduler": "lambdalinear"}, + {"override /tokenizer": "wan2pt2_tokenizer"}, + {"override /cluster": "gcp_iad_gb200"}, + {"override /vlm_config": "qwen3_vl_mot_vlm_32b_instruct_gcp"}, + {"override /checkpoint": "gcp"}, + { + "override /callbacks": [ + "basic", + "optimization", + "job_monitor", + "training_stats", + ] + }, + "_self_", + ], + job=dict( + project="cosmos3_action", + group="cosmos3_action_32b_pretrained", + name=exp_name, + ), + model=dict( + config=dict( + action_gen=True, + max_action_dim=max_action_dim, + joint_attn_implementation="two_way", + resolution="720", + state_ch=48, + latent_downsample_factor=16, + state_t=300, + max_num_tokens_after_packing=45056, + rectified_flow_training_config=dict( + train_time_weight="uniform", + train_time_video_distribution="waver", + loss_scale=10.0, + use_discrete_rf=False, + shift={"256": 1, "480": 2, "720": 3}, + ), + diffusion_expert_config=dict( + patch_spatial=2, + position_embedding_type="unified_3d_mrope", + unified_3d_mrope_reset_spatial_ids=True, + unified_3d_mrope_temporal_modality_margin=15000, + enable_fps_modulation=True, + base_fps=24, + ), + tokenizer=dict( + bucket_name="${job.cluster.object_store_bucket_pretrained}", + object_store_credential_path_pretrained="${job.cluster.object_store_credential_pretrained}", + encode_chunk_frames={"256": 68, "480": 24, "720": 12}, + encode_exact_durations=[17, 61, 73], + ), + parallelism=dict( + data_parallel_shard_degree=64, + use_torch_compile=True, + ), + vlm_config=dict( + use_system_prompt=False, + pretrained_weights=dict(enabled=True), + ), + ), + ), + optimizer=dict( + lr=1e-4, + betas=[0.9, 0.99], + weight_decay=0.05, + keys_to_select=[ + "moe_gen", + "time_embedder", + "vae2llm", + "llm2vae", + "action2llm", + "llm2action", + "action_modality_embed", + ], + lr_multipliers={ + "action2llm": action_param_lr_multipliers, + "llm2action": action_param_lr_multipliers, + "action_modality_embed": action_param_lr_multipliers, + }, + ), + scheduler=dict( + f_max=[0.95], + f_min=[0.3], + warm_up_steps=[0], + cycle_lengths=[training_iterations], + ), + trainer=dict( + max_iter=training_iterations, + logging_iter=50, + callbacks=dict( + grad_clip=dict(clip_norm=1.0), + manual_gc=dict(every_n=200), + compile_tokenizer=dict(enabled=True), + straggler_detection=dict(enabled=True, report_freq=50), + norm_monitor=dict(every_n=100), + sigma_loss_analysis=dict(every_n=500, every_n_viz=500), + ), + compile_config=dict(recompile_limit=100, use_duck_shape=False), + ), + checkpoint=dict( + save_iter=500, + load_path=PRETRAINED_CHECKPOINT, + load_training_state=False, + strict_resume=True, + keys_to_skip_loading=keys_to_skip_loading, + ), + dataloader_train=L(IterativeJointDataLoader)( + dataloaders=dataloaders, + tokenizer_spatial_compression_factor="${model.config.tokenizer.spatial_compression_factor}", + tokenizer_temporal_compression_factor="${model.config.tokenizer.temporal_compression_factor}", + patch_spatial="${model.config.diffusion_expert_config.patch_spatial}", + max_sequence_length="${model.config.max_num_tokens_after_packing}", + ), + ) + + +cosmos3_pretrained_32b = make_32b_experiment("cosmos3_pretrained_32b", datasets=[]) +cosmos3_pretrained_32b["job"]["group"] = "pretrained_config" +cosmos3_pretrained_32b["job"]["name"] = "cosmos3_pretrained_32b" + +cs = ConfigStore.instance() +cs.store( + group="experiment", + package="_global_", + name="cosmos3_pretrained_32b", + node=cosmos3_pretrained_32b, +) diff --git a/cosmos_framework/configs/base/experiment/action/pretrained_config/cosmos3_8b.py b/cosmos_framework/configs/base/experiment/action/pretrained_config/cosmos3_8b.py new file mode 100644 index 0000000..ec60928 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/action/pretrained_config/cosmos3_8b.py @@ -0,0 +1,237 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Centralized Cosmos3 VFM 8B pretrained model config for Action experiments. + +Provides make_8b_experiment(), a factory that builds complete experiment configs +from the 8B multires recipe_v7 pretrained base (originally 720p, fine-tuned at 480p). + +Usage — downstream experiment files just pass datasets:: + + from cosmos_framework.configs.base.experiment.action.pretrained_config.cosmos3_8b import ( + make_8b_experiment, + ) + + exp = make_8b_experiment("my_exp", [DATASET_BRIDGE, DATASET_UMI]) + cs.store(group="experiment", package="_global_", name="my_exp", node=exp) + + # Override specific fields after creation: + exp["scheduler"]["cycle_lengths"] = [4000] + exp["trainer"]["max_iter"] = 4000 + +The raw ``cosmos3_pretrained_8b`` dict is also registered in ConfigStore for +Hydra defaults inheritance (``defaults=["/experiment/cosmos3_pretrained_8b", "_self_"]``). +""" + +import copy + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.dataloaders import InfiniteDataLoader +from cosmos_framework.data.vfm.action.unified_dataset import wrap_dataset +from cosmos_framework.data.vfm.joint_dataloader import IterativeJointDataLoader + +PRETRAINED_CHECKPOINT = ( + # "cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/t2w_mot_exp302_000_qwen3_vl_8b_multires_recipe_v7/checkpoints/iter_000043500/" + "cosmos3_vfm/t2w_mot_8b_qwen3_vl_runs/t2w_mot_exp406_000_qwen3_vl_8b_multires_recipe_midtraining_v5/checkpoints/iter_000035000/" # SOT52 +) + + +_DEFAULT_KEYS_TO_SKIP = ["action2llm", "llm2action", "action_modality_embed", "action_pos_embed"] + + +def make_8b_experiment( + exp_name: str, + datasets: list, + *, + batch_size: int = 4, + num_workers: int = 16, + training_iterations: int = 25_000, + use_deterministic_seed: bool = False, + action_data_ratio: int = 1, + max_action_dim: int = 64, + action_param_lr_multipliers: float = 5.0, + extra_dataloaders: dict[str, dict] | None = None, + keys_to_skip_loading: list[str] = _DEFAULT_KEYS_TO_SKIP, +) -> dict: + """Build a Cosmos3 8B Action experiment config with the given dataset subset. + + Args: + extra_dataloaders: Additional dataloader entries (e.g. VFM image/video + dataloaders) to include alongside ``action_data`` in the + ``IterativeJointDataLoader``. Keys are dataloader names and values + are ``dict(dataloader=..., ratio=...)`` entries. + keys_to_skip_loading: Checkpoint keys to skip when loading. Defaults to + action-specific layers. Pass ``[]`` when resuming from an action + checkpoint to load all weights. + """ + if keys_to_skip_loading: + log.warning( + f"keys_to_skip_loading={keys_to_skip_loading} — action layers will NOT be loaded from the " + "checkpoint. Set keys_to_skip_loading=[] when resuming from an action " + "checkpoint to avoid dropping trained action layers." + ) + + dataloaders: dict[str, dict] = {} + if extra_dataloaders is not None: + dataloaders.update(extra_dataloaders) + dataloaders["action_data"] = dict( + dataloader=L(InfiniteDataLoader)( + dataset=L(wrap_dataset)( + list_of_datasets=copy.deepcopy(datasets), + tokenizer_config="${model.config.vlm_config.tokenizer}", + cfg_dropout_rate=0.1, + max_action_dim="${model.config.max_action_dim}", + shard_across_workers=True, + append_duration_fps_timestamps=True, + append_resolution_info=True, + ), + batch_size=batch_size, + num_workers=num_workers, + pin_memory=True, + use_deterministic_seed=use_deterministic_seed, + in_order=False, + multiprocessing_context="spawn", + ), + ratio=action_data_ratio, + ) + + return dict( + defaults=[ + {"override /data_train": None}, + {"override /data_val": None}, + {"override /model": "mot_fsdp"}, + {"override /optimizer": "fusedadamw"}, + {"override /scheduler": "lambdalinear"}, + {"override /tokenizer": "wan2pt2_tokenizer"}, + {"override /cluster": "gcp_iad_gb200"}, + {"override /vlm_config": "qwen3_vl_mot_vlm_8b_instruct_gcp"}, + {"override /checkpoint": "gcp"}, + { + "override /callbacks": [ + "basic", + "optimization", + "job_monitor", + "training_stats", + ] + }, + "_self_", + ], + job=dict( + project="cosmos3_action", + group="cosmos3_action_8b_pretrained", + name=exp_name, + ), + model=dict( + config=dict( + action_gen=True, + max_action_dim=max_action_dim, + joint_attn_implementation="two_way", + resolution="720", + state_ch=48, + latent_downsample_factor=16, + state_t=300, + max_num_tokens_after_packing=45056, + rectified_flow_training_config=dict( + train_time_weight="uniform", + train_time_video_distribution="waver", + loss_scale=10.0, + use_discrete_rf=False, + shift={"256": 3, "480": 5, "720": 10}, + ), + diffusion_expert_config=dict( + patch_spatial=2, + position_embedding_type="unified_3d_mrope", + unified_3d_mrope_reset_spatial_ids=True, + unified_3d_mrope_temporal_modality_margin=15000, + enable_fps_modulation=True, + base_fps=24, + ), + tokenizer=dict( + bucket_name="${job.cluster.object_store_bucket_pretrained}", + object_store_credential_path_pretrained="${job.cluster.object_store_credential_pretrained}", + encode_chunk_frames={"256": 68, "480": 24, "720": 12}, + encode_exact_durations=[17, 61, 73], + ), + parallelism=dict( + data_parallel_shard_degree=8, + use_torch_compile=True, + ), + vlm_config=dict( + use_system_prompt=False, + pretrained_weights=dict(enabled=True), + ), + ), + ), + optimizer=dict( + lr=1e-4, + betas=[0.9, 0.99], + weight_decay=0.05, + keys_to_select=[ + "moe_gen", + "time_embedder", + "vae2llm", + "llm2vae", + "action2llm", + "llm2action", + "action_modality_embed", + ], + lr_multipliers={ + "action2llm": action_param_lr_multipliers, + "llm2action": action_param_lr_multipliers, + "action_modality_embed": action_param_lr_multipliers, + }, + ), + scheduler=dict( + f_max=[0.4], + f_min=[0.0], + warm_up_steps=[0], + cycle_lengths=[training_iterations], + ), + trainer=dict( + max_iter=training_iterations, + logging_iter=50, + callbacks=dict( + grad_clip=dict(clip_norm=1.0), + manual_gc=dict(every_n=200), + compile_tokenizer=dict(enabled=True, warmup_resolutions=["256", "480", "720"]), + straggler_detection=dict(enabled=True, report_freq=50), + + norm_monitor=dict(every_n=100), + sigma_loss_analysis=dict(every_n=500, every_n_viz=500), + ), + compile_config=dict(recompile_limit=100, use_duck_shape=False), + ), + checkpoint=dict( + save_iter=500, + load_path=PRETRAINED_CHECKPOINT, + load_training_state=False, + strict_resume=True, + keys_to_skip_loading=keys_to_skip_loading, + ), + dataloader_train=L(IterativeJointDataLoader)( + dataloaders=dataloaders, + tokenizer_spatial_compression_factor="${model.config.tokenizer.spatial_compression_factor}", + tokenizer_temporal_compression_factor="${model.config.tokenizer.temporal_compression_factor}", + patch_spatial="${model.config.diffusion_expert_config.patch_spatial}", + max_sequence_length="${model.config.max_num_tokens_after_packing}", + ), + ) + + +# --------------------------------------------------------------------------- +# Hydra defaults inheritance config generated from make_8b_experiment(). +# --------------------------------------------------------------------------- +cosmos3_pretrained_8b = make_8b_experiment("cosmos3_pretrained_8b", datasets=[]) +cosmos3_pretrained_8b["job"]["group"] = "pretrained_config" +cosmos3_pretrained_8b["job"]["name"] = "cosmos3_pretrained_8b" + +cs = ConfigStore.instance() +cs.store( + group="experiment", + package="_global_", + name="cosmos3_pretrained_8b", + node=cosmos3_pretrained_8b, +) diff --git a/cosmos_framework/configs/base/experiment/posttrain_video/__init__.py b/cosmos_framework/configs/base/experiment/posttrain_video/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/configs/base/experiment/posttrain_video/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/configs/base/experiment/sft/__init__.py b/cosmos_framework/configs/base/experiment/sft/__init__.py new file mode 100644 index 0000000..28a81be --- /dev/null +++ b/cosmos_framework/configs/base/experiment/sft/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 diff --git a/cosmos_framework/configs/base/experiment/sft/action_fdm_sft_nano.py b/cosmos_framework/configs/base/experiment/sft/action_fdm_sft_nano.py new file mode 100644 index 0000000..40aa5e3 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/sft/action_fdm_sft_nano.py @@ -0,0 +1,269 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Action forward-dynamics SFT on Bridge LeRobot — "nano" variant. + +Dataset root is taken from ``$DATASET_PATH``; set it before launching:: + + export DATASET_PATH=/path/to/bridge_orig_lerobot + +Usage:: + + CUDA_VISIBLE_DEVICES=0 PYTHONPATH=. torchrun --nproc_per_node=1 \\ + --master_port=12341 -m cosmos_framework.scripts.train \\ + --config=configs/base/config.py -- \\ + experiment=action_fdm_sft_nano \\ + checkpoint.load_path= +""" + +import copy + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils.lazy_config import LazyDict + +from cosmos_framework.configs.base.experiment.sft.models.nano_model_config import NANO_MODEL_CONFIG +from cosmos_framework.data.vfm.action.bridge_orig_lerobot_dataset import BridgeOrigLeRobotDataset +from cosmos_framework.data.vfm.action.dataloaders import InfiniteDataLoader +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry, wrap_dataset +from cosmos_framework.data.vfm.joint_dataloader import IterativeJointDataLoader + +cs = ConfigStore.instance() + + +def _action_fdm_nano_model_config() -> dict: + cfg = copy.deepcopy(NANO_MODEL_CONFIG) + cfg["max_num_tokens_after_packing"] = -1 + cfg["diffusion_expert_config"]["load_weights_from_pretrained"] = False + cfg["diffusion_expert_config"]["max_vae_latent_side_after_patchify"] = 52 + cfg["rectified_flow_training_config"]["loss_scale"] = 10.0 + cfg["rectified_flow_training_config"]["image_loss_scale"] = None + cfg["tokenizer"]["encode_exact_durations"] = [17] + return cfg + + +_BRIDGE_DATASETS = [ + L(dataset_entry)( + name="bridge", + dataset=L(BridgeOrigLeRobotDataset)( + action_normalization="quantile", + chunk_length=16, + enable_fast_init=False, + fps=5.0, + mode="forward_dynamics", + pose_convention="backward_framewise", + root="${oc.env:DATASET_PATH}", + split="train", + split_seed=42, + split_val_ratio=0.02, + viewpoint="ego_view", + ), + ratio=1.0, + resolution="480", + ), +] + + +action_fdm_sft_nano = LazyDict( + dict( + defaults=[ + {"override /model": "mot_fsdp"}, + {"override /data_train": None}, + {"override /data_val": None}, + {"override /optimizer": "fusedadamw"}, + {"override /scheduler": "lambdalinear"}, + {"override /checkpoint": "s3"}, + { + "override /callbacks": [ + "basic", + "optimization", + "job_monitor", + "generation", + ] + }, + {"override /ema": "power"}, + {"override /tokenizer": "wan2pt2_tokenizer"}, + {"override /sound_tokenizer": None}, + {"override /cluster": None}, + {"override /vlm_config": None}, + {"override /ckpt_type": "dcp"}, + "_self_", + ], + job=dict( + project="cosmos3_action_bridge", + group="action_bridge", + name="action_fdm_sft_nano", + wandb_mode="offline", + ), + model=dict( + config=_action_fdm_nano_model_config(), + ), + optimizer=dict( + betas=[0.9, 0.99], + eps=1.0e-08, + fused=True, + keys_to_select=[ + "moe_gen", + "time_embedder", + "vae2llm", + "llm2vae", + "action2llm", + "llm2action", + "action_modality_embed", + ], + lr=2.0e-04, + lr_multipliers={ + "action2llm": 5.0, + "action_modality_embed": 5.0, + "llm2action": 5.0, + }, + weight_decay=0.05, + ), + scheduler=dict( + cycle_lengths=[20000], + f_max=[0.4], + f_min=[0.05], + f_start=[1.0e-06], + verbosity_interval=0, + warm_up_steps=[100], + ), + trainer=dict( + distributed_parallelism="fsdp", + grad_accum_iter=1, + logging_iter=50, + max_iter=16000, + run_validation=False, + run_validation_on_start=False, + save_zero_checkpoint=False, + seed=0, + timeout_period=999999999, + validation_iter=100, + compile_config=dict(recompile_limit=100, use_duck_shape=False), + cudnn=dict(benchmark=True, deterministic=False), + ddp=dict(broadcast_buffers=True, find_unused_parameters=False, static_graph=True), + grad_scaler_args=dict(enabled=False), + callbacks=dict( + compile_tokenizer=dict( + compile_after_iterations=3, + enabled=True, + warmup_resolutions=["480"], + ), + grad_clip=dict(clip_norm=1.0, force_finite=True), + heart_beat=dict(every_n=200, save_s3=False, step_size=1, update_interval_in_minute=20), + iter_speed=dict(every_n=50, hit_thres=50, save_s3=False, save_s3_every_log_n=500), + manual_gc=dict(every_n=200, gc_level=1, warm_up=5), + mfu=dict( + backwardpass_ratio=2.0, + every_n=50, + grad_accum_iter=2, + hit_thres=5, + include_padding=True, + include_vae_encoder=True, + ), + moe_specialization=dict(every_n=250), + moe_stability=dict(every_n=250), + norm_monitor=dict( + every_n=100, + layer_norm_only=False, + log_stat_wandb=True, + save_s3=False, + step_size=1, + track_activations=True, + ), + ofu=dict(every_n=50, hit_thres=5), + param_count=dict(save_s3=False), + sequence_packing_padding=dict(every_n=50), + sigma_loss_analysis=dict(every_n=500, every_n_viz=500, save_s3=False), + skip_nan_step=dict(max_consecutive_nan=100), + straggler_detection=dict(enabled=True, report_freq=50), + training_stats=dict(log_freq=100), + wandb_2x=dict( + logging_iter_multipler=2, + save_logging_iter_multipler=1, + save_s3=False, + ), + ), + ), + checkpoint=dict( + broadcast_via_filesystem=True, + dcp_async_mode_enabled=True, + enable_gcs_patch_in_boto3=False, + keys_to_skip_loading=[ + "net_ema.", + # "action2llm", + # "llm2action", + # "action_modality_embed", + # "action_pos_embed", + ], + load_ema_to_reg=False, + load_path="???", # OmegaConf MISSING — must be set via override at launch + load_training_state=False, + only_load_scheduler_state=False, + save_iter=100, + strict_resume=True, + verbose=True, + load_from_object_store=dict( + bucket="", + credentials="", + enabled=False, + ), + save_to_object_store=dict( + bucket="", + credentials="", + enabled=False, + ), + ), + dataloader_train=L(IterativeJointDataLoader)( + audio_sample_rate=48000, + max_samples_per_batch=32, + max_sequence_length=None, + patch_spatial=2, + seed=42, + sound_latent_fps=0, + tokenizer_spatial_compression_factor=16, + tokenizer_temporal_compression_factor=4, + dataloaders=dict( + action_data=dict( + ratio=1, + dataloader=L(InfiniteDataLoader)( + batch_size=4, + in_order=False, + multiprocessing_context="spawn", + num_workers=4, + pin_memory=True, + seed=42, + use_deterministic_seed=True, + dataset=L(wrap_dataset)( + action_channel_masking=True, + append_duration_fps_timestamps=True, + append_idle_frames=False, + append_resolution_info=True, + caption_key="ai_caption", + cfg_dropout_rate=0.1, + format_prompt_as_json=False, + idle_frames_dropout=0.05, + keep_aspect_ratio=True, + list_of_datasets=_BRIDGE_DATASETS, + max_action_dim=64, + pad_keys=None, + resolution=None, + shard_across_workers=True, + text_token_key="text_token_ids", + video_temporal_downsample=4, + tokenizer_config="${model.config.vlm_config.tokenizer}", + ), + ), + ), + ), + ), + upload_reproducible_setup=False, + ), + flags={"allow_objects": True}, +) + + +for _item in [action_fdm_sft_nano]: + _name = [k for k, v in globals().items() if v is _item][0] + _item["job"]["name"] = _name + "_${now:%Y-%m-%d}_${now:%H-%M-%S}" + cs.store(group="experiment", package="_global_", name=_name, node=_item) diff --git a/cosmos_framework/configs/base/experiment/sft/action_fdm_sft_nano_datapacker.py b/cosmos_framework/configs/base/experiment/sft/action_fdm_sft_nano_datapacker.py new file mode 100644 index 0000000..34afcab --- /dev/null +++ b/cosmos_framework/configs/base/experiment/sft/action_fdm_sft_nano_datapacker.py @@ -0,0 +1,167 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Action FDM SFT on Bridge LeRobot — DataPackerDataLoader variant. + +Equivalent to ``action_fdm_sft_nano`` but replaces the +IterativeJointDataLoader + InfiniteDataLoader stack with DataPackerDataLoader ++ BridgeFDMDataPacker. Sample processing is identical (same +ActionTransformPipeline params); batching changes from fixed batch_size=4 to +token-budget packing capped at max_batch_size=64. + +Usage:: + + DATASET_PATH=/path/to/bridge_orig_lerobot torchrun \\ + --nproc_per_node=4 --master_port=12341 -m cosmos_framework.scripts.train \\ + --config=configs/base/config.py -- \\ + experiment=action_fdm_sft_nano_datapacker \\ + checkpoint.load_path= +""" + +from __future__ import annotations + +import copy +import math + +import torch +from hydra.core.config_store import ConfigStore + +from cosmos_framework.data.vfm.action.bridge_orig_lerobot_dataset import BridgeOrigLeRobotDataset +from cosmos_framework.data.vfm.action.transforms import ActionTransformPipeline +from cosmos_framework.data.vfm.action.unified_dataset import MapToIterableAdapter +from cosmos_framework.data.vfm.data_packer import DataPacker +from cosmos_framework.data.vfm.data_packer_dataloader import DataPackerDataLoader +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.experiment.sft.action_fdm_sft_nano import action_fdm_sft_nano + +cs = ConfigStore.instance() + + +# --------------------------------------------------------------------------- +# Dataset factory +# --------------------------------------------------------------------------- + + +def _get_bridge_iterable_dataset(**kwargs) -> MapToIterableAdapter: + """Wrap BridgeOrigLeRobotDataset in MapToIterableAdapter for DataPackerDataLoader. + + BridgeOrigLeRobotDataset uses deferred shard registration: _register_sources() + must be called before the dataset yields any items. wrap_dataset() does this via + ActionUnifiedIterableDataset.assign_worker; here we call it explicitly. + """ + ds = BridgeOrigLeRobotDataset(**kwargs) + ds._register_sources() + return MapToIterableAdapter(ds) + + +# --------------------------------------------------------------------------- +# DataPacker +# --------------------------------------------------------------------------- + + +class BridgeFDMDataPacker(DataPacker): + """DataPacker adapter for BridgeOrigLeRobotDataset + ActionTransformPipeline (FDM). + + Applies the same ActionTransformPipeline as the wrap_dataset call in + action_fdm_sft_nano. The sft_collate_fn output format matches what + OmniMoTModel.training_step expects (same as ActionDataPacker in the + libero_policy_datapacker_experiment). + """ + + def __init__( + self, + tokenizer_spatial_compression_factor: int = 16, + tokenizer_temporal_compression_factor: int = 4, + patch_spatial: int = 2, + tokenizer_config=None, + cfg_dropout_rate: float = 0.1, + max_action_dim: int = 64, + action_channel_masking: bool = True, + ) -> None: + self._spatial = tokenizer_spatial_compression_factor + self._temporal = tokenizer_temporal_compression_factor + self._patch = patch_spatial + self._transform = ActionTransformPipeline( + pad_keys=["video"], + tokenizer_config=tokenizer_config, + cfg_dropout_rate=cfg_dropout_rate, + max_action_dim=max_action_dim, + action_channel_masking=action_channel_masking, + append_duration_fps_timestamps=True, + append_resolution_info=True, + ) + + def sft_process_sample(self, item: dict) -> dict: + return self._transform(item, resolution=None) + + def compute_num_tokens(self, sample: dict) -> int: + tokens = 1 + len(sample.get("text_token_ids", [])) + v = sample.get("video") + if v is not None: + _, T, H, W = v.shape + latent_h = math.ceil(H / (self._spatial * self._patch)) + latent_w = math.ceil(W / (self._spatial * self._patch)) + latent_t = 1 + (T - 1) // self._temporal + tokens += latent_h * latent_w * latent_t + 2 + return tokens + + def sft_collate_fn(self, samples: list, max_len: int, ignore_label_id: int = -100) -> dict: + return { + "text_token_ids": [[s["text_token_ids"]] for s in samples], + "video": [s.get("video") for s in samples], + "action": [s.get("action") for s in samples], + "padding_mask": [s.get("padding_mask") for s in samples], + "image_size": [s.get("image_size") for s in samples], + "fps": torch.tensor([float(s.get("fps", 0.0)) for s in samples]), + "domain_id": [s.get("domain_id") for s in samples], + "sequence_plan": [s.get("sequence_plan") for s in samples], + "raw_action_dim": [s.get("raw_action_dim") for s in samples], + "ai_caption": [s.get("ai_caption", "") for s in samples], + } + + +# --------------------------------------------------------------------------- +# Experiment registration +# --------------------------------------------------------------------------- + +_exp = copy.deepcopy(action_fdm_sft_nano) +_exp["dataloader_train"] = L(DataPackerDataLoader)( + data_source=L(_get_bridge_iterable_dataset)( + action_normalization="quantile", + chunk_length=16, + enable_fast_init=False, + fps=5.0, + mode="forward_dynamics", + pose_convention="backward_framewise", + root="${oc.env:DATASET_PATH}", + split="train", + split_seed=42, + split_val_ratio=0.02, + viewpoint="ego_view", + ), + data_packer=L(BridgeFDMDataPacker)( + tokenizer_spatial_compression_factor="${model.config.tokenizer.spatial_compression_factor}", + tokenizer_temporal_compression_factor="${model.config.tokenizer.temporal_compression_factor}", + patch_spatial="${model.config.diffusion_expert_config.patch_spatial}", + tokenizer_config="${model.config.vlm_config.tokenizer}", + cfg_dropout_rate=0.1, + max_action_dim="${model.config.max_action_dim}", + action_channel_masking=True, + ), + max_tokens=999_999, + max_batch_size=64, + pool_size=16, + num_workers=4, + prefetch_factor=4, + persistent_workers=True, + pin_memory=True, +) +_exp["job"]["name"] = "action_fdm_sft_nano_datapacker_${now:%Y-%m-%d}_${now:%H-%M-%S}" + +# Smoke-test overrides: skip the S3 VLM backbone download and use the HF +# tokenizer variant so the run is self-contained (no object-store credentials). +_exp["model"]["config"]["tokenizer"]["bucket_name"] = "" +_exp["model"]["config"]["vlm_config"]["pretrained_weights"]["enabled"] = False +_exp["model"]["config"]["vlm_config"]["tokenizer"]["config_variant"] = "hf" + +cs.store(group="experiment", package="_global_", name="action_fdm_sft_nano_datapacker", node=_exp) diff --git a/cosmos_framework/configs/base/experiment/sft/action_fdm_sft_super.py b/cosmos_framework/configs/base/experiment/sft/action_fdm_sft_super.py new file mode 100644 index 0000000..db1ffcb --- /dev/null +++ b/cosmos_framework/configs/base/experiment/sft/action_fdm_sft_super.py @@ -0,0 +1,279 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Action forward-dynamics SFT on Bridge LeRobot — "super" variant. + +LoRA fine-tune on top of the Qwen3-VL-32B-Instruct backbone (CP=2 / DP=4 with +activation checkpointing). Only the LoRA adapters on the gen MoE attention +projections train; the rest of the backbone stays frozen. + +Dataset root is taken from ``$DATASET_PATH`` (Bridge LeRobot root, same as the +nano FDM variant). ``checkpoint.load_path`` is a required override (Hydra +``???`` placeholder); supply it on the CLI or in a downstream experiment. + +Usage:: + + CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 PYTHONPATH=. torchrun --nproc_per_node=8 \\ + --master_port=12341 -m cosmos_framework.scripts.train \\ + --config=configs/base/config.py -- \\ + experiment=action_fdm_sft_super checkpoint.load_path= +""" + +import copy + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils.lazy_config import LazyDict + +from cosmos_framework.configs.base.experiment.sft.models.super_model_config import SUPER_MODEL_CONFIG +from cosmos_framework.data.vfm.action.bridge_orig_lerobot_dataset import BridgeOrigLeRobotDataset +from cosmos_framework.data.vfm.action.dataloaders import InfiniteDataLoader +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry, wrap_dataset +from cosmos_framework.data.vfm.joint_dataloader import IterativeJointDataLoader + +cs = ConfigStore.instance() + + +def _action_fdm_super_model_config() -> dict: + """SUPER_MODEL_CONFIG baseline + action-FDM-specific overrides. + + Mirrors the action_fdm_sft_nano pattern (action_gen on, FDM-style loss + scales, encode_exact_durations=[17]) on top of the 32B / LoRA / DP=4 / + CP=2 super baseline. + """ + cfg = copy.deepcopy(SUPER_MODEL_CONFIG) + cfg["action_gen"] = True + cfg["max_action_dim"] = 64 + cfg["max_num_tokens_after_packing"] = -1 + cfg["diffusion_expert_config"]["load_weights_from_pretrained"] = False + cfg["diffusion_expert_config"]["max_vae_latent_side_after_patchify"] = 52 + cfg["parallelism"]["use_torch_compile"] = True + cfg["rectified_flow_training_config"]["loss_scale"] = 10.0 + cfg["rectified_flow_training_config"]["image_loss_scale"] = None + cfg["tokenizer"]["encode_exact_durations"] = [17] + return cfg + + +_BRIDGE_DATASETS = [ + L(dataset_entry)( + name="bridge", + dataset=L(BridgeOrigLeRobotDataset)( + action_normalization="quantile", + chunk_length=16, + enable_fast_init=False, + fps=5.0, + mode="forward_dynamics", + pose_convention="backward_framewise", + root="${oc.env:DATASET_PATH}", + split="train", + split_seed=42, + split_val_ratio=0.02, + viewpoint="ego_view", + ), + ratio=1.0, + resolution="480", + ), +] + + +action_fdm_sft_super = LazyDict( + dict( + defaults=[ + {"override /model": "mot_fsdp"}, + {"override /data_train": None}, + {"override /data_val": None}, + {"override /optimizer": "fusedadamw"}, + {"override /scheduler": "lambdalinear"}, + {"override /checkpoint": "s3"}, + { + "override /callbacks": [ + "basic", + "optimization", + "job_monitor", + "generation", + ] + }, + {"override /ema": "power"}, + {"override /tokenizer": "wan2pt2_tokenizer"}, + {"override /sound_tokenizer": None}, + {"override /cluster": None}, + {"override /vlm_config": None}, + {"override /ckpt_type": "dcp"}, + "_self_", + ], + job=dict( + project="cosmos3_action_bridge", + group="action_bridge", + name="action_fdm_sft_super", + wandb_mode="offline", + ), + model=dict( + config=_action_fdm_super_model_config(), + ), + optimizer=dict( + betas=[0.9, 0.99], + eps=1.0e-08, + fused=True, + keys_to_select=["lora_"], + lr=2.0e-04, + lr_multipliers={}, + weight_decay=0.05, + ), + scheduler=dict( + cycle_lengths=[20000], + f_max=[0.4], + f_min=[0.05], + f_start=[1.0e-06], + verbosity_interval=0, + warm_up_steps=[100], + ), + trainer=dict( + distributed_parallelism="fsdp", + grad_accum_iter=1, + logging_iter=50, + max_iter=16000, + run_validation=False, + run_validation_on_start=False, + save_zero_checkpoint=False, + seed=0, + timeout_period=999999999, + validation_iter=100, + compile_config=dict(recompile_limit=100, use_duck_shape=False), + cudnn=dict(benchmark=True, deterministic=False), + ddp=dict(broadcast_buffers=True, find_unused_parameters=False, static_graph=True), + grad_scaler_args=dict(enabled=False), + callbacks=dict( + compile_tokenizer=dict( + compile_after_iterations=3, + enabled=True, + warmup_resolutions=["480"], + ), + dataloader_speed=dict(every_n=100, save_s3=False, step_size=1), + grad_clip=dict(clip_norm=1.0, force_finite=True), + heart_beat=dict(every_n=200, save_s3=False, step_size=1, update_interval_in_minute=20), + iter_speed=dict(every_n=50, hit_thres=50, save_s3=False, save_s3_every_log_n=500), + manual_gc=dict(every_n=200, gc_level=1, warm_up=5), + mfu=dict( + backwardpass_ratio=2.0, + every_n=50, + grad_accum_iter=2, + hit_thres=5, + include_padding=True, + include_vae_encoder=True, + ), + moe_specialization=dict(every_n=250), + moe_stability=dict(every_n=250), + norm_monitor=dict( + every_n=100, + layer_norm_only=False, + log_stat_wandb=True, + save_s3=False, + step_size=1, + track_activations=True, + ), + ofu=dict(every_n=50, hit_thres=5), + param_count=dict(save_s3=False), + sequence_packing_padding=dict(every_n=50), + sigma_loss_analysis=dict(every_n=500, every_n_viz=500, save_s3=False), + skip_nan_step=dict(max_consecutive_nan=100), + straggler_detection=dict(enabled=True, report_freq=50), + training_stats=dict(log_freq=100), + wandb_2x=dict( + logging_iter_multipler=2, + save_logging_iter_multipler=1, + save_s3=False, + ), + expert_heatmap=dict(every_n=1000), + device_monitor=dict( + every_n=200, + log_memory_detail=True, + save_s3=False, + step_size=1, + upload_every_n_mul=5, + ), + ), + ), + checkpoint=dict( + broadcast_via_filesystem=True, + dcp_async_mode_enabled=True, + enable_gcs_patch_in_boto3=True, + keys_to_skip_loading=[ + "net_ema.", + # "action2llm", + # "llm2action", + # "action_modality_embed", + # "action_pos_embed", + "lora_", + ], + load_ema_to_reg=False, + load_path="???", # OmegaConf MISSING — must be set via override at launch + load_training_state=False, + only_load_scheduler_state=False, + save_iter=100, + strict_resume=False, + verbose=True, + load_from_object_store=dict( + bucket="", + credentials="", + enabled=False, + ), + save_to_object_store=dict( + bucket="", + credentials="", + enabled=False, + ), + ), + dataloader_train=L(IterativeJointDataLoader)( + audio_sample_rate=48000, + max_samples_per_batch=64, + max_sequence_length=None, + patch_spatial=2, + seed=42, + sound_latent_fps=0, + tokenizer_spatial_compression_factor=16, + tokenizer_temporal_compression_factor=4, + dataloaders=dict( + action_data=dict( + ratio=1, + dataloader=L(InfiniteDataLoader)( + batch_size=4, + in_order=False, + multiprocessing_context="spawn", + num_workers=4, + pin_memory=True, + seed=42, + use_deterministic_seed=True, + dataset=L(wrap_dataset)( + action_channel_masking=True, + append_duration_fps_timestamps=True, + append_idle_frames=False, + append_resolution_info=True, + caption_key="ai_caption", + cfg_dropout_rate=0.1, + format_prompt_as_json=False, + idle_frames_dropout=0.05, + keep_aspect_ratio=True, + list_of_datasets=_BRIDGE_DATASETS, + max_action_dim=64, + pad_keys=None, + resolution=None, + shard_across_workers=True, + text_token_key="text_token_ids", + video_temporal_downsample=4, + tokenizer_config="${model.config.vlm_config.tokenizer}", + ), + ), + ), + ), + ), + upload_reproducible_setup=False, + ), + flags={"allow_objects": True}, +) + + +for _item in [action_fdm_sft_super]: + _name = [k for k, v in globals().items() if v is _item][0] + _item["job"]["name"] = _name + "_${now:%Y-%m-%d}_${now:%H-%M-%S}" + cs.store(group="experiment", package="_global_", name=_name, node=_item) diff --git a/cosmos_framework/configs/base/experiment/sft/action_policy_sft_nano.py b/cosmos_framework/configs/base/experiment/sft/action_policy_sft_nano.py new file mode 100644 index 0000000..e661e5a --- /dev/null +++ b/cosmos_framework/configs/base/experiment/sft/action_policy_sft_nano.py @@ -0,0 +1,303 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Action policy SFT on the LIBERO suite — "nano" variant. + +Dataset roots are taken from ``$DATASET_PATH``; export it before launching so +that the four LIBERO suites (``libero_10`` / ``libero_object`` / +``libero_spatial`` / ``libero_goal``) resolve under their corresponding +subdirectories. + +``checkpoint.load_path`` is a required override (Hydra ``???`` placeholder); +supply it on the CLI or in a downstream experiment that inherits this one. + +Usage:: + + CUDA_VISIBLE_DEVICES=0 PYTHONPATH=. torchrun --nproc_per_node=1 \\ + --master_port=12341 -m cosmos_framework.scripts.train \\ + --config=configs/base/config.py -- \\ + experiment=action_policy_sft_nano checkpoint.load_path= +""" + +import copy + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils.lazy_config import LazyDict + +from cosmos_framework.configs.base.experiment.sft.models.nano_model_config import NANO_MODEL_CONFIG +from cosmos_framework.data.vfm.action.dataloaders import InfiniteDataLoader +from cosmos_framework.data.vfm.action.libero_dataset import LIBERODataset +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry, wrap_dataset +from cosmos_framework.data.vfm.joint_dataloader import IterativeJointDataLoader + +cs = ConfigStore.instance() + + +def _action_policy_nano_model_config() -> dict: + cfg = copy.deepcopy(NANO_MODEL_CONFIG) + cfg["log_enc_time_every_n"] = 50 + cfg["diffusion_expert_config"]["load_weights_from_pretrained"] = False + cfg["rectified_flow_training_config"]["loss_scale"] = 10.0 + cfg["rectified_flow_training_config"]["image_loss_scale"] = None + cfg["tokenizer"]["encode_exact_durations"] = [17, 61, 73] + return cfg + + +_LIBERO_DATASETS = [ + L(dataset_entry)( + name="libero", + dataset=L(LIBERODataset)( + action_normalization="quantile_rot", + action_space="frame_wise_relative", + action_stats_path=( + "cosmos_framework/data/vfm/action/normalizers/" + "libero_native_frame_wise_relative_rot6d.json" + ), + camera_mode="concat_view", + chunk_length=16, + download_videos=False, + embodiment_type="libero", + force_cache_sync=False, + fps=20, + image_size=256, + mode="policy", + pose_coordinate_frame="native", + repo_id=[ + "libero_10", + "libero_object", + "libero_spatial", + "libero_goal", + ], + root=[ + "${oc.env:DATASET_PATH}/libero_10", + "${oc.env:DATASET_PATH}/libero_object", + "${oc.env:DATASET_PATH}/libero_spatial", + "${oc.env:DATASET_PATH}/libero_goal", + ], + rotation_space="6d", + seed=0, + skip_video_loading=False, + split="train", + tolerance_s=0.0001, + val_ratio=0.01, + video_backend="torchcodec", + ), + ratio=1.0, + resolution=None, + ), +] + + +action_policy_sft_nano = LazyDict( + dict( + defaults=[ + {"override /model": "mot_fsdp"}, + {"override /data_train": None}, + {"override /data_val": None}, + {"override /optimizer": "fusedadamw"}, + {"override /scheduler": "lambdalinear"}, + {"override /checkpoint": "s3"}, + { + "override /callbacks": [ + "basic", + "optimization", + "job_monitor", + "generation", + ] + }, + {"override /ema": "power"}, + {"override /tokenizer": "wan2pt2_tokenizer"}, + {"override /sound_tokenizer": None}, + {"override /cluster": None}, + {"override /vlm_config": None}, + {"override /ckpt_type": "dcp"}, + "_self_", + ], + job=dict( + project="cosmos3_action_libero", + group="action_libero", + name="action_policy_sft_nano", + wandb_mode="offline", + ), + model=dict( + config=_action_policy_nano_model_config(), + ), + optimizer=dict( + betas=[0.9, 0.99], + eps=1.0e-08, + fused=True, + keys_to_select=[ + "moe_gen", + "time_embedder", + "vae2llm", + "llm2vae", + "action2llm", + "llm2action", + "action_modality_embed", + ], + lr=5.0e-05, + lr_multipliers={ + "action2llm": 5.0, + "action_modality_embed": 5.0, + "llm2action": 5.0, + }, + weight_decay=0.05, + ), + scheduler=dict( + cycle_lengths=[20000], + f_max=[1.0], + f_min=[0.0], + f_start=[1.0e-06], + verbosity_interval=0, + warm_up_steps=[500], + ), + trainer=dict( + distributed_parallelism="fsdp", + grad_accum_iter=1, + logging_iter=100, + max_iter=16000, + run_validation=False, + run_validation_on_start=False, + save_zero_checkpoint=False, + seed=0, + timeout_period=999999999, + validation_iter=100, + compile_config=dict(recompile_limit=100, use_duck_shape=False), + cudnn=dict(benchmark=True, deterministic=False), + ddp=dict(broadcast_buffers=True, find_unused_parameters=False, static_graph=True), + grad_scaler_args=dict(enabled=False), + callbacks=dict( + compile_tokenizer=dict( + compile_after_iterations=3, + enabled=True, + warmup_resolutions=["256", "480", "720"], + ), + dataloader_speed=dict(every_n=50, save_s3=False, step_size=1), + grad_clip=dict(clip_norm=1.0, force_finite=True), + heart_beat=dict(every_n=200, save_s3=False, step_size=1, update_interval_in_minute=20), + iter_speed=dict(every_n=50, hit_thres=50, save_s3=False, save_s3_every_log_n=500), + manual_gc=dict(every_n=200, gc_level=1, warm_up=5), + mfu=dict( + backwardpass_ratio=2.0, + every_n=50, + grad_accum_iter=2, + hit_thres=5, + include_padding=True, + include_vae_encoder=True, + ), + moe_specialization=dict(every_n=250), + moe_stability=dict(every_n=250), + norm_monitor=dict( + every_n=50, + layer_norm_only=False, + log_stat_wandb=True, + save_s3=False, + step_size=1, + track_activations=True, + ), + ofu=dict(every_n=50, hit_thres=5), + param_count=dict(save_s3=False), + sequence_packing_padding=dict(every_n=50), + sigma_loss_analysis=dict(every_n=500, every_n_viz=500, save_s3=False), + skip_nan_step=dict(max_consecutive_nan=100), + straggler_detection=dict(enabled=True, report_freq=50), + training_stats=dict(log_freq=100), + wandb_2x=dict( + logging_iter_multipler=2, + save_logging_iter_multipler=1, + save_s3=False, + ), + expert_heatmap=dict(every_n=500), + device_monitor=dict( + every_n=200, + log_memory_detail=True, + save_s3=False, + step_size=1, + upload_every_n_mul=5, + ), + ), + ), + checkpoint=dict( + broadcast_via_filesystem=False, + dcp_async_mode_enabled=True, + enable_gcs_patch_in_boto3=True, + keys_to_skip_loading=[ + "net_ema.", + # "action2llm", + # "llm2action", + # "action_modality_embed", + # "action_pos_embed", + ], + load_ema_to_reg=False, + load_path="???", # OmegaConf MISSING — must be set via override at launch + load_training_state=False, + only_load_scheduler_state=False, + save_iter=100, + strict_resume=True, + verbose=True, + load_from_object_store=dict( + bucket="", + credentials="", + enabled=False, + ), + save_to_object_store=dict( + bucket="", + credentials="", + enabled=False, + ), + ), + dataloader_train=L(IterativeJointDataLoader)( + audio_sample_rate=48000, + max_samples_per_batch=256, + max_sequence_length=None, + patch_spatial=2, + seed=42, + sound_latent_fps=0, + tokenizer_spatial_compression_factor=16, + tokenizer_temporal_compression_factor=4, + dataloaders=dict( + action_data=dict( + ratio=1, + dataloader=L(InfiniteDataLoader)( + batch_size=128, + in_order=False, + multiprocessing_context="spawn", + num_workers=4, + pin_memory=True, + seed=42, + use_deterministic_seed=True, + dataset=L(wrap_dataset)( + action_channel_masking=True, + append_duration_fps_timestamps=True, + append_idle_frames=False, + append_resolution_info=True, + caption_key="ai_caption", + cfg_dropout_rate=0.1, + format_prompt_as_json=False, + idle_frames_dropout=0.05, + keep_aspect_ratio=True, + list_of_datasets=_LIBERO_DATASETS, + max_action_dim=64, + pad_keys=None, + resolution=None, + shard_across_workers=True, + text_token_key="text_token_ids", + video_temporal_downsample=4, + tokenizer_config="${model.config.vlm_config.tokenizer}", + ), + ), + ), + ), + ), + upload_reproducible_setup=False, + ), + flags={"allow_objects": True}, +) + + +for _item in [action_policy_sft_nano]: + _name = [k for k, v in globals().items() if v is _item][0] + _item["job"]["name"] = _name + "_${now:%Y-%m-%d}_${now:%H-%M-%S}" + cs.store(group="experiment", package="_global_", name=_name, node=_item) diff --git a/cosmos_framework/configs/base/experiment/sft/action_policy_sft_nano_datapacker.py b/cosmos_framework/configs/base/experiment/sft/action_policy_sft_nano_datapacker.py new file mode 100644 index 0000000..d9590fb --- /dev/null +++ b/cosmos_framework/configs/base/experiment/sft/action_policy_sft_nano_datapacker.py @@ -0,0 +1,187 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Action policy SFT on LIBERO — DataPackerDataLoader variant. + +Equivalent to ``action_policy_sft_nano`` but replaces the +IterativeJointDataLoader + InfiniteDataLoader stack with DataPackerDataLoader ++ ActionDataPacker. Sample processing is identical (same +ActionTransformPipeline params); batching changes from fixed batch_size=128 to +token-budget packing capped at max_batch_size=256. + +Usage:: + + DATASET_PATH=/path/to/libero_datasets torchrun \\ + --nproc_per_node=4 --master_port=12341 -m cosmos_framework.scripts.train \\ + --config=configs/base/config.py -- \\ + experiment=action_policy_sft_nano_datapacker \\ + checkpoint.load_path= +""" + +from __future__ import annotations + +import copy +import math + +import torch +from hydra.core.config_store import ConfigStore + +from cosmos_framework.data.vfm.action.libero_dataset import LIBERODataset +from cosmos_framework.data.vfm.action.transforms import ActionTransformPipeline +from cosmos_framework.data.vfm.action.unified_dataset import MapToIterableAdapter +from cosmos_framework.data.vfm.data_packer import DataPacker +from cosmos_framework.data.vfm.data_packer_dataloader import DataPackerDataLoader +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.experiment.sft.action_policy_sft_nano import action_policy_sft_nano + +cs = ConfigStore.instance() + + +# --------------------------------------------------------------------------- +# Dataset factory +# --------------------------------------------------------------------------- + + +def _get_libero_iterable_dataset(**kwargs) -> MapToIterableAdapter: + """Wrap LIBERODataset in MapToIterableAdapter for DataPackerDataLoader. + + LIBERODataset is map-style; MapToIterableAdapter yields uniformly random + items indefinitely so _IterableWrapper inside DataPackerDataLoader can + apply DP sharding on top. + """ + return MapToIterableAdapter(LIBERODataset(**kwargs)) + + +# --------------------------------------------------------------------------- +# DataPacker +# --------------------------------------------------------------------------- + + +class ActionDataPacker(DataPacker): + """DataPacker adapter for LIBERODataset + ActionTransformPipeline (policy mode). + + Applies the same ActionTransformPipeline as the wrap_dataset call in + action_policy_sft_nano. The sft_collate_fn output format matches what + OmniMoTModel.training_step expects. + """ + + def __init__( + self, + tokenizer_spatial_compression_factor: int = 16, + tokenizer_temporal_compression_factor: int = 4, + patch_spatial: int = 2, + tokenizer_config=None, + cfg_dropout_rate: float = 0.1, + max_action_dim: int = 64, + action_channel_masking: bool = True, + ) -> None: + self._spatial = tokenizer_spatial_compression_factor + self._temporal = tokenizer_temporal_compression_factor + self._patch = patch_spatial + self._transform = ActionTransformPipeline( + pad_keys=["video"], + tokenizer_config=tokenizer_config, + cfg_dropout_rate=cfg_dropout_rate, + max_action_dim=max_action_dim, + action_channel_masking=action_channel_masking, + append_duration_fps_timestamps=True, + append_resolution_info=True, + ) + + def sft_process_sample(self, item: dict) -> dict: + return self._transform(item, resolution=None) + + def compute_num_tokens(self, sample: dict) -> int: + tokens = 1 + len(sample.get("text_token_ids", [])) + v = sample.get("video") + if v is not None: + _, T, H, W = v.shape + latent_h = math.ceil(H / (self._spatial * self._patch)) + latent_w = math.ceil(W / (self._spatial * self._patch)) + latent_t = 1 + (T - 1) // self._temporal + tokens += latent_h * latent_w * latent_t + 2 + return tokens + + def sft_collate_fn(self, samples: list, max_len: int, ignore_label_id: int = -100) -> dict: + return { + "text_token_ids": [[s["text_token_ids"]] for s in samples], + "video": [s.get("video") for s in samples], + "action": [s.get("action") for s in samples], + "padding_mask": [s.get("padding_mask") for s in samples], + "image_size": [s.get("image_size") for s in samples], + "fps": torch.tensor([float(s.get("fps", 0.0)) for s in samples]), + "domain_id": [s.get("domain_id") for s in samples], + "sequence_plan": [s.get("sequence_plan") for s in samples], + "raw_action_dim": [s.get("raw_action_dim") for s in samples], + "ai_caption": [s.get("ai_caption", "") for s in samples], + } + + +# --------------------------------------------------------------------------- +# Experiment registration +# --------------------------------------------------------------------------- + +_exp = copy.deepcopy(action_policy_sft_nano) +_exp["dataloader_train"] = L(DataPackerDataLoader)( + data_source=L(_get_libero_iterable_dataset)( + action_normalization="quantile_rot", + action_space="frame_wise_relative", + action_stats_path=( + "cosmos_framework/data/vfm/action/normalizers/" + "libero_native_frame_wise_relative_rot6d.json" + ), + camera_mode="concat_view", + chunk_length=16, + download_videos=False, + embodiment_type="libero", + force_cache_sync=False, + fps=20, + image_size=256, + mode="policy", + pose_coordinate_frame="native", + repo_id=[ + "libero_10", + "libero_object", + "libero_spatial", + "libero_goal", + ], + root=[ + "${oc.env:DATASET_PATH}/libero_10", + "${oc.env:DATASET_PATH}/libero_object", + "${oc.env:DATASET_PATH}/libero_spatial", + "${oc.env:DATASET_PATH}/libero_goal", + ], + rotation_space="6d", + seed=0, + skip_video_loading=False, + split="train", + tolerance_s=0.0001, + val_ratio=0.01, + video_backend="torchcodec", + ), + data_packer=L(ActionDataPacker)( + tokenizer_spatial_compression_factor="${model.config.tokenizer.spatial_compression_factor}", + tokenizer_temporal_compression_factor="${model.config.tokenizer.temporal_compression_factor}", + patch_spatial="${model.config.diffusion_expert_config.patch_spatial}", + tokenizer_config="${model.config.vlm_config.tokenizer}", + cfg_dropout_rate=0.1, + max_action_dim="${model.config.max_action_dim}", + action_channel_masking=True, + ), + max_tokens=999_999, + max_batch_size=256, + pool_size=16, + num_workers=4, + prefetch_factor=4, + persistent_workers=True, + pin_memory=True, +) +_exp["job"]["name"] = "action_policy_sft_nano_datapacker_${now:%Y-%m-%d}_${now:%H-%M-%S}" + +# Smoke-test overrides: skip the S3 VLM backbone download and use the HF +# tokenizer variant so the run is self-contained (no object-store credentials). +_exp["model"]["config"]["tokenizer"]["bucket_name"] = "" +_exp["model"]["config"]["vlm_config"]["pretrained_weights"]["enabled"] = False +_exp["model"]["config"]["vlm_config"]["tokenizer"]["config_variant"] = "hf" + +cs.store(group="experiment", package="_global_", name="action_policy_sft_nano_datapacker", node=_exp) diff --git a/cosmos_framework/configs/base/experiment/sft/action_policy_sft_super.py b/cosmos_framework/configs/base/experiment/sft/action_policy_sft_super.py new file mode 100644 index 0000000..fd7317f --- /dev/null +++ b/cosmos_framework/configs/base/experiment/sft/action_policy_sft_super.py @@ -0,0 +1,306 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Action policy SFT on the LIBERO suite — "super" variant. + +LoRA fine-tune on top of the Qwen3-VL-32B-Instruct backbone (CP=2 / DP=4 with +activation checkpointing). Only the LoRA adapters on the gen MoE attention +projections train; the rest of the backbone stays frozen. + +Dataset roots are taken from ``$DATASET_PATH``; export it before launching so +that the four LIBERO suites (``libero_10`` / ``libero_object`` / +``libero_spatial`` / ``libero_goal``) resolve under their corresponding +subdirectories. + +``checkpoint.load_path`` is a required override (Hydra ``???`` placeholder); +supply it on the CLI or in a downstream experiment that inherits this one. + +Usage:: + + CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 PYTHONPATH=. torchrun --nproc_per_node=8 \\ + --master_port=12341 -m cosmos_framework.scripts.train \\ + --config=configs/base/config.py -- \\ + experiment=action_policy_sft_super checkpoint.load_path= +""" + +import copy + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils.lazy_config import LazyDict + +from cosmos_framework.configs.base.experiment.sft.models.super_model_config import SUPER_MODEL_CONFIG +from cosmos_framework.data.vfm.action.dataloaders import InfiniteDataLoader +from cosmos_framework.data.vfm.action.libero_dataset import LIBERODataset +from cosmos_framework.data.vfm.action.unified_dataset import dataset_entry, wrap_dataset +from cosmos_framework.data.vfm.joint_dataloader import IterativeJointDataLoader + +cs = ConfigStore.instance() + + +def _action_policy_super_model_config() -> dict: + """SUPER_MODEL_CONFIG baseline + action-policy-specific overrides. + + Mirrors action_policy_sft_nano (action_gen on, encode_exact_durations= + [17, 61, 73], log_enc_time_every_n=50, policy-style loss scales) on top + of the 32B / LoRA / DP=4 / CP=2 super baseline. + """ + cfg = copy.deepcopy(SUPER_MODEL_CONFIG) + cfg["action_gen"] = True + cfg["log_enc_time_every_n"] = 50 + cfg["max_action_dim"] = 64 + cfg["max_num_tokens_after_packing"] = 16384 + cfg["diffusion_expert_config"]["load_weights_from_pretrained"] = False + cfg["parallelism"]["use_torch_compile"] = True + cfg["rectified_flow_training_config"]["loss_scale"] = 10.0 + cfg["rectified_flow_training_config"]["image_loss_scale"] = None + cfg["tokenizer"]["encode_exact_durations"] = [17, 61, 73] + return cfg + + +_LIBERO_DATASETS = [ + L(dataset_entry)( + name="libero", + dataset=L(LIBERODataset)( + action_normalization="quantile_rot", + action_space="frame_wise_relative", + action_stats_path=( + "cosmos_framework/data/vfm/action/normalizers/" + "libero_native_frame_wise_relative_rot6d.json" + ), + camera_mode="concat_view", + chunk_length=16, + download_videos=False, + embodiment_type="libero", + force_cache_sync=False, + fps=20, + image_size=256, + mode="policy", + pose_coordinate_frame="native", + repo_id=[ + "libero_10", + "libero_object", + "libero_spatial", + "libero_goal", + ], + root=[ + "${oc.env:DATASET_PATH}/libero_10", + "${oc.env:DATASET_PATH}/libero_object", + "${oc.env:DATASET_PATH}/libero_spatial", + "${oc.env:DATASET_PATH}/libero_goal", + ], + rotation_space="6d", + seed=0, + skip_video_loading=False, + split="train", + tolerance_s=0.0001, + val_ratio=0.01, + video_backend="torchcodec", + ), + ratio=1.0, + resolution=None, + ), +] + + +action_policy_sft_super = LazyDict( + dict( + defaults=[ + {"override /model": "mot_fsdp"}, + {"override /data_train": None}, + {"override /data_val": None}, + {"override /optimizer": "fusedadamw"}, + {"override /scheduler": "lambdacosine"}, + {"override /checkpoint": "s3"}, + { + "override /callbacks": [ + "basic", + "optimization", + "job_monitor", + "generation", + ] + }, + {"override /ema": "power"}, + {"override /tokenizer": "wan2pt2_tokenizer"}, + {"override /sound_tokenizer": None}, + {"override /cluster": None}, + {"override /vlm_config": None}, + {"override /ckpt_type": "dcp"}, + "_self_", + ], + job=dict( + project="cosmos3_action_libero", + group="action_libero", + name="action_policy_sft_super", + wandb_mode="offline", + ), + model=dict( + config=_action_policy_super_model_config(), + ), + optimizer=dict( + betas=[0.9, 0.99], + eps=1.0e-08, + fused=True, + keys_to_select=["lora_"], + lr=5.0e-05, + lr_multipliers={}, + weight_decay=0.05, + ), + scheduler=dict( + cycle_lengths=[20000], + f_max=[1.0], + f_min=[0.0], + f_start=[1.0e-06], + verbosity_interval=0, + warm_up_steps=[500], + ), + trainer=dict( + distributed_parallelism="fsdp", + grad_accum_iter=1, + logging_iter=100, + max_iter=16000, + run_validation=False, + run_validation_on_start=False, + save_zero_checkpoint=False, + seed=0, + timeout_period=999999999, + validation_iter=100, + compile_config=dict(recompile_limit=100, use_duck_shape=False), + cudnn=dict(benchmark=True, deterministic=False), + ddp=dict(broadcast_buffers=True, find_unused_parameters=False, static_graph=True), + grad_scaler_args=dict(enabled=False), + callbacks=dict( + compile_tokenizer=dict( + compile_after_iterations=3, + enabled=True, + warmup_resolutions=["256", "480", "720"], + ), + dataloader_speed=dict(every_n=50, save_s3=False, step_size=1), + grad_clip=dict(clip_norm=1.0, force_finite=True), + heart_beat=dict(every_n=200, save_s3=False, step_size=1, update_interval_in_minute=20), + iter_speed=dict(every_n=50, hit_thres=50, save_s3=False, save_s3_every_log_n=500), + manual_gc=dict(every_n=200, gc_level=1, warm_up=5), + mfu=dict( + backwardpass_ratio=2.0, + every_n=50, + grad_accum_iter=2, + hit_thres=5, + include_padding=True, + include_vae_encoder=True, + ), + moe_specialization=dict(every_n=250), + moe_stability=dict(every_n=250), + norm_monitor=dict( + every_n=50, + layer_norm_only=False, + log_stat_wandb=True, + save_s3=False, + step_size=1, + track_activations=True, + ), + ofu=dict(every_n=50, hit_thres=5), + param_count=dict(save_s3=False), + sequence_packing_padding=dict(every_n=50), + sigma_loss_analysis=dict(every_n=500, every_n_viz=500, save_s3=False), + skip_nan_step=dict(max_consecutive_nan=100), + straggler_detection=dict(enabled=True, report_freq=50), + training_stats=dict(log_freq=100), + wandb_2x=dict( + logging_iter_multipler=2, + save_logging_iter_multipler=1, + save_s3=False, + ), + expert_heatmap=dict(every_n=500), + device_monitor=dict( + every_n=200, + log_memory_detail=True, + save_s3=False, + step_size=1, + upload_every_n_mul=5, + ), + ), + ), + checkpoint=dict( + broadcast_via_filesystem=False, + dcp_async_mode_enabled=True, + enable_gcs_patch_in_boto3=True, + keys_to_skip_loading=[ + "net_ema.", + # "action2llm", + # "llm2action", + # "action_modality_embed", + # "action_pos_embed", + "lora_", + ], + load_ema_to_reg=False, + load_path="???", # OmegaConf MISSING — must be set via override at launch + load_training_state=False, + only_load_scheduler_state=False, + save_iter=100, + strict_resume=False, + verbose=True, + load_from_object_store=dict( + bucket="", + credentials="", + enabled=False, + ), + save_to_object_store=dict( + bucket="", + credentials="", + enabled=False, + ), + ), + dataloader_train=L(IterativeJointDataLoader)( + audio_sample_rate=48000, + max_samples_per_batch=256, + max_sequence_length=None, + patch_spatial=2, + seed=42, + sound_latent_fps=0, + tokenizer_spatial_compression_factor=16, + tokenizer_temporal_compression_factor=4, + dataloaders=dict( + action_data=dict( + ratio=1, + dataloader=L(InfiniteDataLoader)( + batch_size=4, + in_order=False, + multiprocessing_context="spawn", + num_workers=4, + pin_memory=True, + seed=42, + use_deterministic_seed=True, + dataset=L(wrap_dataset)( + action_channel_masking=True, + append_duration_fps_timestamps=True, + append_idle_frames=False, + append_resolution_info=True, + caption_key="ai_caption", + cfg_dropout_rate=0.1, + format_prompt_as_json=False, + idle_frames_dropout=0.05, + keep_aspect_ratio=True, + list_of_datasets=_LIBERO_DATASETS, + max_action_dim=64, + pad_keys=None, + resolution=None, + shard_across_workers=True, + text_token_key="text_token_ids", + video_temporal_downsample=4, + tokenizer_config="${model.config.vlm_config.tokenizer}", + ), + ), + ), + ), + ), + upload_reproducible_setup=False, + ), + flags={"allow_objects": True}, +) + + +for _item in [action_policy_sft_super]: + _name = [k for k, v in globals().items() if v is _item][0] + _item["job"]["name"] = _name + "_${now:%Y-%m-%d}_${now:%H-%M-%S}" + cs.store(group="experiment", package="_global_", name=_name, node=_item) diff --git a/cosmos_framework/configs/base/experiment/sft/models/__init__.py b/cosmos_framework/configs/base/experiment/sft/models/__init__.py new file mode 100644 index 0000000..28a81be --- /dev/null +++ b/cosmos_framework/configs/base/experiment/sft/models/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 diff --git a/cosmos_framework/configs/base/experiment/sft/models/nano_model_config.py b/cosmos_framework/configs/base/experiment/sft/models/nano_model_config.py new file mode 100644 index 0000000..9be1b7b --- /dev/null +++ b/cosmos_framework/configs/base/experiment/sft/models/nano_model_config.py @@ -0,0 +1,152 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Shared Nano-tier (Qwen3-VL-8B) ``model.config`` baseline for SFT experiments. + +Consumers must ``copy.deepcopy`` this constant before mutating it. Baseline +mirrors ``vision_sft_nano`` (HF-cluster deployment with empty tokenizer/vlm +paths, video-style loss scales, ``load_weights_from_pretrained=True``). Action +experiments override the divergent fields after deepcopy; see +``action_fdm_sft_nano`` / ``action_policy_sft_nano`` for the override list. +""" + +from cosmos_framework.utils.lazy_config import LazyCall as L + +from cosmos_framework.configs.base.defaults.vlm import ( + create_qwen2_tokenizer_with_download, + create_vlm_config, +) +from cosmos_framework.model.vfm.mot.unified_mot import Qwen3VLMoTConfig, Qwen3VLTextForCausalLM + + +NANO_MODEL_CONFIG = dict( + action_gen=True, + causal_training_strategy="none", + input_caption_key="ai_caption", + input_image_key="images", + input_video_key="video", + joint_attn_implementation="two_way", + latent_downsample_factor=16, + log_enc_time_every_n=100, + max_action_dim=64, + max_num_tokens_after_packing=45056, + num_embodiment_domains=32, + resolution="720", + sound_gen=False, + sound_latent_fps=25, + state_ch=48, + state_t=300, + video_temporal_causal=False, + vision_gen=True, + diffusion_expert_config=dict( + base_fps=24, + enable_fps_modulation=True, + load_weights_from_pretrained=True, + max_vae_latent_side_after_patchify=20, + patch_spatial=2, + position_embedding_type="unified_3d_mrope", + rope_h_extrapolation_ratio=1.0, + rope_t_extrapolation_ratio=1.0, + rope_w_extrapolation_ratio=1.0, + timestep_range=1.0, + unified_3d_mrope_reset_spatial_ids=True, + unified_3d_mrope_temporal_modality_margin=15000, + ), + ema=dict( + enabled=True, + iteration_shift=0, + rate=0.1, + ), + lbl=dict( + coeff_gen=None, + coeff_und=None, + method="local", + ), + parallelism=dict( + cfg_parallel_shard_degree=1, + compile_dynamic=True, + compiled_region="language", + context_parallel_shard_degree=1, + coordinate_descent_tuning=False, + data_parallel_shard_degree=8, + enable_inference_mode=False, + max_autotune_pointwise=False, + precision="bfloat16", + use_cuda_graphs=False, + use_torch_compile=True, + ), + activation_checkpointing=dict( + mode="full", + ), + rectified_flow_inference_config=dict( + num_train_timesteps=1000, + scheduler_type="unipc", + shift=1, + use_dynamic_shifting=False, + ), + rectified_flow_training_config=dict( + action_loss_weight=10.0, + high_sigma_ratio=0.05, + high_sigma_timesteps_max=1000, + high_sigma_timesteps_min=995, + image_loss_scale=1.0, + independent_action_schedule=False, + loss_scale=1.0, + normalize_loss_by_active=False, + shift={"256": 3, "480": 5, "720": 10}, + train_time_action_distribution="logitnormal", + train_time_image_distribution="logitnormal", + train_time_sound_distribution="logitnormal", + train_time_video_distribution="waver", + train_time_weight="uniform", + use_discrete_rf=False, + use_dynamic_shift=False, + use_high_sigma_strategy=False, + use_high_sigma_strategy_action=False, + ), + tokenizer=dict( + bucket_name="", + chunk_duration=93, + encode_chunk_frames={"256": 68, "480": 24, "720": 12}, + encode_exact_durations=None, + keep_decoder_cache=False, + object_store_credential_path_pretrained="", + spatial_compression_factor=16, + temporal_compression_factor=4, + use_streaming_encode=False, + vae_path="pretrained/tokenizers/video/wan2pt2/Wan2.2_VAE.pth", + ), + vlm_config=dict( + layer_module="Qwen2MoTDecoderLayer", + model_name="Qwen/Qwen3-VL-8B-Instruct", + tie_word_embeddings=False, + use_system_prompt=False, + pretrained_weights=dict( + enabled=False, + backbone_path=( + "s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/" + "Qwen/Qwen3-VL-8B-Instruct/" + ), + credentials_path="", + enable_gcs_patch_in_boto3=True, + ), + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file=( + "cosmos_framework/model/vfm/vlm/qwen3_vl/configs/" + "Qwen3-VL-8B-Instruct.json" + ), + ), + freeze_und=False, + layer_module="MoTDecoderLayer", + qk_norm_for_text=True, + tie_word_embeddings=True, + ), + ), + tokenizer=L(create_qwen2_tokenizer_with_download)( + config_variant="hf", + pretrained_model_name="Qwen/Qwen3-VL-8B-Instruct", + ), + ), +) diff --git a/cosmos_framework/configs/base/experiment/sft/models/super_model_config.py b/cosmos_framework/configs/base/experiment/sft/models/super_model_config.py new file mode 100644 index 0000000..ee4bca0 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/sft/models/super_model_config.py @@ -0,0 +1,170 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Shared Super-tier (Qwen3-VL-32B) ``model.config`` baseline for SFT experiments. + +Consumers must ``copy.deepcopy`` this constant before mutating it. Baseline +mirrors ``vision_sft_super`` (LoRA-only fine-tune, EMA off, no torch.compile, +DP=4 / CP=2 FSDP topology, action gen off) with the deployment-env +canonicalization applied — bucket / credentials paths empty, the VLM +tokenizer uses ``config_variant="hf"``, and ``backbone_path`` carries the +sanitized Qwen3-VL-32B URI (matching the ``NANO_MODEL_CONFIG`` policy and +keeping ``export_model --vit`` resolvable via the checkpoint catalog). + +Two differences from a literal extraction of the legacy inline block: + +- ``base_config`` uses ``Qwen3VLMoTConfig.from_json_file`` rather than the + raw HF ``Qwen3VLTextConfig.from_json_file``. The MoT wrapper exposes the + ``full_config`` property that ``OmniMoTModel.build_net`` reads at + ``omni_mot_model.py:327`` — pointing at the raw HF config crashes training + with ``AttributeError: 'Qwen3VLTextConfig' object has no attribute + 'full_config'``. +- ``bucket_name``, ``object_store_credential_path_pretrained``, + ``pretrained_weights.credentials_path`` are the empty string, and + ``vlm_config.tokenizer.config_variant="hf"``. +""" + +from cosmos_framework.utils.lazy_config import LazyCall as L + +from cosmos_framework.configs.base.defaults.vlm import ( + create_qwen2_tokenizer_with_download, + create_vlm_config, +) +from cosmos_framework.model.vfm.mot.unified_mot import Qwen3VLMoTConfig, Qwen3VLTextForCausalLM + + +SUPER_MODEL_CONFIG = dict( + action_gen=False, + causal_training_strategy="none", + input_caption_key="ai_caption", + input_image_key="images", + input_video_key="video", + joint_attn_implementation="two_way", + latent_downsample_factor=16, + log_enc_time_every_n=100, + lora_alpha=32, + lora_enabled=True, + lora_rank=16, + lora_target_modules="q_proj_moe_gen,k_proj_moe_gen,v_proj_moe_gen,o_proj_moe_gen", + max_action_dim=32, + max_num_tokens_after_packing=45056, + num_embodiment_domains=32, + resolution="720", + sound_gen=False, + sound_latent_fps=25, + state_ch=48, + state_t=300, + video_temporal_causal=False, + vision_gen=True, + diffusion_expert_config=dict( + base_fps=24, + enable_fps_modulation=True, + load_weights_from_pretrained=True, + max_vae_latent_side_after_patchify=20, + patch_spatial=2, + position_embedding_type="unified_3d_mrope", + rope_h_extrapolation_ratio=1.0, + rope_t_extrapolation_ratio=1.0, + rope_w_extrapolation_ratio=1.0, + timestep_range=1.0, + unified_3d_mrope_reset_spatial_ids=True, + unified_3d_mrope_temporal_modality_margin=15000, + ), + ema=dict( + enabled=False, + iteration_shift=0, + rate=0.1, + ), + lbl=dict( + coeff_gen=None, + coeff_und=None, + method="local", + ), + parallelism=dict( + cfg_parallel_shard_degree=1, + compile_dynamic=True, + compiled_region="language", + context_parallel_shard_degree=2, + coordinate_descent_tuning=False, + data_parallel_shard_degree=4, + enable_inference_mode=False, + max_autotune_pointwise=False, + precision="bfloat16", + use_cuda_graphs=False, + use_torch_compile=False, + ), + activation_checkpointing=dict( + mode="full", + ), + rectified_flow_inference_config=dict( + num_train_timesteps=1000, + scheduler_type="unipc", + shift=1, + use_dynamic_shifting=False, + ), + rectified_flow_training_config=dict( + action_loss_weight=10.0, + high_sigma_ratio=0.05, + high_sigma_timesteps_max=1000, + high_sigma_timesteps_min=995, + image_loss_scale=1.0, + independent_action_schedule=False, + loss_scale=1.0, + normalize_loss_by_active=False, + shift={"256": 3, "480": 5, "720": 10}, + train_time_action_distribution="logitnormal", + train_time_image_distribution="logitnormal", + train_time_sound_distribution="logitnormal", + train_time_video_distribution="waver", + train_time_weight="uniform", + use_discrete_rf=False, + use_dynamic_shift=False, + use_high_sigma_strategy=False, + use_high_sigma_strategy_action=False, + ), + tokenizer=dict( + bucket_name="", + chunk_duration=93, + encode_chunk_frames={"256": 68, "480": 24, "720": 12}, + encode_exact_durations=None, + keep_decoder_cache=False, + object_store_credential_path_pretrained="", + spatial_compression_factor=16, + temporal_compression_factor=4, + use_streaming_encode=False, + vae_path="pretrained/tokenizers/video/wan2pt2/Wan2.2_VAE.pth", + ), + vlm_config=dict( + layer_module="Qwen2MoTDecoderLayer", + model_name="Qwen/Qwen3-VL-32B-Instruct", + tie_word_embeddings=False, + use_system_prompt=False, + pretrained_weights=dict( + enabled=False, + backbone_path=( + "s3://nv-00-10206-checkpoint/cosmos3/pretrained/huggingface/" + "Qwen/Qwen3-VL-32B-Instruct/" + ), + credentials_path="", + enable_gcs_patch_in_boto3=True, + ), + model_instance=L(Qwen3VLTextForCausalLM)( + config=L(create_vlm_config)( + base_config=L(Qwen3VLMoTConfig.from_json_file)( + json_file=( + "cosmos_framework/model/vfm/vlm/qwen3_vl/configs/" + "Qwen3-VL-32B-Instruct.json" + ), + ), + freeze_und=False, + layer_module="MoTDecoderLayer", + qk_norm_for_text=True, + tie_word_embeddings=True, + ), + ), + tokenizer=L(create_qwen2_tokenizer_with_download)( + config_variant="hf", + pretrained_model_name="Qwen/Qwen3-VL-32B-Instruct", + ), + ), +) diff --git a/cosmos_framework/configs/base/experiment/sft/vision_sft_nano.py b/cosmos_framework/configs/base/experiment/sft/vision_sft_nano.py new file mode 100644 index 0000000..13968a7 --- /dev/null +++ b/cosmos_framework/configs/base/experiment/sft/vision_sft_nano.py @@ -0,0 +1,284 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""``vision_sft_nano`` — 1-to-1 conversion of + + cosmos3-internal/cosmos3/configs/experiment/vision_sft_nano.yaml + +into the prerelease layout. Same Qwen3-VL-8B / Cosmos3-Nano backbone, T2V/I2V/V2V +SFT, PackingDataLoader + RankPartitionedDataLoader stack. EMA enabled. + +Translation notes (compared with the source YAML): + * ``cosmos3._src.*`` module paths were rewritten to ``cosmos_framework.*`` (the + prerelease module layout). + * ``_target_`` references for ``model``, ``optimizer``, ``scheduler``, + ``checkpoint``, ``callbacks``, ``ema``, ``tokenizer``, ``cluster``, + ``vlm_config``, and ``ckpt_type`` flow from the ``defaults:`` group + choices, matching the YAML's ``defaults:`` list (``_self_`` is placed + LAST per prerelease convention so the experiment overrides the + defaults, but no setting changes semantically). + * The YAML's giant ``model_parallel`` block, ``trainer.profiling``, + ``trainer.straggler_detection`` and ``trainer.type`` are populated by + the base ``Config`` (``cosmos_framework/configs/base/config.py``) defaults and + are therefore omitted here. + +``checkpoint.load_path`` is left as ``???`` (a Hydra MISSING marker); supply +via CLI / a downstream experiment that inherits from this one. + +Usage:: + + CUDA_VISIBLE_DEVICES=0 PYTHONPATH=. torchrun --nproc_per_node=1 \\ + --master_port=12341 -m cosmos_framework.scripts.train \\ + --config=cosmos_framework/configs/base/config.py -- \\ + experiment=vision_sft_nano \\ + checkpoint.load_path= +""" + +import copy + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils.lazy_config import LazyDict + +from cosmos_framework.configs.base.experiment.sft.models.nano_model_config import NANO_MODEL_CONFIG +from cosmos_framework.data.vfm.joint_dataloader import ( + PackingDataLoader, + RankPartitionedDataLoader, +) +from cosmos_framework.data.vfm.local_datasets.sft_dataset import get_sft_dataset + +cs = ConfigStore.instance() + + +vision_sft_nano = LazyDict( + dict( + defaults=[ + {"override /model": "mot_fsdp"}, + {"override /data_train": None}, + {"override /data_val": None}, + {"override /optimizer": "adamw"}, + # YAML used `scheduler: warmup_cosine_lr` but that group is only + # registered in cosmos_framework/configs/base/vlm/defaults/optimizer.py + # (reachable from the vlm config tree). The base vfm config path + # only knows `lambdacosine`, which also sets + # lr_scheduler_type="LambdaCosine" — behaviorally identical. + {"override /scheduler": "lambdacosine"}, + {"override /checkpoint": "s3"}, + { + "override /callbacks": [ + "basic", + "optimization", + "job_monitor", + "generation", + ] + }, + {"override /ema": "power"}, + {"override /tokenizer": "wan2pt2_tokenizer"}, + {"override /sound_tokenizer": None}, + {"override /cluster": None}, + {"override /vlm_config": None}, + {"override /ckpt_type": "dcp"}, + "_self_", + ], + job=dict( + project="cosmos3", + group="sft", + name="vision_sft_nano", + wandb_mode="disabled", + ), + model=dict( + config=copy.deepcopy(NANO_MODEL_CONFIG), + ), + optimizer=dict( + betas=[0.9, 0.95], + eps=1.0e-06, + fused=True, + keys_to_select=[ + "moe_gen", + "time_embedder", + "vae2llm", + "llm2vae", + ], + lr=5.0e-04, + lr_multipliers={}, + optimizer_type="AdamW", + weight_decay=0, + ), + scheduler=dict( + lr_scheduler_type="LambdaCosine", + cycle_lengths=[1000], + f_max=[1.0], + f_min=[0.0], + f_start=[0.0], + verbosity_interval=0, + warm_up_steps=[50], + ), + trainer=dict( + distributed_parallelism="fsdp", + grad_accum_iter=2, + logging_iter=1, + max_iter=500, + max_val_iter=None, + # YAML had `memory_format: preserve_format` as a string, but the + # prerelease trainer passes this verbatim to model.to(memory_format=…) + # which requires a torch.memory_format enum (not a string). + # Omit and let the framework default apply, matching how + # mixed_modality_sft_nano.py handles it. + run_validation=False, + run_validation_on_start=False, + save_zero_checkpoint=False, + seed=42, + timeout_period=999999999, + validation_iter=100, + compile_config=dict(recompile_limit=8, use_duck_shape=False), + cudnn=dict(benchmark=True, deterministic=False), + ddp=dict(broadcast_buffers=True, find_unused_parameters=False, static_graph=True), + grad_scaler_args=dict(enabled=False), + callbacks=dict( + compile_tokenizer=dict( + compile_after_iterations=3, + enabled=False, + warmup_resolutions=None, + ), + dataloader_speed=dict(every_n=100, save_s3=False, step_size=1), + device_monitor=dict( + every_n=200, + log_memory_detail=True, + save_s3=False, + step_size=1, + upload_every_n_mul=5, + ), + expert_heatmap=dict(every_n=1000), + grad_clip=dict(clip_norm=0.1, force_finite=True), + heart_beat=dict(every_n=200, save_s3=False, step_size=1, update_interval_in_minute=20), + iter_speed=dict(every_n=1, hit_thres=50, save_s3=False, save_s3_every_log_n=500), + low_precision=dict(update_iter=1), + manual_gc=dict(every_n=5, gc_level=1, warm_up=1), + norm_monitor=dict( + every_n=100, + layer_norm_only=False, + log_stat_wandb=True, + model_key=None, + save_s3=False, + step_size=1, + track_activations=True, + ), + param_count=dict(save_s3=False), + sequence_packing_padding=dict(every_n=50), + sigma_loss_analysis=dict(every_n=500, every_n_viz=500, save_s3=False), + skip_nan_step=dict(max_consecutive_nan=100), + training_stats=dict(log_freq=100), + wandb_2x=dict( + logging_iter_multipler=2, + save_logging_iter_multipler=1, + save_s3=False, + ), + wandb_val=dict(save_s3=False), + ), + ), + checkpoint=dict( + broadcast_via_filesystem=False, + dcp_async_mode_enabled=False, + enable_gcs_patch_in_boto3=True, + keys_not_to_resume=[], + keys_to_skip_loading=["net_ema."], + load_ema_to_reg=False, + load_path="???", # supply via CLI / downstream experiment + load_training_state=False, + only_load_scheduler_state=False, + save_iter=100, + strict_resume=True, + verbose=True, + hf_export=dict( + enabled=False, + export_every_n=1, + hf_repo_id=None, + upload_to_object_store=dict( + bucket="", + credentials="", + enabled=False, + ), + ), + jit=dict( + device="cuda", + dtype="bfloat16", + enabled=False, + input_shape=None, + strict=True, + ), + load_from_object_store=dict( + bucket="", + credentials="", + enabled=False, + ), + save_to_object_store=dict( + bucket="", + credentials="", + enabled=False, + ), + ), + dataloader_train=L(PackingDataLoader)( + audio_sample_rate=48000, + dataset_name="default", + max_samples_per_batch=None, + max_sequence_length=45056, + patch_spatial=2, + sound_latent_fps=0, + tokenizer_spatial_compression_factor=16, + tokenizer_temporal_compression_factor=4, + dataloader=L(RankPartitionedDataLoader)( + batch_size=1, + in_order=True, + num_workers=4, + persistent_workers=True, + pin_memory=True, + prefetch_factor=4, + sampler=None, + datasets=dict( + video=dict( + ratio=1, + dataset=L(get_sft_dataset)( + append_duration_fps_timestamps=True, + append_resolution_info=True, + caption_suffix="", + cfg_dropout_keep_metadata=False, + cfg_dropout_rate=0.1, + # 70% T2V, 20% I2V (first frame), 10% V2V (first 5 frames / 2 latent frames) + conditioning_config={0: 0.7, 1: 0.2, 2: 0.1}, + conditioning_fps=-1, + conditioning_fps_noise_std=0.0, + frame_selection_mode="first", + jsonl_paths=["${oc.env:DATASET_PATH}/train/video_dataset_file.jsonl"], + min_short_edge=0, + num_video_frames=-1, + resolution="256", + sample_by_window=False, + temporal_compression_factor=4, + temporal_interval_mode="max_30fps", + use_system_prompt=False, + # YAML spells this out as + # _target_: create_qwen2_tokenizer_with_download + # config_variant: gcp + # but that pins the dataset's tokenizer to the GCP + # variant, requiring credentials/gcp_checkpoint.secret. + # Use a Hydra interpolation instead so launchers + # (e.g. launch_vision_sft_nano_toml.sh) can flip + # model.config.vlm_config.tokenizer.config_variant=hf + # and have the dataset inherit the same setting. + tokenizer_config="${model.config.vlm_config.tokenizer}", + ), + ), + ), + ), + ), + dataloader_val=None, + upload_reproducible_setup=False, + ), + flags={"allow_objects": True}, +) + + +for _item in [vision_sft_nano]: + _name = [k for k, v in globals().items() if v is _item][0] + cs.store(group="experiment", package="_global_", name=_name, node=_item) diff --git a/cosmos_framework/configs/base/experiment/sft/vision_sft_super.py b/cosmos_framework/configs/base/experiment/sft/vision_sft_super.py new file mode 100644 index 0000000..1806cbb --- /dev/null +++ b/cosmos_framework/configs/base/experiment/sft/vision_sft_super.py @@ -0,0 +1,305 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""``vision_sft_super`` — 1-to-1 conversion of + + cosmos3-internal/cosmos3/configs/experiment/vision_sft_super.yaml + +into the prerelease layout. Sibling of ``vision_sft_nano``: same source-YAML +shape, same Hydra defaults, same dataloader stack +(PackingDataLoader + RankPartitionedDataLoader), same three forced +bucket-C deviations (see docs/vision_sft_nano_conversion.md). Key model-side +differences vs nano: + + * Qwen3-VL-**32B**-Instruct backbone (not 8B). + * LoRA-only fine-tune: ``lora_enabled=True``, ``lora_rank=16``, + ``lora_alpha=32``, target modules + ``q_proj_moe_gen,k_proj_moe_gen,v_proj_moe_gen,o_proj_moe_gen``. + * EMA disabled; ``action_gen=False``. + * Parallelism: ``data_parallel_shard_degree=4``, + ``context_parallel_shard_degree=2``, ``use_torch_compile=False``. + * Optimizer trains only ``lora_`` keys at ``lr=5e-4``. + * Checkpoint resume non-strict; ``keys_to_skip_loading`` includes + ``lora_`` so LoRA tensors are NOT loaded from the base checkpoint + (they're freshly initialized). + +Translation notes (mirror vision_sft_nano.py): + * ``cosmos3._src.*`` paths rewritten to ``cosmos_framework.*``. + * ``_self_`` placed LAST in defaults (prerelease convention) instead + of FIRST (YAML convention) — see vision_sft_nano.py docstring. + * ``model_parallel``, ``trainer.profiling``, ``trainer.straggler_detection``, + ``trainer.type`` blocks are omitted — populated by base Config defaults. + +Three forced deviations from the YAML literal (identical to nano): + 1. ``override /scheduler: lambdacosine`` (YAML used ``warmup_cosine_lr``, + which is only registered in the vlm config tree). + 2. dataset's ``tokenizer_config`` is a Hydra interpolation + (``"${model.config.vlm_config.tokenizer}"``) instead of a literal + ``L(create_qwen2_tokenizer_with_download)(config_variant="gcp")`` so + launcher tail overrides (``…config_variant=hf``) reach it. + 3. ``trainer.memory_format`` omitted; YAML literal was the string + ``preserve_format`` which the prerelease trainer can't coerce. + +``checkpoint.load_path`` is left as ``???``; supply via CLI / a downstream +experiment that inherits from this one. + +Usage:: + + PYTHONPATH=. torchrun --nproc_per_node=8 \\ + --master_port=12341 -m cosmos_framework.scripts.train \\ + --config=cosmos_framework/configs/base/config.py -- \\ + experiment=vision_sft_super \\ + checkpoint.load_path= +""" + +import copy + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils.lazy_config import LazyDict + +from cosmos_framework.configs.base.experiment.sft.models.super_model_config import SUPER_MODEL_CONFIG +from cosmos_framework.data.vfm.joint_dataloader import ( + PackingDataLoader, + RankPartitionedDataLoader, +) +from cosmos_framework.data.vfm.local_datasets.sft_dataset import get_sft_dataset + +cs = ConfigStore.instance() + + +vision_sft_super = LazyDict( + dict( + defaults=[ + {"override /model": "mot_fsdp"}, + {"override /data_train": None}, + {"override /data_val": None}, + {"override /optimizer": "adamw"}, + # YAML used `scheduler: warmup_cosine_lr` but that group is only + # registered in cosmos_framework/configs/base/vlm/defaults/optimizer.py + # (reachable from the vlm config tree). The base vfm config path + # only knows `lambdacosine`, which also sets + # lr_scheduler_type="LambdaCosine" — behaviorally identical. + {"override /scheduler": "lambdacosine"}, + {"override /checkpoint": "s3"}, + { + "override /callbacks": [ + "basic", + "optimization", + "job_monitor", + "generation", + ] + }, + {"override /ema": "power"}, + {"override /tokenizer": "wan2pt2_tokenizer"}, + {"override /sound_tokenizer": None}, + {"override /cluster": None}, + {"override /vlm_config": None}, + {"override /ckpt_type": "dcp"}, + "_self_", + ], + job=dict( + project="cosmos3", + group="sft", + name="vision_sft_super", + wandb_mode="disabled", + ), + model=dict( + config=copy.deepcopy(SUPER_MODEL_CONFIG), + ), + optimizer=dict( + betas=[0.9, 0.95], + disable_weight_decay_for_1d_params=False, + eps=1.0e-06, + fused=True, + # LoRA-only fine-tune: optimize only parameters whose name contains + # `lora_` (matches the prefixes injected by the LoRA adapter wiring). + keys_to_select=["lora_"], + lr=5.0e-04, + lr_multipliers={}, + optimizer_type="AdamW", + weight_decay=0, + ), + scheduler=dict( + lr_scheduler_type="LambdaCosine", + cycle_lengths=[1000], + f_max=[1.0], + f_min=[0.0], + f_start=[0.0], + verbosity_interval=0, + warm_up_steps=[50], + ), + trainer=dict( + distributed_parallelism="fsdp", + grad_accum_iter=2, + logging_iter=1, + max_iter=500, + max_val_iter=None, + # YAML had `memory_format: preserve_format` as a string, but the + # prerelease trainer passes this verbatim to model.to(memory_format=…) + # which requires a torch.memory_format enum (not a string). + # Omit and let the framework default apply, matching what + # vision_sft_nano.py / mixed_modality_sft_nano.py do. + run_validation=False, + run_validation_on_start=False, + save_zero_checkpoint=False, + seed=42, + timeout_period=999999999, + validation_iter=100, + compile_config=dict(recompile_limit=8, use_duck_shape=False), + cudnn=dict(benchmark=True, deterministic=False), + ddp=dict(broadcast_buffers=True, find_unused_parameters=False, static_graph=True), + grad_scaler_args=dict(enabled=False), + callbacks=dict( + compile_tokenizer=dict( + compile_after_iterations=3, + enabled=False, + warmup_resolutions=["256", "480", "720"], + ), + dataloader_speed=dict(every_n=100, save_s3=False, step_size=1), + device_monitor=dict( + every_n=200, + log_memory_detail=True, + save_s3=False, + step_size=1, + upload_every_n_mul=5, + ), + expert_heatmap=dict(every_n=1000), + grad_clip=dict(clip_norm=0.1, force_finite=True), + heart_beat=dict(every_n=200, save_s3=False, step_size=1, update_interval_in_minute=20), + iter_speed=dict(every_n=1, hit_thres=50, save_s3=False, save_s3_every_log_n=500), + low_precision=dict(update_iter=1), + manual_gc=dict(every_n=5, gc_level=1, warm_up=1), + norm_monitor=dict( + every_n=100, + layer_norm_only=False, + log_stat_wandb=True, + model_key=None, + save_s3=False, + step_size=1, + track_activations=True, + ), + param_count=dict(save_s3=False), + sequence_packing_padding=dict(every_n=50), + sigma_loss_analysis=dict(every_n=500, every_n_viz=500, save_s3=False), + skip_nan_step=dict(max_consecutive_nan=100), + training_stats=dict(log_freq=100), + wandb_2x=dict( + logging_iter_multipler=2, + save_logging_iter_multipler=1, + save_s3=False, + ), + wandb_val=dict(save_s3=False), + ), + ), + checkpoint=dict( + broadcast_via_filesystem=False, + dcp_async_mode_enabled=False, + enable_gcs_patch_in_boto3=True, + keys_not_to_resume=[], + # `lora_` added so LoRA tensors are NOT loaded from the base DCP + # checkpoint (which doesn't have them) — they're freshly initialized. + keys_to_skip_loading=["net_ema.", "lora_"], + load_ema_to_reg=False, + load_path="???", # supply via CLI / downstream experiment + load_training_state=False, + only_load_scheduler_state=False, + save_iter=100, + # Non-strict resume because LoRA tensors are absent in the base + # checkpoint (see keys_to_skip_loading above). + strict_resume=False, + verbose=True, + hf_export=dict( + enabled=False, + export_every_n=1, + hf_repo_id=None, + upload_to_object_store=dict( + bucket="", + credentials="", + enabled=False, + ), + ), + jit=dict( + device="cuda", + dtype="bfloat16", + enabled=False, + input_shape=None, + strict=True, + ), + load_from_object_store=dict( + bucket="", + credentials="", + enabled=False, + ), + save_to_object_store=dict( + bucket="", + credentials="", + enabled=False, + ), + ), + dataloader_train=L(PackingDataLoader)( + audio_sample_rate=48000, + dataset_name="default", + max_samples_per_batch=None, + max_sequence_length=45056, + patch_spatial=2, + sound_latent_fps=0, + tokenizer_spatial_compression_factor=16, + tokenizer_temporal_compression_factor=4, + dataloader=L(RankPartitionedDataLoader)( + batch_size=1, + in_order=True, + num_workers=4, + persistent_workers=True, + pin_memory=True, + prefetch_factor=4, + sampler=None, + datasets=dict( + video=dict( + ratio=1, + dataset=L(get_sft_dataset)( + append_duration_fps_timestamps=True, + append_resolution_info=True, + caption_suffix="", + cfg_dropout_keep_metadata=False, + cfg_dropout_rate=0.1, + # 70% T2V, 20% I2V (first frame), 10% V2V (first 5 frames / 2 latent frames) + conditioning_config={0: 0.7, 1: 0.2, 2: 0.1}, + conditioning_fps=-1, + conditioning_fps_noise_std=0.0, + frame_selection_mode="first", + jsonl_paths=["${oc.env:DATASET_PATH}/train/video_dataset_file.jsonl"], + min_short_edge=0, + num_video_frames=-1, + resolution="256", + sample_by_window=False, + temporal_compression_factor=4, + temporal_interval_mode="max_30fps", + use_system_prompt=False, + # YAML spells this out as + # _target_: create_qwen2_tokenizer_with_download + # config_variant: gcp + # pretrained_model_name: Qwen/Qwen3-VL-32B-Instruct + # but that pins the dataset's tokenizer to the GCP + # variant, requiring credentials/gcp_checkpoint.secret. + # Use a Hydra interpolation instead so launchers + # (e.g. launch_vision_sft_super_toml.sh) can flip + # model.config.vlm_config.tokenizer.config_variant=hf + # and have the dataset inherit the same setting. + tokenizer_config="${model.config.vlm_config.tokenizer}", + ), + ), + ), + ), + ), + dataloader_val=None, + upload_reproducible_setup=False, + ), + flags={"allow_objects": True}, +) + + +for _item in [vision_sft_super]: + _name = [k for k, v in globals().items() if v is _item][0] + cs.store(group="experiment", package="_global_", name=_name, node=_item) diff --git a/cosmos_framework/configs/base/vlm/__init__.py b/cosmos_framework/configs/base/vlm/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/configs/base/vlm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/configs/base/vlm/config.py b/cosmos_framework/configs/base/vlm/config.py new file mode 100644 index 0000000..66dc3ed --- /dev/null +++ b/cosmos_framework/configs/base/vlm/config.py @@ -0,0 +1,52 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from cosmos_framework.trainer import ImaginaireTrainer +from cosmos_framework.utils import log +from cosmos_framework.utils.config_helper import import_all_modules_from_package +from cosmos_framework.configs.base.vlm.defaults.callbacks import register_callbacks +from cosmos_framework.configs.base.vlm.defaults.checkpointer import register_checkpoint, register_ckpt_type +from cosmos_framework.configs.base.vlm.defaults.config import Config + +from cosmos_framework.configs.base.vlm.defaults.model import register_model +from cosmos_framework.configs.base.vlm.defaults.optimizer import register_optimizer, register_scheduler +from cosmos_framework.configs.base.vlm.defaults.vlm_policy import register_vlm_policy + + +def make_config() -> Config: + c = Config( + model=None, + optimizer=None, + scheduler=None, + dataloader_train=None, + dataloader_val=None, + ) + + # Specifying values through instances of attrs + c.job.project = "cosmos_reason2" + c.job.group = "debug" + c.job.name = "delete_${now:%Y-%m-%d}_${now:%H-%M-%S}" + + # Unified path: ImaginaireTrainer drives both VLM and VFM. + c.trainer.type = ImaginaireTrainer + c.trainer.straggler_detection.enabled = False + c.trainer.max_iter = 400_000 + c.trainer.logging_iter = 20 + c.trainer.validation_iter = 100 + c.trainer.run_validation = False + c.trainer.callbacks = None + c.trainer.cudnn.benchmark = False + c.upload_reproducible_setup = True + + # Call this function to register config groups for advanced overriding. the order follows the default config groups + register_model() + register_vlm_policy() + # Register dataloader configs + log.info("Registering optimizer, scheduler, checkpoint, ckpt type, and callbacks") + register_optimizer() + register_scheduler() + register_checkpoint() + register_ckpt_type() + register_callbacks() + import_all_modules_from_package("cosmos_framework.configs.base.vlm.experiment", reload=True) + return c diff --git a/cosmos_framework/configs/base/vlm/defaults/__init__.py b/cosmos_framework/configs/base/vlm/defaults/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/configs/base/vlm/defaults/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/configs/base/vlm/defaults/callbacks.py b/cosmos_framework/configs/base/vlm/defaults/callbacks.py new file mode 100644 index 0000000..4d222d4 --- /dev/null +++ b/cosmos_framework/configs/base/vlm/defaults/callbacks.py @@ -0,0 +1,87 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Dataloader config options. +Based on projects/cosmos/ar/v1/configs/registry.py +""" + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.callbacks.manual_gc import ManualGarbageCollection +from cosmos_framework.utils.lazy_config import PLACEHOLDER +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils.callback import LowPrecisionCallback, WandBCallback +from cosmos_framework.callbacks.dataloader_state import DataLoaderStateCallback + +from cosmos_framework.callbacks.grad_clip import GradClip +from cosmos_framework.callbacks.hf_export import HFExportCallback +from cosmos_framework.callbacks.iter_speed import IterSpeed +from cosmos_framework.callbacks.learning_rate_logger import LearningRateLogger +from cosmos_framework.callbacks.log_tensor_shape import LogTensorShapeCallback +from cosmos_framework.callbacks.param_count import ParamCount +from cosmos_framework.callbacks.wandb_log import WandbCallback as WandBCallbackMultiplier +from cosmos_framework.callbacks.wandb_vis import VisualizationLoggingCallback +from cosmos_framework.configs.base.defaults.callbacks import JOB_MONITOR_CALLBACKS + +# from cosmos_framework.utils.callback import NVTXCallback + + +def register_callbacks(): + cs = ConfigStore.instance() + BASIC_CALLBACKS = dict( + iter_speed=L(IterSpeed)( # does not use model or optimizer + every_n="${trainer.logging_iter}", + save_s3="${upload_reproducible_setup}", + save_s3_every_log_n=500, + hit_thres=50, + ), + manual_gc=L(ManualGarbageCollection)(every_n=5), # does not use model or optimizer + wandb=L(WandBCallback)(), + param_count=L(ParamCount)( # use model + save_s3="${upload_reproducible_setup}", + ), + grad_clip=L(GradClip)(clip_norm=1.0, force_finite=False), # use model + learning_rate_logger=L(LearningRateLogger)(every_n=10), + low_precision=L(LowPrecisionCallback)( + update_iter=1, + config=PLACEHOLDER, + trainer=PLACEHOLDER, + ), # reads model.precision; no extra kwarg needed + + # nvtx=L(NVTXCallback)(synchronize=True), + ) + + LOG_CALLBACKS = dict( + wandb_10x=L(WandBCallbackMultiplier)( + logging_iter_multipler=10, + save_logging_iter_multipler=1, + save_s3="${upload_reproducible_setup}", + ), + wandb_2x=L(WandBCallbackMultiplier)( + logging_iter_multipler=2, + save_logging_iter_multipler=1, + save_s3="${upload_reproducible_setup}", + ), + log_tensor_shape=L(LogTensorShapeCallback)(num_log=10), + dataloader_state=L(DataLoaderStateCallback)( + distributor_type="${data_setting.distributor_type}", + ), + ) + + cs.store(group="callbacks", package="trainer.callbacks", name="basic_vlm", node=BASIC_CALLBACKS) + cs.store(group="callbacks", package="trainer.callbacks", name="basic_log", node=LOG_CALLBACKS) + cs.store(group="callbacks", package="trainer.callbacks", name="job_monitor", node=JOB_MONITOR_CALLBACKS) + + DATA_VIS_CALLBACKS_QWEN = dict( + wandb_vis=L(VisualizationLoggingCallback)( + every_n=500, + ), + ) + cs.store(group="callbacks", package="trainer.callbacks", name="data_vis_qwen", node=DATA_VIS_CALLBACKS_QWEN) + + HF_EXPORT_CALLBACKS = dict( + hf_export=L(HFExportCallback)( + dtype="${model.config.policy.parallelism.precision}", + ), + ) + cs.store(group="callbacks", package="trainer.callbacks", name="hf_export", node=HF_EXPORT_CALLBACKS) diff --git a/cosmos_framework/configs/base/vlm/defaults/checkpointer.py b/cosmos_framework/configs/base/vlm/defaults/checkpointer.py new file mode 100644 index 0000000..246d87a --- /dev/null +++ b/cosmos_framework/configs/base/vlm/defaults/checkpointer.py @@ -0,0 +1,93 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Dict + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils import config +from cosmos_framework.checkpoint.dummy import Checkpointer as DummyCheckpointer +from cosmos_framework.utils.config import CheckpointConfig +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.checkpoint.dcp import DistributedCheckpointer + +local_object_store = config.ObjectStoreConfig( + enabled=False, +) + +pdx_object_store = config.ObjectStoreConfig( + enabled=True, + credentials="credentials/pdx_vfm_checkpoint.secret", + bucket="checkpoints", +) + +s3_object_store = config.ObjectStoreConfig( + enabled=True, + credentials="credentials/s3_training.secret", + bucket="checkpoints-us-east-1", +) + +s3_eu_object_store = config.ObjectStoreConfig( + enabled=True, + credentials="credentials/s3_training_eu.secret", + bucket="checkpoints-eu-west-3", +) + +gcp_object_store = config.ObjectStoreConfig( + enabled=True, + credentials="credentials/gcp_checkpoint.secret", + bucket="nv-00-10206-checkpoint-experiments", +) + +CHECKPOINT_LOCAL = CheckpointConfig( + save_to_object_store=local_object_store, + load_from_object_store=local_object_store, + save_iter=5000, + broadcast_via_filesystem=True, + dcp_async_mode_enabled=True, +) + +CHECKPOINT_PDX = CheckpointConfig( + save_to_object_store=pdx_object_store, + load_from_object_store=pdx_object_store, + save_iter=5000, + broadcast_via_filesystem=True, + dcp_async_mode_enabled=True, +) + +CHECKPOINT_S3 = CheckpointConfig( + save_to_object_store=s3_object_store, + load_from_object_store=s3_object_store, + save_iter=5000, + broadcast_via_filesystem=True, + dcp_async_mode_enabled=True, +) + +CHECKPOINT_GCP = CheckpointConfig( + save_to_object_store=gcp_object_store, + load_from_object_store=gcp_object_store, + save_iter=1000, + load_path="", + load_training_state=False, + strict_resume=True, + enable_gcs_patch_in_boto3=True, + dcp_async_mode_enabled=True, +) + + +def register_checkpoint() -> None: + cs = ConfigStore.instance() + cs.store(group="checkpoint", package="checkpoint", name="local", node=CHECKPOINT_LOCAL) + cs.store(group="checkpoint", package="checkpoint", name="pdx", node=CHECKPOINT_PDX) + cs.store(group="checkpoint", package="checkpoint", name="s3", node=CHECKPOINT_S3) + cs.store(group="checkpoint", package="checkpoint", name="gcp", node=CHECKPOINT_GCP) + + +DUMMY_CHECKPOINTER: Dict[str, str] = L(DummyCheckpointer)() +DISTRIBUTED_CHECKPOINTER: Dict[str, str] = L(DistributedCheckpointer)() + + +def register_ckpt_type() -> None: + cs = ConfigStore.instance() + cs.store(group="ckpt_type", package="checkpoint.type", name="dummy", node=DUMMY_CHECKPOINTER) + cs.store(group="ckpt_type", package="checkpoint.type", name="dcp", node=DISTRIBUTED_CHECKPOINTER) diff --git a/cosmos_framework/configs/base/vlm/defaults/config.py b/cosmos_framework/configs/base/vlm/defaults/config.py new file mode 100644 index 0000000..e9f527c --- /dev/null +++ b/cosmos_framework/configs/base/vlm/defaults/config.py @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Any, List + +import attrs + +from cosmos_framework.utils import config +from cosmos_framework.configs.base.vlm.defaults.training import PolicyConfig, TrainConfig + + +@attrs.define(slots=False) +class DataSetting: + """Configuration for data. + + Attributes: + qwen_max_video_token_length: Maximum video token length. + qwen_target_fps: Target fps for video sampling. + text_chat_order: Order of text items in user messages. + distributor_type: "with_replace" (WeightedShardlistBasic) or "no_replace" (NoReplaceShardlistBasic). + distributor_seed: Seed for the distributor. + """ + + qwen_max_video_token_length: int = 8192 + qwen_max_image_token_length: int = 8192 + qwen_target_fps: float = 4.0 + text_chat_order: str = attrs.field( + default="text_end", + validator=attrs.validators.in_({"text_end", "text_start", "random"}), + ) + temporal_localization_output_format: str = attrs.field( + default="random", + validator=attrs.validators.in_({"dense_video_caption", "temporal_localization", "temporal_caption", "random"}), + ) + temporal_localization_fps: float = 1.0 + # For packed dataset + max_batch_size: int = 1 + max_tokens: int = 16000 + # "with_replace" (WeightedShardlistBasic) or "no_replace" (NoReplaceShardlistBasic). + distributor_type: str = attrs.field( + default="with_replace", + validator=attrs.validators.in_({"with_replace", "no_replace"}), + ) + distributor_seed: int = 1993 + webdataset_detshuffle: bool = False + num_data_workers: int = 8 + data_prefetch_factor: int = 1 + val_split_ratio: float = 0.0 + + +@attrs.define(slots=False) +class Config(config.Config): + train: TrainConfig = TrainConfig() + policy: PolicyConfig = PolicyConfig() + data_setting: DataSetting = DataSetting() + defaults: List[Any] = attrs.field( + factory=lambda: [ + "_self_", + {"model": "vlm_fsdp"}, + {"vlm_policy": None}, + {"data_train": None}, + {"data_val": None}, + {"optimizer": "fusedadamw"}, + {"scheduler": "lambdacosine"}, + {"checkpoint": "s3"}, + {"ckpt_type": "dcp"}, + {"callbacks": ["basic_vlm"]}, + {"experiment": None}, + ] + ) diff --git a/cosmos_framework/configs/base/vlm/defaults/dataloader.py b/cosmos_framework/configs/base/vlm/defaults/dataloader.py new file mode 100644 index 0000000..36b878d --- /dev/null +++ b/cosmos_framework/configs/base/vlm/defaults/dataloader.py @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from torch.utils.data import DataLoader + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils.config_helper import ConfigStore +from cosmos_framework.data.vfm.vlm.collate_fn import custom_collate +from cosmos_framework.data.vfm.vlm.debug_data_qwen import DebugQwenDataset +from cosmos_framework.data.vfm.vlm.dummy_data_qwen import DummyQwenDataset +from cosmos_framework.data.vfm.processors import build_processor_lazy + + +# Debug dataset +def create_debug_dataloader_config_qwen( + num_images, loss_on_completion_only: bool = True, use_dummy_image: bool = False +): + return L(DataLoader)( + dataset=L(DebugQwenDataset)( + tokenizer=L(build_processor_lazy)( + tokenizer_type="${model.config.policy.backbone.model_name}", + credentials="${checkpoint.load_from_object_store.credentials}", + bucket="${checkpoint.load_from_object_store.bucket}", + ), + num_images=num_images, + seq_len="${model.config.policy.model_max_length}", + image_token_len="${model.config.policy.qwen_max_video_token_length}", + # use_dummy_image=use_dummy_image, + ), + num_workers=8, + prefetch_factor=4, + batch_size=1, + sampler=None, + persistent_workers=False, + pin_memory=True, + collate_fn=custom_collate, + ) + + +def create_dummy_dataloader_config_qwen(): + return L(DataLoader)( + dataset=L(DummyQwenDataset)( + tokenizer=L(build_processor_lazy)( + tokenizer_type="${model.config.policy.backbone.model_name}", + credentials="${checkpoint.load_from_object_store.credentials}", + bucket="${checkpoint.load_from_object_store.bucket}", + ), + num_visual_tokens="${model.config.policy.qwen_max_video_token_length}", + total_tokens="${model.config.policy.model_max_length}", + batch_size="${dataloader_train.batch_size}", + ), + num_workers=8, + prefetch_factor=4, + batch_size=1, + sampler=None, + persistent_workers=False, + pin_memory=True, + collate_fn=custom_collate, + ) + + +def register_data_debug(): + cs = ConfigStore.instance() + for split in ["train", "val"]: + cs.store( + group=f"data_{split}", + package=f"dataloader_{split}", + name="debug_image_data_qwen", # This data is from pixtral model output, expected to have low loss ~1.4 + node=create_debug_dataloader_config_qwen(1), + ) + cs.store( + group=f"data_{split}", + package=f"dataloader_{split}", + name="dummy_image_data_qwen", + node=create_dummy_dataloader_config_qwen(), + ) + + +def register_data(): + register_data_debug() diff --git a/cosmos_framework/configs/base/vlm/defaults/model.py b/cosmos_framework/configs/base/vlm/defaults/model.py new file mode 100644 index 0000000..e787dfa --- /dev/null +++ b/cosmos_framework/configs/base/vlm/defaults/model.py @@ -0,0 +1,23 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.configs.base.defaults.model_config import VLMModelConfig +from cosmos_framework.model.vfm.vlm_model import VLMModel + +VLM_FSDP_CONFIG = dict( + trainer=dict( + distributed_parallelism="fsdp", + ), + model=L(VLMModel)( + config=VLMModelConfig(), + checkpoint="${checkpoint}", + ), +) + + +def register_model(): + cs = ConfigStore.instance() + cs.store(group="model", package="_global_", name="vlm_fsdp", node=VLM_FSDP_CONFIG) diff --git a/cosmos_framework/configs/base/vlm/defaults/optimizer.py b/cosmos_framework/configs/base/vlm/defaults/optimizer.py new file mode 100644 index 0000000..6632dc8 --- /dev/null +++ b/cosmos_framework/configs/base/vlm/defaults/optimizer.py @@ -0,0 +1,45 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 +"""Hydra config registrations for VLM optimizer + LR scheduler.""" + +from typing import Any + +from cosmos_framework.configs.base.defaults.optimizer import ( + register_optimizers, + register_schedulers, +) + +# Shared optimizer kwargs for both fusedadamw and adamw registrations. +# ``lr_multipliers``: vision_encoder backbone at 0.1x base LR; everything else +# at 1.0x. Substring match; only entries != 1.0 need to appear. See +# ``_filter_params_grouped`` in ``vfm/utils/optimizer.py``. +VLM_OPTIMIZER_KWARGS: dict[str, Any] = dict( + lr=2e-6, + weight_decay=0.1, + betas=(0.9, 0.95), + fused=True, + keys_to_select=[], + lr_multipliers={"vision_encoder": 0.1}, +) + +# ``f_start`` / ``f_min`` are ratios of the optimizer's base ``lr``: +# effective_init_lr = lr * f_start +# effective_end_lr = lr * f_min +# Update these together with ``lr`` if you want absolute LR endpoints to stay fixed. +VLM_LAMBDACOSINE_KWARGS: dict[str, Any] = dict( + warm_up_steps=[1000], + cycle_lengths=["${trainer.max_iter}"], + f_start=[0.05], + f_max=[1.0], + f_min=[0.5], +) + + +def register_optimizer() -> None: + """VLM project-root entry point.""" + register_optimizers(VLM_OPTIMIZER_KWARGS) + + +def register_scheduler() -> None: + """VLM project-root entry point.""" + register_schedulers(VLM_LAMBDACOSINE_KWARGS) diff --git a/cosmos_framework/configs/base/vlm/defaults/training.py b/cosmos_framework/configs/base/vlm/defaults/training.py new file mode 100644 index 0000000..89e2660 --- /dev/null +++ b/cosmos_framework/configs/base/vlm/defaults/training.py @@ -0,0 +1,112 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from dataclasses import MISSING, field +from typing import Union + +import attrs +import torch + +from cosmos_framework.utils.config import make_freezable +from cosmos_framework.configs.base.defaults.activation_checkpointing import ActivationCheckpointingConfig +from cosmos_framework.configs.base.defaults.parallelism import ParallelismConfig +from cosmos_framework.configs.base.defaults.vlm import VLMConfig + + +def skip_ui_field(*, default=MISSING, default_factory=MISSING, **kwargs): + metadata = kwargs.pop("metadata", {}) + metadata["skip_ui"] = True + if default_factory is not MISSING: + return field(default_factory=default_factory, metadata=metadata, **kwargs) + elif default is not MISSING: + return field(default=default, metadata=metadata, **kwargs) + else: + raise ValueError("Must provide either default or default_factory.") + + +@make_freezable +@attrs.define(slots=False) +class TrainPolicyConfig: + mini_batch: int = 1 + type: str = "sft" + + +@make_freezable +@attrs.define(slots=False) +class FP8: + enable_fp8: bool = False + + +@make_freezable +@attrs.define(slots=False) +class TrainConfig: + # Master parameter dtype for FSDP. Activation / model dtype lives on the + # shared ParallelismConfig as policy.parallelism.precision. + master_dtype: str = "float32" + + # The data type for reduction in FSDP + fsdp_reduce_dtype: str = "float32" + + # Whether to offload the model to CPU if using FSDP + fsdp_offload: bool = False + + # Reshard the param after forward pass in FSDP + fsdp_reshard_after_forward: str = "default" + + # The batch size for training per iteration in one replica, this is the local batch size for each gradient accumulation step + train_batch_per_replica: int = 1 + + # The interval of train step for synchronizing weights between replicas. + sync_weight_interval: int = 1 + + # Train policy + train_policy: TrainPolicyConfig = TrainPolicyConfig() + fp8: FP8 = FP8() + deterministic: bool = False + + def key_values(self): + return {k: v for k, v in self.__dict__.items() if not k.startswith("_")} + + @property + def master_torch_dtype(self): + return { + "bfloat16": torch.bfloat16, + "float16": torch.float16, + "float32": torch.float32, + }[self.master_dtype] + + @property + def fsdp_reduce_torch_dtype(self): + return {"float32": torch.float32}[self.fsdp_reduce_dtype] + + +# Why we does not make this freezable? +# Because we need to path the cache model dir as backbone.model_name to the cosmos-rl model to use the +# model weights downloaded from s3. If cosmos-rl support reading model from s3 directly, we can make it freezable. +@attrs.define(slots=False) +class PolicyConfig: + # Parallelism configuration + parallelism: ParallelismConfig = ParallelismConfig() + + # Activation checkpointing policy for the VLM backbone. Only ``mode`` + # is consumed — see ActivationCheckpointingConfig. + activation_checkpointing: ActivationCheckpointingConfig = ActivationCheckpointingConfig() + + # VLM backbone identity, shared with OmniMoTModelConfig.vlm_config. + backbone: VLMConfig = VLMConfig() + # The maximum length for training, longer than this will be ignored for training stability + model_max_length: int = 16000 + + # The maximum length for video tokens, only applied to qwen model + qwen_max_video_token_length: int = 8000 + + # Extra model config + lora: Union[str, None] = None + enable_liger_kernel: bool = False + trainable_map: Union[str, None] = None + monkey_patch_for_text_only_data: bool = False + + # HF attention impl. Default "cosmos" routes through cosmos_framework.model.attention + # (NATTEN/blackwell-fmha on GB200). Override to "flash_attention_2", + # "sdpa", or "eager" for fallback. + attn_implementation: str = "cosmos" diff --git a/cosmos_framework/configs/base/vlm/defaults/vlm_policy.py b/cosmos_framework/configs/base/vlm/defaults/vlm_policy.py new file mode 100644 index 0000000..de2f535 --- /dev/null +++ b/cosmos_framework/configs/base/vlm/defaults/vlm_policy.py @@ -0,0 +1,155 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.configs.base.defaults.vlm import VLMConfig +from cosmos_framework.configs.base.vlm.defaults.training import PolicyConfig + +# Each entry replaces cfg.model.config.policy via package="model.config.policy". +# Sibling to the VFM vlm_config group at +# cosmos_framework/configs/base/defaults/vlm.py: that group binds VLMConfig +# SKUs onto OmniMoTModelConfig.vlm_config; this group binds PolicyConfig SKUs +# onto VLMModelConfig.policy. Both groups compose the same VLMConfig: VFM as +# vlm_config: VLMConfig, VLM as policy.backbone: VLMConfig. + +qwen2_5_vl_7b = PolicyConfig(backbone=VLMConfig(model_name="Qwen/Qwen2.5-VL-7B-Instruct")) + +eagle_er_1p7b = PolicyConfig( + backbone=VLMConfig(model_name="eagle_er_qwen3_1p7b_siglip_400m"), + model_max_length=16000, +) + +internvl3_5_1b = PolicyConfig( + backbone=VLMConfig(model_name="OpenGVLab/InternVL3_5-1B-HF"), + model_max_length=16000, # 40960 is the max length by default. +) + +internvl3_5_2b = PolicyConfig( + backbone=VLMConfig(model_name="OpenGVLab/InternVL3_5-2B-HF"), + model_max_length=16000, # 40960 is the max length by default. +) + +qwen3_vl_2b = PolicyConfig(backbone=VLMConfig(model_name="Qwen/Qwen3-VL-2B-Init")) + +qwen3_vl_30b_a3b_instruct = PolicyConfig(backbone=VLMConfig(model_name="Qwen/Qwen3-VL-30B-A3B-Instruct")) + +qwen3_vl_30b_a3b_thinking = PolicyConfig(backbone=VLMConfig(model_name="Qwen/Qwen3-VL-30B-A3B-Thinking")) + +qwen3_vl_235b_a22b_thinking = PolicyConfig(backbone=VLMConfig(model_name="Qwen/Qwen3-VL-235B-A22B-Thinking")) + +qwen3_vl_8b_thinking = PolicyConfig(backbone=VLMConfig(model_name="Qwen/Qwen3-VL-8B-Thinking")) + +qwen3_vl_8b_instruct = PolicyConfig(backbone=VLMConfig(model_name="Qwen/Qwen3-VL-8B-Instruct")) + +qwen3_vl_2b_instruct = PolicyConfig(backbone=VLMConfig(model_name="Qwen/Qwen3-VL-2B-Instruct")) + +qwen3_vl_2b_thinking = PolicyConfig(backbone=VLMConfig(model_name="Qwen/Qwen3-VL-2B-Thinking")) + +qwen3_vl_4b_instruct = PolicyConfig(backbone=VLMConfig(model_name="Qwen/Qwen3-VL-4B-Instruct")) + +qwen3_vl_4b_thinking = PolicyConfig(backbone=VLMConfig(model_name="Qwen/Qwen3-VL-4B-Thinking")) + +qwen3_vl_32b_instruct = PolicyConfig(backbone=VLMConfig(model_name="Qwen/Qwen3-VL-32B-Instruct")) + +nemotron_nano_12b_v2_vl_bf16 = PolicyConfig(backbone=VLMConfig(model_name="nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16")) + + +def register_vlm_policy(): + cs = ConfigStore.instance() + cs.store( + group="vlm_policy", + package="model.config.policy", + name="qwen2_5_vl_7b", + node=qwen2_5_vl_7b, + ) + cs.store( + group="vlm_policy", + package="model.config.policy", + name="eagle_er_1p7b", + node=eagle_er_1p7b, + ) + cs.store( + group="vlm_policy", + package="model.config.policy", + name="internvl3_5_1b", + node=internvl3_5_1b, + ) + cs.store( + group="vlm_policy", + package="model.config.policy", + name="internvl3_5_2b", + node=internvl3_5_2b, + ) + cs.store( + group="vlm_policy", + package="model.config.policy", + name="qwen3_vl_2b", + node=qwen3_vl_2b, + ) + cs.store( + group="vlm_policy", + package="model.config.policy", + name="qwen3_vl_30b_a3b_instruct", + node=qwen3_vl_30b_a3b_instruct, + ) + cs.store( + group="vlm_policy", + package="model.config.policy", + name="qwen3_vl_30b_a3b_thinking", + node=qwen3_vl_30b_a3b_thinking, + ) + cs.store( + group="vlm_policy", + package="model.config.policy", + name="qwen3_vl_235b_a22b_thinking", + node=qwen3_vl_235b_a22b_thinking, + ) + cs.store( + group="vlm_policy", + package="model.config.policy", + name="qwen3_vl_8b_thinking", + node=qwen3_vl_8b_thinking, + ) + cs.store( + group="vlm_policy", + package="model.config.policy", + name="qwen3_vl_8b_instruct", + node=qwen3_vl_8b_instruct, + ) + cs.store( + group="vlm_policy", + package="model.config.policy", + name="qwen3_vl_2b_instruct", + node=qwen3_vl_2b_instruct, + ) + cs.store( + group="vlm_policy", + package="model.config.policy", + name="qwen3_vl_2b_thinking", + node=qwen3_vl_2b_thinking, + ) + cs.store( + group="vlm_policy", + package="model.config.policy", + name="qwen3_vl_4b_instruct", + node=qwen3_vl_4b_instruct, + ) + cs.store( + group="vlm_policy", + package="model.config.policy", + name="qwen3_vl_4b_thinking", + node=qwen3_vl_4b_thinking, + ) + cs.store( + group="vlm_policy", + package="model.config.policy", + name="qwen3_vl_32b_instruct", + node=qwen3_vl_32b_instruct, + ) + cs.store( + group="vlm_policy", + package="model.config.policy", + name="nemotron_nano_12b_v2_vl_bf16", + node=nemotron_nano_12b_v2_vl_bf16, + ) diff --git a/cosmos_framework/configs/base/vlm/experiment/__init__.py b/cosmos_framework/configs/base/vlm/experiment/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/configs/base/vlm/experiment/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/configs/base/vlm/experiment/llava_ov_datapacker_experiment.py b/cosmos_framework/configs/base/vlm/experiment/llava_ov_datapacker_experiment.py new file mode 100644 index 0000000..ef9921a --- /dev/null +++ b/cosmos_framework/configs/base/vlm/experiment/llava_ov_datapacker_experiment.py @@ -0,0 +1,377 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""VLM training on lmms-lab/LLaVA-OneVision-Data via DataPackerDataLoader. + +Self-contained — inlines the Phase 2 VLMModel/FSDP2 base (formerly the +``pre_exp012_000_phase2_vlm_smoke_4gpu_8b`` smoke recipe in +``pre_exp012_phase2_vlm_smoke.py``) and replaces the dataloader with the +OSS-facing DataPackerDataLoader + VLMDataPacker pattern. Hydra defaults +below pin the VLM model (``vlm_fsdp`` / ``qwen3_vl_8b_instruct``), the +checkpoint backend, and callbacks. + +The dataset is loaded in streaming mode from the HuggingFace Hub so no local +download is required. Each record is converted from ShareGPT conversation +format to the OpenAI message format expected by Qwen3-VL's processor, then +tokenized in the DataLoader worker via ``processor.apply_chat_template``. + +Resume semantics +---------------- +The streaming HF dataset is a ``datasets.IterableDataset``, which +``DataPackerDataLoader`` flags with ``_has_dp_meta=False`` (see +``data_packer_dataloader.py:317-321`` — "Stateful resume is not supported for +IterableDataset sources"). On checkpoint save the dataloader shard stores +placeholder ``(epoch=0, index=0)`` per worker — VLMDataPacker.sft_collate_fn +stamps these zeros explicitly because the stream has no meaningful position +to record. When resuming with ``checkpoint.load_training_state=true``: + + - model / optim / scheduler / trainer state restore correctly (iter + counter, optimizer momentum, LR schedule position all continue). + - dataloader stream position does NOT restore; the streamed dataset + re-yields from the beginning, so the first N resumed iters see the + same samples as the first N iters of the original run. + +For a true position-stateful resume, swap the data_source to a map-style +dataset (``load_dataset(..., streaming=False)``). + +Usage (smoke test):: + + torchrun --nproc_per_node=4 --master_port=12344 -m cosmos_framework.scripts.train \\ + --config=cosmos_framework/configs/base/vlm/config.py -- \\ + experiment=pre_exp012_llava_ov_datapacker \\ + "model.config.policy.backbone.model_name=/path/to/Siglip2-Qwen3-1.7B-BF16-Alignment" \\ + trainer.max_iter=10 trainer.logging_iter=1 \\ + job.wandb_mode=disabled ckpt_type=dummy + +See ``launch_vlm_llava_ov.sh`` for a ready-to-run shell script. +""" + +from __future__ import annotations + +from typing import Any + +from hydra.core.config_store import ConfigStore + +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils.lazy_config import LazyDict, instantiate +from cosmos_framework.data.vfm.data_packer import DataPacker +from cosmos_framework.data.vfm.data_packer_dataloader import DataPackerDataLoader +from cosmos_framework.data.vfm.processors import build_processor +from cosmos_framework.utils.vlm.constant import IGNORE_INDEX, PROCESSOR_KEYS_TO_ADD + +cs = ConfigStore.instance() + + +# --------------------------------------------------------------------------- +# LLaVA-OneVision-Data source factory +# +# Loads lmms-lab/LLaVA-OneVision-Data in streaming mode so no local download +# is needed. streaming=True returns an IterableDataset which DataPackerDataLoader +# wraps directly. +# --------------------------------------------------------------------------- + + +def build_vlm_datapacker_dataloader(**kwargs) -> "DataPackerDataLoader": + """Thin wrapper around DataPackerDataLoader that drops schema keys injected by + OmegaConf when the parent experiment's VLMRecipeDataLoader schema merges with + our DataPackerDataLoader config (e.g. ``storage_type``). + """ + for _spurious in ("storage_type",): + kwargs.pop(_spurious, None) + return DataPackerDataLoader(**kwargs) + + +def get_llava_ov_streaming( + subset: str = "si", + split: str = "train", +) -> Any: + """Load lmms-lab/LLaVA-OneVision-Data as a streaming HuggingFace IterableDataset. + + Args: + subset: Dataset config/subset name. ``"si"`` (single-image, ~1M samples) + is the standard choice; pass any valid config name from the Hub. + split: Dataset split (default ``"train"``). + + Returns: + A streaming ``datasets.IterableDataset`` whose items have keys: + ``id``, ``image`` (PIL.Image), ``conversations`` (ShareGPT format). + """ + try: + from datasets import load_dataset + except ImportError as exc: + raise ImportError("pip install datasets to use lmms-lab/LLaVA-OneVision-Data") from exc + + ds = load_dataset( + "lmms-lab/LLaVA-OneVision-Data", + name=subset, + split=split, + streaming=True, + ) + # Pre-filter to remove records without an image or conversations so + # sft_process_sample never receives unparseable samples (DataPacker's + # packing engine does not tolerate None returns from sft_process_sample). + return ds.filter(lambda x: x.get("image") is not None and len(x.get("conversations") or []) >= 2) + + +# --------------------------------------------------------------------------- +# VLMDataPacker +# +# Bridges lmms-lab/LLaVA-OneVision-Data (ShareGPT format) into the +# VLMModel training loop. +# +# Three-step pipeline per sample: +# 1. Convert ShareGPT (from/value) → OpenAI messages (role/content). +# 2. Apply processor.apply_chat_template → input_ids, pixel_values, etc. +# 3. Build labels by masking non-assistant tokens with IGNORE_INDEX. +# --------------------------------------------------------------------------- + + +class VLMDataPacker(DataPacker): + """DataPacker adapter for lmms-lab/LLaVA-OneVision-Data + Qwen3-VL processor. + + Converts ShareGPT-format image+conversation samples into the + ``input_ids / labels / pixel_values / image_grid_thw`` batch dict that + ``VLMModel.training_step`` expects. + + Designed for ``max_batch_size=1`` — each packed batch is a single sample. + The ``sft_collate_fn`` adds a leading batch dimension to 1-D tensors + (``input_ids``, ``labels``, ``attention_mask``) while leaving + ``pixel_values`` and ``image_grid_thw`` in their native flat shapes, + matching what Qwen3-VL's forward pass expects. + """ + + def __init__( + self, + tokenizer_config: Any, + max_seq_len: int = 16000, + ignore_index: int = IGNORE_INDEX, + ) -> None: + self._max_seq_len = max_seq_len + self._ignore_index = ignore_index + # Instantiate if tokenizer_config is a Hydra LazyCall; use directly if already built. + self._processor = ( + tokenizer_config if hasattr(tokenizer_config, "apply_chat_template") else instantiate(tokenizer_config) + ) + + # ------------------------------------------------------------------ + # Internal helpers + # ------------------------------------------------------------------ + + @staticmethod + def _decode_image(image: Any) -> Any: + """Decode a HuggingFace streaming image to PIL. + + In streaming mode HuggingFace delivers images as + ``{"bytes": bytes, "path": str}`` dicts rather than decoded PIL Images. + """ + if isinstance(image, dict): + import io + + from PIL import Image + + raw = image.get("bytes") + if raw: + return Image.open(io.BytesIO(raw)).convert("RGB") + path = image.get("path") + if path: + return Image.open(path).convert("RGB") + return None + return image + + def _sharegpt_to_openai(self, item: dict) -> list[dict]: + """Convert ShareGPT conversation to OpenAI message format. + + LLaVA-OneVision-Data records use ``from``/``value`` pairs where the + human turn may contain a ```` placeholder. We strip the + placeholder and attach the PIL image as a separate content block. + """ + conversations = item.get("conversations", []) + image = self._decode_image(item.get("image")) # PIL.Image or None + messages: list[dict] = [] + image_inserted = False + + for turn in conversations: + role = "user" if turn["from"] == "human" else "assistant" + text = turn["value"].replace("", "").strip() + + if role == "user" and not image_inserted and image is not None: + content: Any = [ + {"type": "image", "image": image}, + {"type": "text", "text": text}, + ] + image_inserted = True + else: + content = text + + messages.append({"role": role, "content": content}) + + return messages + + # ------------------------------------------------------------------ + # DataPacker protocol + # ------------------------------------------------------------------ + + def sft_process_sample(self, item: dict) -> dict: + """Convert one LLaVA-OV record to VLM training tensors.""" + messages = self._sharegpt_to_openai(item) + inputs = self._processor.apply_chat_template( + messages, + tokenize=True, + add_generation_prompt=False, + ) + input_ids = inputs["input_ids"] # [N] + + token_mask = self._processor.add_assistant_tokens_mask(input_ids) # [N] bool + labels = input_ids.clone() # [N] + labels[~token_mask] = self._ignore_index + + result: dict = { + "input_ids": input_ids, + "labels": labels, + } + for key in PROCESSOR_KEYS_TO_ADD: + if key in inputs and inputs[key] is not None: + result[key] = inputs[key] + + return result + + def compute_num_tokens(self, sample: dict) -> int: + """Token count = sequence length (input_ids).""" + return int(sample["input_ids"].shape[0]) # [N] → scalar + + def sft_collate_fn( + self, + samples: list[dict], + max_len: int, + ignore_label_id: int = IGNORE_INDEX, + ) -> dict: + """Assemble one VLM training batch. + + Designed for ``max_batch_size=1``. 1-D sequence tensors get an + unsqueezed batch dimension; ``pixel_values`` / ``image_grid_thw`` + stay in the flat format Qwen3-VL expects. + """ + assert len(samples) == 1, f"VLMDataPacker expects max_batch_size=1, got {len(samples)}" + s = samples[0] + + import torch + + worker_info = torch.utils.data.get_worker_info() + worker_id = worker_info.id if worker_info is not None else 0 + + batch: dict = { + "input_ids": s["input_ids"].unsqueeze(0), # [1,N] + "labels": s["labels"].unsqueeze(0), # [1,N] + "sample_worker_id": torch.tensor([worker_id]), # [1] + "sample_epoch": torch.tensor([0]), # [1] streaming has no epoch concept + "sample_index": torch.tensor([0]), # [1] streaming has no global index + } + + if "attention_mask" in s and s["attention_mask"] is not None: + batch["attention_mask"] = s["attention_mask"].unsqueeze(0) # [1,N] + + # Vision tensors: pixel_values [P,C] and image_grid_thw [1,3] stay flat. + for key in ("pixel_values", "pixel_values_videos", "image_grid_thw", "video_grid_thw", "second_per_grid_ts"): + if key in s and s[key] is not None: + batch[key] = s[key] + + return batch + + +# --------------------------------------------------------------------------- +# Experiment registration +# --------------------------------------------------------------------------- + + +pre_exp012_llava_ov_datapacker = LazyDict( + dict( + # Hydra defaults — inlined from the former pre_exp012_000_phase2_vlm_smoke_4gpu_8b + # smoke recipe. data_train/data_val intentionally omitted because the + # dataloader_train below is a self-contained DataPackerDataLoader; pulling in + # the smoke's s3 webdataset defaults would let storage_type schema bleed into + # our DataPackerDataLoader config. + defaults=[ + {"override /checkpoint": "s3"}, + {"override /model": "vlm_fsdp"}, + {"override /vlm_policy": "qwen3_vl_8b_instruct"}, + {"override /callbacks": ["basic_vlm", "basic_log"]}, + "_self_", + ], + job=dict( + name="pre_exp012_llava_ov_datapacker_${now:%Y-%m-%d}_${now:%H-%M-%S}", + group="vlm_llava_ov_demo", + ), + trainer=dict( + max_iter=10, + logging_iter=1, + run_validation=False, + ), + optimizer=dict( + lr=1e-5, + fused=True, + ), + model=dict( + config=dict( + # Phase 2 requires a trainable_params regex; ".*" = full fine-tune. + freeze=dict( + trainable_params=[".*"], + ), + policy=dict( + parallelism=dict( + data_parallel_shard_degree=4, + data_parallel_replicate_degree=-1, + ), + ), + ), + ), + # Local-only mode: disable the parent's object-store IO and clear the + # S3 credentials/bucket so maybe_download_hf_model_from_s3 falls back + # to HuggingFace Hub (avoids opening credentials/s3_training.secret in + # OSS smoke runs). Pattern mirrors vision_sft_nano.py. + checkpoint=dict( + # Don't save checkpoints during smoke runs. + save_iter=100000, + load_from_object_store=dict(enabled=False, credentials="", bucket=""), + save_to_object_store=dict(enabled=False, credentials="", bucket=""), + ), + # Replace the S3 WebDataset-based dataloader with DataPackerDataLoader + # pointing at lmms-lab/LLaVA-OneVision-Data streamed from HuggingFace Hub. + dataloader_train=L(build_vlm_datapacker_dataloader)( + data_source=L(get_llava_ov_streaming)( + subset="ai2d(gpt4v)", + split="train", + ), + data_packer=L(VLMDataPacker)( + tokenizer_config=L(build_processor)( + tokenizer_type="${model.config.policy.backbone.model_name}", + # OSS smoke mode: route the processor download through the + # HF Hub fallback rather than the S3 default (which would + # try to open credentials/s3_training.secret). + config_variant="hf", + ), + max_seq_len="${dataloader_train.max_tokens}", + ignore_index=IGNORE_INDEX, + ), + max_tokens=16000, + max_batch_size=1, + pool_size=16, + num_workers=2, + prefetch_factor=2, + persistent_workers=True, + pin_memory=True, + ), + dataloader_val=None, + # Suppress S3 uploads in callbacks (iter_speed.save_s3, param_count.save_s3, + # wandb_*.save_s3 all interpolate from ${upload_reproducible_setup}). Mirrors + # the VFM SFT experiments under cosmos/configs/base/experiment/sft/. + upload_reproducible_setup=False, + ), + flags={"allow_objects": True}, +) + +cs.store( + group="experiment", + package="_global_", + name="pre_exp012_llava_ov_datapacker", + node=pre_exp012_llava_ov_datapacker, +) diff --git a/cosmos_framework/configs/base/vlm/experiment/utils.py b/cosmos_framework/configs/base/vlm/experiment/utils.py new file mode 100644 index 0000000..78fbd95 --- /dev/null +++ b/cosmos_framework/configs/base/vlm/experiment/utils.py @@ -0,0 +1,16 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from dataclasses import dataclass, field +from typing import Dict, List + + +@dataclass +class Experiment: + job_exp: str + nnode: int + command_args: List[str] + job_name: str = None + init_command: str = "" + job_group: str = None + extra_env_vars: Dict[str, str] = field(default_factory=dict) diff --git a/cosmos_framework/configs/base/vlm/freeze_config.py b/cosmos_framework/configs/base/vlm/freeze_config.py new file mode 100644 index 0000000..9629782 --- /dev/null +++ b/cosmos_framework/configs/base/vlm/freeze_config.py @@ -0,0 +1,33 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 +"""VLM freeze config (read by ``vlm_model._apply_freeze_config``).""" + +import attrs + +from cosmos_framework.utils.config import make_freezable + + +@make_freezable +@attrs.define(slots=False) +class VLMFreezeConfig: + """Selects which parts of a VLM stay trainable. + + Applied at model construction, before the optimizer is built; the optimizer + only sees the resulting ``requires_grad`` state. + """ + + # Named freeze flags. Supported architectures: Qwen2.5-VL, + # Qwen3-VL (dense + MoE), InternVL3_5. + freeze_vision_encoder: bool = False + freeze_mm_projector: bool = False + freeze_llm: bool = False + + # Regex-based freeze (mutually exclusive with each other). + # trainable_params: whitelist — only matching params are trainable. + # frozen_params: blacklist — matching params get frozen. + trainable_params: list[str] | None = None + frozen_params: list[str] | None = None + + def __attrs_post_init__(self) -> None: + if self.trainable_params is not None and self.frozen_params is not None: + raise ValueError("VLMFreezeConfig: set at most one of trainable_params or frozen_params, not both.") diff --git a/cosmos_framework/configs/toml_config/__init__.py b/cosmos_framework/configs/toml_config/__init__.py new file mode 100644 index 0000000..28a81be --- /dev/null +++ b/cosmos_framework/configs/toml_config/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 diff --git a/cosmos_framework/configs/toml_config/sft_config.py b/cosmos_framework/configs/toml_config/sft_config.py new file mode 100644 index 0000000..5bcbf23 --- /dev/null +++ b/cosmos_framework/configs/toml_config/sft_config.py @@ -0,0 +1,719 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""TOML-readable pydantic schema for SFT experiment knobs. + +Sibling of ``toml_config_helper.py`` — this file holds the pydantic models +(the schema the TOML must conform to). All conversion logic (TOML → +override list, ``PATH_REMAPS``, etc.) lives in ``toml_config_helper.py``. +""" + +from __future__ import annotations + +import tomllib +from pathlib import Path +from typing import Any, Optional + +from pydantic import BaseModel, ConfigDict, Field + +from cosmos_framework.configs.toml_config.toml_config_helper import ( + TASK_TO_BASE_CONFIG, + build_hydra_overrides, +) + + +# Common config for every model in this file: +# - ``extra="forbid"`` → unknown TOML keys raise ValidationError (typo guard). +# - ``protected_namespaces=()`` → silence the ``model_*`` field-name warning so +# the ``model:`` field on SFTExperimentConfig is allowed. +_PYDANTIC_MODEL_CONFIG = ConfigDict(extra="forbid", protected_namespaces=()) + + +# ---------------------------------------------------------------- job +class JobConfig(BaseModel): + """Run identity + meta-fields that pick the Hydra config tree to load.""" + + model_config = _PYDANTIC_MODEL_CONFIG + + task: str = Field( + default="vfm", + description=( + "META — chooses which make_config() to call: " + "'vfm' → cosmos_framework/configs/base/config.py (video foundation model), " + "'vlm' → cosmos_framework/configs/base/vlm/config.py (vision-language model). " + "Also picks the path-remap rules in toml_config_helper.PATH_REMAPS." + ), + ) + experiment: str = Field( + default="", + description=( + "META — names the Hydra experiment LazyDict registered in " + "ConfigStore under experiment/. Resolved at load time via " + "the 'experiment=' Hydra-CLI override " + "(e.g. 'action_fdm_sft_nano')." + ), + ) + project: str = Field( + default="", + description=( + "Wandb project (team-level bucket). Flows to config.job.project " + "and is what shows up under 'Projects' in the wandb UI." + ), + ) + group: str = Field( + default="", + description=( + "Wandb group — sub-label under for clustering related " + "runs (e.g. 'sft', 'action_bridge'). Flows to config.job.group." + ), + ) + name: str = Field( + default="", + description=( + "Wandb run name. Flows to config.job.name and forms part of the " + "output-dir path: $IMAGINAIRE_OUTPUT_ROOT///. " + "Leave empty (or use Hydra ${now:%Y-%m-%d}_${now:%H-%M-%S}) to " + "get an auto-timestamped subdir." + ), + ) + wandb_mode: str = Field( + default="disabled", + description=( + "Wandb upload mode: 'online' (real-time, needs WANDB_API_KEY), " + "'offline' (log locally, sync later with `wandb sync`), or " + "'disabled' (no wandb at all)." + ), + ) + + +# ---------------------------------------------------------------- model +class EMAConfig(BaseModel): + """Exponential Moving Average of the generation-pathway weights. + + Lands at ``model.config.ema.*`` on both VFM and VLM. When enabled the + trainer keeps a second fp32 copy of the trainable params updated as + ``ema_w = (1 - rate^k) · w_curr + rate^k · ema_w_prev``. EMA weights + are used for inference; the live weights keep training. + """ + + model_config = _PYDANTIC_MODEL_CONFIG + + enabled: bool = Field( + default=True, + description=( + "Turn EMA tracking on/off. Full fine-tunes typically enable it; " + "LoRA recipes leave it off because the adapter weights are tiny." + ), + ) + rate: float = Field( + default=0.1, + description=( + "Base EMA decay rate. Lower = slower decay = EMA tracks the live " + "weights more tightly. Effective per-step rate is ramped by the " + "iteration counter so the EMA 'warms up' from init." + ), + ) + iteration_shift: int = Field( + default=0, + description=( + "Step offset added before computing the warmup ramp. Use a " + "positive value when resuming so the EMA doesn't reset to " + "'early-iter' decay strength." + ), + ) + + +class ParallelismConfig(BaseModel): + """FSDP / context-parallel / classifier-free-guidance topology. + + Lands at ``model.config.parallelism.*`` on VFM and at + ``model.config.policy.parallelism.*`` on VLM (see PATH_REMAPS). + """ + + model_config = _PYDANTIC_MODEL_CONFIG + + use_torch_compile: bool = Field( + default=False, + description=( + "torch.compile the network. Big speedup on stable shapes; " + "conflicts with some custom CUDA kernels and deterministic modes." + ), + ) + compile_dynamic: bool = Field( + default=True, + description=( + "When use_torch_compile=True, recompile per-shape rather than " + "specializing for one static shape. Required for the " + "compile_tokenizer callback's progressive warmup." + ), + ) + data_parallel_shard_degree: int = Field( + default=-1, + description=( + "FSDP shard degree. -1 = auto-fit WORLD_SIZE from torchrun. " + "Set explicitly when you want the run to fail loudly on the " + "wrong GPU count." + ), + ) + data_parallel_replicate_degree: int = Field( + default=1, + description=( + "FSDP replicate degree (HSDP). >1 adds an outer replicate loop " + "so the same shard topology runs N times in parallel; usually " + "only needed for very large clusters." + ), + ) + context_parallel_shard_degree: int = Field( + default=1, + description=( + "Context-parallel shard degree. >1 splits the sequence dimension " + "across this many ranks, which lets long-context models fit in " + "memory. Used by super-tier configs (DP=4, CP=2 → 8 GPUs)." + ), + ) + cfg_parallel_shard_degree: int = Field( + default=1, + description=( + "Classifier-free-guidance parallel shard degree. Splits the " + "duplicated conditional/unconditional forward across ranks. " + "Almost always 1 for SFT." + ), + ) + precision: str = Field( + default="bfloat16", + description=( + "Compute dtype for the network forward/backward. 'bfloat16' is " + "standard for Hopper/Blackwell. Master weights live in fp32 " + "separately (see TrainConfig on VLM)." + ), + ) + + +class ActivationCheckpointingConfig(BaseModel): + """Recompute activations during backward to trade FLOPs for memory. + + Lands at ``model.config.activation_checkpointing.*`` (VFM) or + ``model.config.policy.activation_checkpointing.*`` (VLM). + """ + + model_config = _PYDANTIC_MODEL_CONFIG + + mode: str = Field( + default="full", + description=( + "AC mode: 'selective' (per-op SAC; save matmuls/FMHA, recompute " + "the rest — MoT path only), 'full' (checkpoint each whole " + "transformer block), or 'none' (no checkpointing — fastest but " + "highest memory)." + ), + ) + save_ops_regex: list[str] = Field( + default_factory=lambda: ["fmha"], + description=( + "Regex patterns for ops to KEEP saved when mode='selective'. " + "Ignored in 'full'/'none' mode. Default keeps flash/multi-head-" + "attention outputs." + ), + ) + preserve_rng_state: bool = Field( + default=True, + description=( + "Stash and restore CUDA RNG across recompute boundaries. Required " + "for deterministic results vs. non-checkpointed runs; small slowdown." + ), + ) + determinism_check: str = Field( + default="default", + description=( + "Forwarded to torch.utils.checkpoint. 'default' disables the " + "extra determinism check; 'match' cross-checks recomputed " + "activations against the original (debug-only, very slow)." + ), + ) + + +class ModelTokenizerConfig(BaseModel): + """Video tokenizer (VAE) settings. VFM only — VLM skips this sub-tree.""" + + model_config = _PYDANTIC_MODEL_CONFIG + + vae_path: str = Field( + default="pretrained/tokenizers/video/wan2pt2/Wan2.2_VAE.pth", + description=( + "Path to Wan2.2_VAE.pth. SFT recipes typically pass this via " + "env interpolation: vae_path = '${oc.env:WAN_VAE_PATH}'." + ), + ) + + +class BackboneConfig(BaseModel): + """Foundation backbone settings. VLM only — VFM keeps its backbone + wiring inline in the experiment Python (vlm_config.model_instance) + and skips this sub-tree. + """ + + model_config = _PYDANTIC_MODEL_CONFIG + + model_name: str = Field( + default="???", + description=( + "HF repo ID or local snapshot path of the VLM backbone " + "(e.g. 'Qwen/Qwen3-VL-8B-Instruct'). Drives AutoConfig + " + "AutoModel selection (architecture). Remapped to " + "'model.config.policy.backbone.model_name' on VLM; skipped on " + "VFM. Default '???' is the OmegaConf MISSING sentinel — " + "recognized by build_hydra_overrides and skipped, so the " + "experiment Python's default takes effect when the TOML omits " + "[model.backbone]." + ), + ) + safetensors_path: str = Field( + default="???", + description=( + "Optional local path to a .safetensors file (or directory) used " + "for weight loading. When set, overrides the auto-downloaded " + "snapshot under model_name; the architecture is still driven by " + "model_name. Useful for pointing at a converted/finetuned " + "checkpoint while keeping the public HF model_name for tokenizer " + "and architecture discovery. Remapped to " + "'model.config.policy.backbone.safetensors_path' on VLM; " + "skipped on VFM. Default '???' = MISSING sentinel (omitted from " + "overrides; falls back to '' from VLMConfig, which means " + "'use the auto-downloaded model_name snapshot')." + ), + ) + + +class ModelConfig(BaseModel): + """Top-level model knobs. + + Lands at ``model.config.*`` on VFM and on VLM, with sub-tree paths + remapped per PATH_REMAPS (parallelism / activation_checkpointing nest + under ``.policy.*`` on VLM). + """ + + model_config = _PYDANTIC_MODEL_CONFIG + + max_num_tokens_after_packing: int = Field( + default=13312, + description=( + "Token-packing target: max number of tokens after sequence " + "packing. -1 disables the cap. VFM-only — VLM uses " + "data_setting.max_tokens and policy.qwen_max_video_token_length." + ), + ) + joint_attn_implementation: str = Field( + default="two_way", + description=( + "VFM attention layout: 'two_way' (separate U/G blocks with " + "cross-attention), 'three_way' (adds a sparsity-aware third " + "block — NATTEN), or 'flex' (legacy). Used when " + "[job].task='vfm'; skipped on VLM." + ), + ) + attn_implementation: str = Field( + default="cosmos", + description=( + "VLM HF attention impl: 'cosmos' (cosmos NATTEN/Blackwell-FMHA " + "wrapper), 'flash_attention_2' (HF flash-attn-2), 'sdpa' " + "(torch SDPA), or 'eager' (pure-python fallback). Used when " + "[job].task='vlm'; skipped on VFM." + ), + ) + lora_enabled: bool = Field( + default=False, + description=( + "Inject LoRA adapters into the generation pathway BEFORE FSDP " + "wraps the network. Pair with optimizer.keys_to_select=['lora_'] " + "(train only adapters) and checkpoint.keys_to_skip_loading=[" + "..., 'lora_'] (don't load missing adapter tensors). Used by " + "SUPER-tier configs (e.g. vision_sft_super); NANO-tier leaves " + "it off. Skipped on VLM." + ), + ) + lora_rank: int = Field( + default=16, + description=( + "LoRA rank `r`. Adapter shape is (rank × hidden_dim) per target " + "module. Standard values are 4, 8, 16, 32." + ), + ) + lora_alpha: int = Field( + default=32, + description=( + "LoRA scaling factor. Effective magnitude of the adapter update " + "is alpha/rank; rank=16 alpha=32 gives a 2× scale." + ), + ) + lora_target_modules: str = Field( + default="q_proj_moe_gen,k_proj_moe_gen,v_proj_moe_gen,o_proj_moe_gen", + description=( + "Comma-separated substrings of param names that get a LoRA " + "adapter. Defaults target the four MoE-gen projection matrices." + ), + ) + + ema: EMAConfig = Field(default_factory=EMAConfig) + parallelism: ParallelismConfig = Field(default_factory=ParallelismConfig) + activation_checkpointing: ActivationCheckpointingConfig = Field( + default_factory=ActivationCheckpointingConfig + ) + tokenizer: ModelTokenizerConfig = Field(default_factory=ModelTokenizerConfig) + backbone: BackboneConfig = Field(default_factory=BackboneConfig) + + +# ---------------------------------------------------------------- optimizer +class OptimizerConfig(BaseModel): + """AdamW-family optimizer parameters. Same shape on VFM and VLM (eps skipped on VLM).""" + + model_config = _PYDANTIC_MODEL_CONFIG + + betas: list[float] = Field( + default_factory=lambda: [0.9, 0.99], + description=( + "Adam β1, β2 — gradient and squared-gradient EMAs. Standard pair " + "is (0.9, 0.999); SFT recipes commonly use (0.9, 0.99) or " + "(0.9, 0.95) for tighter tracking of recent gradients." + ), + ) + eps: float = Field( + default=1.0e-8, + description=( + "Adam numerical stability epsilon. 1e-8 is the PyTorch default; " + "1e-6 is sometimes used in bf16 to avoid underflow in the " + "squared-gradient denominator. Skipped on VLM (no eps field)." + ), + ) + fused: bool = Field( + default=True, + description=( + "Use the fused AdamW kernel. Faster on modern GPUs; slightly " + "different numerical behavior vs. the foreach implementation." + ), + ) + keys_to_select: list[str] = Field( + default_factory=list, + description=( + "Substring allowlist for params that the optimizer trains. " + "Empty list = train everything. ['lora_'] = LoRA-only fine-tune " + "(freezes everything except adapters)." + ), + ) + lr: float = Field( + default=2.0e-4, + description="Base learning rate.", + ) + lr_multipliers: dict[str, float] = Field( + default_factory=dict, + description=( + "Per-param-group LR multipliers (substring → multiplier). Used " + "by action recipes to e.g. give 'action_modality_embed' 5× the " + "base lr. Substrings not in the dict default to 1.0." + ), + ) + weight_decay: float = Field( + default=0.0, + description="AdamW decoupled weight decay. 0 disables.", + ) + + +# ---------------------------------------------------------------- scheduler +class SchedulerConfig(BaseModel): + """LambdaLinear / LambdaCosine LR scheduler knobs. + + All four ``f_*`` values are **ratios of the optimizer's base lr** — + effective lr at the corresponding milestone = ``lr × f_x``. Each list + has one entry per scheduler cycle. + """ + + model_config = _PYDANTIC_MODEL_CONFIG + + cycle_lengths: list[int] = Field( + default_factory=lambda: [20000], + description=( + "Length of each cycle in optimizer steps. With one entry, the " + "scheduler completes one full warmup→peak→trough cycle over " + "that many iterations." + ), + ) + f_max: list[float] = Field( + default_factory=lambda: [1.0], + description="Peak LR multiplier reached at the end of warmup.", + ) + f_min: list[float] = Field( + default_factory=lambda: [0.0], + description=( + "Final LR multiplier at the end of each cycle (the 'floor'). " + "For LambdaCosine the LR decays toward lr × f_min." + ), + ) + f_start: list[float] = Field( + default_factory=lambda: [1.0e-6], + description=( + "Initial LR multiplier at step 0, before warmup ramps up." + ), + ) + verbosity_interval: int = Field( + default=0, + description=( + "How often the scheduler logs the current LR (in optimizer " + "steps). 0 = silent. VFM only — skipped on VLM." + ), + ) + warm_up_steps: list[int] = Field( + default_factory=lambda: [100], + description=( + "Linear warmup duration in optimizer steps. LR ramps from " + "lr × f_start to lr × f_max linearly over this many iters " + "before the cosine/linear decay begins." + ), + ) + + +# ---------------------------------------------------------------- trainer +class CompileTokenizerCallback(BaseModel): + """Lazy ``torch.compile`` of the VAE tokenizer once shapes stabilize. + + VFM only — skipped on VLM (no tokenizer to compile). + """ + + model_config = _PYDANTIC_MODEL_CONFIG + + compile_after_iterations: int = Field( + default=3, + description=( + "Wait this many training iterations after start before triggering " + "the compile (lets one-shot init / dataloader settle)." + ), + ) + enabled: bool = Field( + default=True, + description="Master switch for the callback.", + ) + warmup_resolutions: Optional[list[str]] = Field( + default=None, + description=( + "Resolutions to 'prime' the compile cache with. The callback " + "runs the tokenizer once per listed resolution so the compiled " + "graph for each is ready before training hits it. None = use " + "whatever resolutions the tokenizer's encode_chunk_frames knows." + ), + ) + + +class GradClipCallback(BaseModel): + """Gradient clipping callback. Present on both VFM and VLM.""" + + model_config = _PYDANTIC_MODEL_CONFIG + + clip_norm: float = Field( + default=1.0, + description=( + "Maximum global L2 norm of the gradient. Steps with a larger " + "norm are rescaled so ||grad|| ≤ clip_norm." + ), + ) + force_finite: bool = Field( + default=True, + description=( + "When True, replace NaN/Inf grads with zero before the step " + "(treats them as no-op rather than crashing). VFM defaults to " + "True; VLM defaults to False." + ), + ) + + +class TrainerCallbacksConfig(BaseModel): + """Only the two callbacks the schema currently surfaces. The full + callbacks dict (norm_monitor, mfu, heart_beat, …) stays in the + experiment Python. + """ + + model_config = _PYDANTIC_MODEL_CONFIG + + compile_tokenizer: CompileTokenizerCallback = Field(default_factory=CompileTokenizerCallback) + grad_clip: GradClipCallback = Field(default_factory=GradClipCallback) + + +class TrainerConfig(BaseModel): + """Trainer-level knobs that the TOML drives directly.""" + + model_config = _PYDANTIC_MODEL_CONFIG + + distributed_parallelism: str = Field( + default="fsdp", + description=( + "Distributed strategy. 'fsdp' (the only supported value today) " + "routes through cosmos's FSDP wrapper." + ), + ) + grad_accum_iter: int = Field( + default=1, + description=( + "Number of micro-batches accumulated before each " + "optimizer.step(). Effective global batch = grad_accum_iter × " + "per-rank batch × world_size." + ), + ) + logging_iter: int = Field( + default=50, + description="Console / wandb log frequency (in optimizer steps).", + ) + max_iter: int = Field( + default=500, + description="Total number of optimizer steps the run will execute.", + ) + callbacks: TrainerCallbacksConfig = Field(default_factory=TrainerCallbacksConfig) + + +# ---------------------------------------------------------------- checkpoint +class CheckpointConfig(BaseModel): + """Resume + save policy. Lands at ``config.checkpoint.*``.""" + + model_config = _PYDANTIC_MODEL_CONFIG + + keys_to_skip_loading: list[str] = Field( + default_factory=list, + description=( + "Substring blocklist applied at load time. Any tensor whose FQN " + "contains one of these substrings is skipped (kept at fresh-" + "init). Used to mask EMA + LoRA + action layers when " + "warm-starting from a base checkpoint without them." + ), + ) + load_path: str = Field( + default="???", + description=( + "Path to the checkpoint directory to load. '???' is the " + "OmegaConf MISSING sentinel — recognized by build_hydra_overrides " + "and skipped, so the user must provide a real path at runtime " + "(via env interpolation or CLI extra-override)." + ), + ) + save_iter: int = Field( + default=100, + description="Save a new checkpoint every N optimizer steps.", + ) + + +# ---------------------------------------------------------------- dataloader_train +class DataloaderTrainConfig(BaseModel): + """Top-level dataloader scalars only. The dataloader's class (LazyCall) + and full pipeline wiring (datasets, packers, …) stay in the experiment + Python — they vary too much between VFM IterativeJointDataLoader, + PackingDataLoader, and VLM DataPackerDataLoader to model uniformly. + """ + + model_config = _PYDANTIC_MODEL_CONFIG + + max_samples_per_batch: Optional[int] = Field( + default=None, + description=( + "Cap on samples per micro-batch. Remapped to 'max_batch_size' " + "on the VLM DataPackerDataLoader. None = no per-count cap " + "(the packer's token budget is what limits batch size)." + ), + ) + max_sequence_length: Optional[int] = Field( + default=None, + description=( + "Cap on tokens per packed sequence. Remapped to 'max_tokens' " + "on the VLM DataPackerDataLoader. None = no per-token cap." + ), + ) + seed: int = Field( + default=42, + description=( + "Dataloader RNG seed. Skipped on VLM (DataPackerDataLoader has " + "no seed ctor kwarg there)." + ), + ) + + +# ---------------------------------------------------------------- top +class SFTExperimentConfig(BaseModel): + """Top-level structured-TOML schema. Each field corresponds to a + top-level ``[
]`` block in the TOML and to a sub-model above. + """ + + model_config = _PYDANTIC_MODEL_CONFIG + + job: JobConfig = Field(default_factory=JobConfig) + model: ModelConfig = Field(default_factory=ModelConfig) + optimizer: OptimizerConfig = Field(default_factory=OptimizerConfig) + scheduler: SchedulerConfig = Field(default_factory=SchedulerConfig) + trainer: TrainerConfig = Field(default_factory=TrainerConfig) + checkpoint: CheckpointConfig = Field(default_factory=CheckpointConfig) + dataloader_train: DataloaderTrainConfig = Field(default_factory=DataloaderTrainConfig) + + +# --------------------------------------------------------------------------- +# End-to-end loader: TOML → validate → Hydra overrides → merged Config. +# --------------------------------------------------------------------------- +def load_experiment_from_toml( + toml_path: str | Path, + extra_overrides: list[str] | None = None, +) -> Any: + """End-to-end loader for the SFT structured-TOML schema. + + The base config module is picked from ``[job].task`` in the TOML: + + - ``task = "vfm"`` → ``cosmos_framework/configs/base/config.py`` + - ``task = "vlm"`` → ``cosmos_framework/configs/base/vlm/config.py`` + + ``extra_overrides`` is appended after the TOML-derived Hydra overrides, so + command-line entries take precedence over TOML values. Each entry must be + Hydra dotted-path syntax (``key.path=value``); the ``--`` separator token + is filtered out. Examples:: + + ["optimizer.lr=1e-5", "trainer.max_iter=200"] + ["model.config.parallelism.data_parallel_shard_degree=4"] + + Calls ``cosmos_framework.utils.config.load_config`` which: + + 1. Imports the base config module and runs ``make_config()``. This + registers every config group (model, ema, tokenizer, ...) and imports + all experiment modules so their ``cs.store(group="experiment", ...)`` + side-effects fire. + 2. Runs ``override(config, overrides)`` — Hydra ``compose`` then resolves + the ``experiment=`` selector against ``ConfigStore`` and applies + the dotted-path overrides we generated from the TOML, followed by + ``extra_overrides``. + + Returns the merged ``Config`` instance, ready for ``launch()``. + """ + with open(toml_path, "rb") as fh: + raw = tomllib.load(fh) + + # Validate structure against the pydantic schema (raises ValidationError on + # unknown keys because of ``extra="forbid"``). + SFTExperimentConfig.model_validate(raw) + + task = raw.get("job", {}).get("task", "vfm") + try: + base_config_path = TASK_TO_BASE_CONFIG[task] + except KeyError as e: + raise ValueError( + f"{toml_path}: [job].task={task!r} is not supported. " + f"Valid values: {sorted(TASK_TO_BASE_CONFIG)}" + ) from e + + overrides = build_hydra_overrides(raw) + + if extra_overrides: + # Filter "--" separator tokens (argparse may include them) and skip empty entries. + # Hydra requires "key=value" shape — reject anything malformed early. + for o in extra_overrides: + if not o or o == "--": + continue + if "=" not in o: + raise ValueError( + f"extra override {o!r} must be Hydra dotted-path syntax " + f"(e.g. 'optimizer.lr=1e-5')." + ) + overrides.append(o) + + # Import lazily so this module stays cheap to import in non-training contexts. + from cosmos_framework.utils.config import load_config + + return load_config(base_config_path, overrides) diff --git a/cosmos_framework/configs/toml_config/toml_config_helper.py b/cosmos_framework/configs/toml_config/toml_config_helper.py new file mode 100644 index 0000000..8531303 --- /dev/null +++ b/cosmos_framework/configs/toml_config/toml_config_helper.py @@ -0,0 +1,194 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Schema-agnostic helpers for the structured-TOML flow. + +``build_hydra_overrides(toml_dict)`` walks the TOML dict and emits a +``["--", "experiment=", "dotted.path=value", ...]`` list compatible +with ``cosmos_framework.utils.config_helper.override``. Per-task path remapping lives +in ``PATH_REMAPS``. + +This module knows nothing about the pydantic schema — the entry point that +ties validation + override-build + Hydra-compose together lives in +``sft_config.py``. TOML → schema validation goes through pydantic's +``BaseModel.model_validate`` directly there. +""" + +from __future__ import annotations + +from typing import Any + + +# Maps ``job.task`` to the base Hydra config that ``make_config()`` lives in. +TASK_TO_BASE_CONFIG: dict[str, str] = { + "vfm": "cosmos_framework/configs/base/config.py", + "vlm": "cosmos_framework/configs/base/vlm/config.py", +} + + +# --------------------------------------------------------------------------- +# Per-task dataclass-path → Hydra-tree-path remapping. +# +# Each task maps a tuple of dataclass-path segments (TOML shape) to one of: +# - a tuple of segments → replace that prefix in the path, keep the rest; +# - ``None`` → skip this leaf entirely (no override emitted). +# +# Resolution is longest-prefix-wins, so a more specific rule like +# ``("model", "parallelism")`` overrides a catch-all like ``("model",)``. +# Paths with no matching rule pass through unchanged. +# +# Editing surface: +# - Add VLM-specific routing rules by inserting entries in ``PATH_REMAPS["vlm"]``. +# - Skip a field for one task: map its prefix to ``None``. +# - Rename a field's Hydra path: map its prefix to the new prefix tuple. +# --------------------------------------------------------------------------- +PATH_REMAPS: dict[str, dict[tuple[str, ...], "tuple[str, ...] | None"]] = { + # VFM (OmniMoTModelConfig): every ``model.`` lives at + # ``model.config.`` in the Hydra tree. ``attn_implementation`` is a + # VLM-only knob — skip it on VFM. Other sections pass through. + "vfm": { + ("model", "attn_implementation"): None, + ("model", "backbone"): None, # VLM-only — VFM has no model.config.backbone + ("model",): ("model", "config"), + }, + # VLM (VLMModelConfig): model.config.{policy, train, freeze, ema} — + # parallelism / activation_checkpointing nest under ``.policy.*``. + # Fields that have no VLM analog map to ``None`` (skip). + "vlm": { + # No VLM analog — skip these leaves + ("model", "max_num_tokens_after_packing"): None, + ("model", "joint_attn_implementation"): None, + ("model", "lora_enabled"): None, + ("model", "lora_rank"): None, + ("model", "lora_alpha"): None, + ("model", "lora_target_modules"): None, + ("model", "tokenizer"): None, # blocks model.tokenizer.* + ("dataloader_train", "seed"): None, + ("optimizer", "eps"): None, # VLM_OPTIMIZER_KWARGS has no eps field + ("scheduler", "verbosity_interval"): None, # VLM_LAMBDACOSINE_KWARGS has no verbosity_interval + ("trainer", "callbacks", "compile_tokenizer"): None, # VFM-only callback (VLM has no torch.compile of the tokenizer) + # Rename / re-route to the VLM path + ("model", "attn_implementation"): ("model", "config", "policy", "attn_implementation"), + ("model", "parallelism"): ("model", "config", "policy", "parallelism"), + ("model", "activation_checkpointing"): ("model", "config", "policy", "activation_checkpointing"), + ("model", "ema"): ("model", "config", "ema"), + ("model", "backbone"): ("model", "config", "policy", "backbone"), + ("dataloader_train", "max_samples_per_batch"): ("dataloader_train", "max_batch_size"), + ("dataloader_train", "max_sequence_length"): ("dataloader_train", "max_tokens"), + # Catch-all for any other model.* sub-keys + ("model",): ("model", "config"), + }, +} + + +# --------------------------------------------------------------------------- +# TOML dict → Hydra override list +# --------------------------------------------------------------------------- +def _apply_remap( + rules: dict[tuple[str, ...], "tuple[str, ...] | None"], + path: list[str], +) -> "list[str] | None": + """Greedy longest-prefix lookup against ``rules``. + + Returns the rewritten path (replacement + tail), ``None`` if a matched + rule says skip, or the original path when no rule matches. + """ + for n in range(len(path), 0, -1): + key = tuple(path[:n]) + if key in rules: + replacement = rules[key] + if replacement is None: + return None + return list(replacement) + path[n:] + return path + + +def build_hydra_overrides(toml_dict: dict) -> list[str]: + """Walk a TOML dict and produce a Hydra override list compatible with + ``cosmos_framework.utils.config_helper.override``. + + Each leaf path is routed through ``PATH_REMAPS[task]`` so dataclass + paths land at the correct Hydra location for VFM vs VLM. ``job.task`` + and ``job.experiment`` are meta-fields — consumed here, not emitted. + """ + overrides: list[str] = ["--"] + + job = dict(toml_dict.get("job", {})) + task = job.pop("task", "vfm") + experiment_name = job.pop("experiment", None) + if not experiment_name: + raise ValueError("[job].experiment is required in the TOML") + overrides.append(f"experiment={experiment_name}") + + if task not in PATH_REMAPS: + raise ValueError( + f"[job].task={task!r} has no remap rules. " + f"Valid values: {sorted(PATH_REMAPS)}" + ) + rules = PATH_REMAPS[task] + + overlay = dict(toml_dict) + overlay["job"] = job + + for top_key, val in overlay.items(): + _emit_with_remap(overrides, [top_key], val, rules) + return overrides + + +def _emit_with_remap( + out: list[str], + prefix: list[str], + value: Any, + rules: dict[tuple[str, ...], "tuple[str, ...] | None"], +) -> None: + if isinstance(value, dict): + if not value: + new_path = _apply_remap(rules, prefix) + if new_path is not None: + out.append(f"{'.'.join(new_path)}={{}}") + return + for k, v in value.items(): + _emit_with_remap(out, prefix + [k], v, rules) + return + # OmegaConf/Hydra MISSING sentinel: treat ``"???"`` in TOML as "field is + # intentionally unset; user supplies it at runtime via CLI extra-override + # (or env interpolation)". Don't emit it as a Hydra override — emitting + # ``key=???`` would parse as MissingMandatoryValue at the next access + # and break --dryrun / pretty_print / to_yaml. + if value == "???": + return + new_path = _apply_remap(rules, prefix) + if new_path is None: + return + out.append(f"{'.'.join(new_path)}={_hydra_format(value)}") + + +def _hydra_format(v: Any, in_list: bool = False) -> str: + """Convert a Python value to a Hydra CLI override RHS. + + ``in_list=True`` indicates the value is being emitted inside a list + literal (``[a,b,c]``); strings then get single-quoted unconditionally + so numeric-looking entries like ``"480"`` stay strings rather than + being coerced to int by Hydra's list parser. + """ + if v is None: + return "null" + if isinstance(v, bool): + return "true" if v else "false" + if isinstance(v, (int, float)): + return str(v) + if isinstance(v, list): + return "[" + ",".join(_hydra_format(x, in_list=True) for x in v) + "]" + if isinstance(v, str): + # Inside a list literal, always quote so numeric-looking strings + # ("480") aren't parsed as int. At top level, quote only when the + # string contains characters Hydra would otherwise interpret — + # commas (sweep / list marker) or whitespace. Env-interpolation + # strings like ``${oc.env:NAME}`` are safe unquoted because Hydra + # recognizes the ``${...}`` form even with a colon inside. + if in_list or "," in v or " " in v: + return f"'{v}'" + return v + return str(v) + + diff --git a/cosmos_framework/data/__init__.py b/cosmos_framework/data/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/data/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/data/imaginaire/webdataset/augmentors/augmentor.py b/cosmos_framework/data/imaginaire/webdataset/augmentors/augmentor.py new file mode 100644 index 0000000..57d2005 --- /dev/null +++ b/cosmos_framework/data/imaginaire/webdataset/augmentors/augmentor.py @@ -0,0 +1,52 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from collections.abc import Iterable +from typing import Any, Generator, Optional + + +class Augmentor: + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + r"""Base augmentor class + + Args: + input_keys (list): List of input keys + output_keys (list): List of output keys + args (dict): Arguments associated with the augmentation + """ + self.input_keys = input_keys + self.output_keys = output_keys + self.args = args + + def __call__(self, *args: Any, **kwds: Any) -> Any: + raise ValueError("Augmentor not implemented") + + +class IterableAugmentor: + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + r"""Base augmentor class + + Args: + input_keys (list): List of input keys + output_keys (list): List of output keys + args (dict): Arguments associated with the augmentation + """ + self.input_keys = input_keys + self.output_keys = output_keys + self.args = args + self.is_generator = True + + def __call__(self, data: Iterable) -> Generator: + r"""Example usage: + + for data_dict in data: + # Do something to data_dict + data_dict["input"] = data_dict["raw_sequence"][:, :-1] + data_dict["target"] = data_dict["raw_sequence"][:, 1:] + # Skip sample if needed + if data_dict["input"].shape[1] < 64: + continue + # Construct a generator + yield data_dict + """ + raise ValueError("Augmentor not implemented") diff --git a/cosmos_framework/data/vfm/__init__.py b/cosmos_framework/data/vfm/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/data/vfm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/data/vfm/action/__init__.py b/cosmos_framework/data/vfm/action/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/data/vfm/action/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/data/vfm/action/action_normalization.py b/cosmos_framework/data/vfm/action/action_normalization.py new file mode 100644 index 0000000..d553161 --- /dev/null +++ b/cosmos_framework/data/vfm/action/action_normalization.py @@ -0,0 +1,49 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Action normalization helpers.""" + +import json +from pathlib import Path + +import numpy as np +import torch + +from cosmos_framework.utils import log + + +def load_action_stats(stats_path: str, stats_key: str = "global") -> dict[str, np.ndarray]: + """Load pre-computed action normalization stats from a JSON file.""" + path = Path(stats_path) + if not path.exists(): + raise FileNotFoundError(f"Action normalization stats not found at {stats_path}.") + log.info(f"Loading action normalization stats from {stats_path}") + with path.open("r") as f: + raw = json.load(f) + if stats_key in raw: + raw = raw[stats_key] + if not isinstance(raw, dict): + raise TypeError(f"Action normalization stats block {stats_key!r} in {stats_path} must be a dict.") + elif stats_key != "global": + raise KeyError(f"Action normalization stats block {stats_key!r} not found in {stats_path}.") + stat_keys = {"mean", "std", "min", "max", "q01", "q99"} + return {k: np.array(v, dtype=np.float32) for k, v in raw.items() if k in stat_keys} + + +def normalize_action( + action: torch.Tensor, + method: str, + stats: dict[str, torch.Tensor], +) -> torch.Tensor: + """Normalize action tensor (all dimensions including gripper).""" + if method == "quantile": + q01, q99 = stats["q01"], stats["q99"] + denom = (q99 - q01).clamp(min=1e-8) + return (2.0 * (action - q01) / denom - 1.0).clamp(-1.0, 1.0) + if method == "meanstd": + return (action - stats["mean"]) / stats["std"].clamp(min=1e-8) + if method == "minmax": + lo, hi = stats["min"], stats["max"] + denom = (hi - lo).clamp(min=1e-8) + return (2.0 * (action - lo) / denom - 1.0).clamp(-1.0, 1.0) + raise ValueError(f"Unknown normalization method: {method!r}") diff --git a/cosmos_framework/data/vfm/action/action_normalization_test.py b/cosmos_framework/data/vfm/action/action_normalization_test.py new file mode 100644 index 0000000..b9e4e87 --- /dev/null +++ b/cosmos_framework/data/vfm/action/action_normalization_test.py @@ -0,0 +1,78 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +import torch + +from cosmos_framework.data.vfm.action.action_normalization import load_action_stats +from cosmos_framework.data.vfm.action.cosmos3_action_lerobot import ( + ActionNormalization, + BaseActionLeRobotDataset, +) + + +class _StatsOnlyDataset(BaseActionLeRobotDataset): + """Minimal shell for testing BaseActionLeRobotDataset normalization helpers.""" + + def __init__(self, stats_path: Path, action_normalization: ActionNormalization | None) -> None: + self._stats_path = stats_path + self._action_normalization = action_normalization + self._norm_stats: dict[str, torch.Tensor] | None = None + + def _normalizer_path(self) -> Path: + return self._stats_path + + +def _write_stats(tmp_path: Path) -> Path: + stats_path = tmp_path / "stats.json" + payload = { + "global": { + "mean": [0.0, 0.0], + "std": [1.0, 1.0], + "min": [0.0, -1.0], + "max": [10.0, 1.0], + "q01": [0.0, -1.0], + "q99": [10.0, 1.0], + }, + "global_raw": { + "mean": [0.0, 0.0], + "std": [1.0, 1.0], + "min": [0.0, 0.0], + "max": [10.0, 100.0], + "q01": [0.0, 0.0], + "q99": [10.0, 100.0], + }, + } + stats_path.write_text(json.dumps(payload)) + return stats_path + + +@pytest.mark.L0 +def test_load_action_stats_selects_global_raw_block(tmp_path: Path) -> None: + stats_path = _write_stats(tmp_path) + + stats = load_action_stats(str(stats_path), stats_key="global_raw") + + assert stats["q99"].tolist() == [10.0, 100.0] + + +@pytest.mark.L0 +def test_quantile_rot_uses_global_raw_stats(tmp_path: Path) -> None: + stats_path = _write_stats(tmp_path) + action = torch.tensor([[5.0, 50.0]], dtype=torch.float32) # [T,D] + + quantile_dataset = _StatsOnlyDataset(stats_path, "quantile") + quantile_rot_dataset = _StatsOnlyDataset(stats_path, "quantile_rot") + + quantile_action = quantile_dataset._normalize_action(action) # [T,D] + quantile_rot_action = quantile_rot_dataset._normalize_action(action) # [T,D] + expected_quantile = torch.tensor([[0.0, 1.0]], dtype=torch.float32) # [T,D] + expected_quantile_rot = torch.tensor([[0.0, 0.0]], dtype=torch.float32) # [T,D] + + torch.testing.assert_close(quantile_action, expected_quantile) + torch.testing.assert_close(quantile_rot_action, expected_quantile_rot) diff --git a/cosmos_framework/data/vfm/action/action_spec.py b/cosmos_framework/data/vfm/action/action_spec.py new file mode 100644 index 0000000..c9c0f38 --- /dev/null +++ b/cosmos_framework/data/vfm/action/action_spec.py @@ -0,0 +1,235 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Action-vector specification: per-dim type label + idle thresholds. + +Single concept: every column of an action vector has a :class:`DimType` label. +Idle detection iterates by type and applies the matching algorithm: + + POS → ‖action[pos_idx]‖ per arm < eps_t + ROT → distance(rot, identity) per group < eps_r + GRIPPER → max |Δgripper| < eps_g (frame 0 idle by convention) + JOINT → max |Δjoint| < joint_threshold (frame 0 idle) + RESERVED → ignored + +An :class:`ActionSpec` is just ``names`` + ``types`` + ``rotation_format``. +Build one declaratively via :func:`build_action_spec` from DSL components:: + + build_action_spec(Pos(), Rot("rot6d"), Gripper()) # 10D single arm + build_action_spec(Pos(), Rot("rot6d")) # 9D no gripper + build_action_spec(Joint(n=14, label="arm"), # 30D joint-space + Joint(n=14, label="end"), + Joint(n=2, label="gripper")) + build_action_spec(Pos(prefix="left"), Rot("rot6d", "left"), Gripper(prefix="left"), + Pos(prefix="right"), Rot("rot6d", "right"), Gripper(prefix="right")) + +Naming convention: + Default ``pos_x``, ``rot_0``, ``gripper``, ``arm_0`` ... + With ``prefix="left"`` (idempotent on trailing ``_``): ``left_pos_x`` ... +""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum +from typing import ClassVar + +from cosmos_framework.data.vfm.action.pose_utils import ( + RotationConvention, + _identity_rotation_vector, +) + + +class DimType(str, Enum): + """Per-column action-dim category (drives idle detection).""" + + POS = "pos" + ROT = "rot" + GRIPPER = "gripper" + JOINT = "joint" + RESERVED = "reserved" + + +@dataclass(frozen=True, slots=True) +class ActionSpec: + """Structural description of an action vector: names + per-dim types. + + All ROT dims share a single ``rotation_format``; mixed formats in one spec + are not supported (raise at build time). + + This struct contains no detection thresholds — those are passed at call + time to :func:`compute_idle_frames` so each dataset can tune them + independently of layout. + """ + + names: list[str] + types: list[DimType] + rotation_format: RotationConvention = "rot6d" + + @property + def dim(self) -> int: + return len(self.names) + + +# --------------------------------------------------------------------------- +# DSL components +# --------------------------------------------------------------------------- + + +def _join_prefix(prefix: str, name: str) -> str: + """Join ``prefix`` and ``name`` with a single ``_``; idempotent on trailing ``_``.""" + return name if not prefix else f"{prefix.rstrip('_')}_{name}" + + +@dataclass(frozen=True) +class Pos: + """Translation block. + + Default 3D (``pos_x``, ``pos_y``, ``pos_z``). For planar tasks (e.g. PushT) + use ``Pos(dim=2)`` → ``pos_x``, ``pos_y``. ``dim >= 4`` falls back to + indexed names ``pos_0``, ``pos_1``, ... + """ + + dim: int = 3 + prefix: str = "" + type: ClassVar[DimType] = DimType.POS + + def names(self) -> list[str]: + if self.dim <= 3: + return [_join_prefix(self.prefix, f"pos_{c}") for c in "xyz"[: self.dim]] + return [_join_prefix(self.prefix, f"pos_{i}") for i in range(self.dim)] + + +@dataclass(frozen=True) +class Rot: + """Rotation block; ``format`` selects the encoding. + + Supported formats and per-dim names: + + - ``rot6d`` → 6 dims, ``rot_0`` ... ``rot_5`` (identity ``[1,0,0,0,1,0]``) + - ``rot9d`` → 9 dims, ``rot_0`` ... ``rot_8`` (identity ``[1,0,0,0,1,0,0,0,1]``) + - ``euler_xyz`` → 3 dims, ``roll``, ``pitch``, ``yaw`` (identity ``[0,0,0]``) + - ``axisangle`` → 3 dims, ``axang_x/y/z`` (identity ``[0,0,0]``) + - ``quat_xyzw`` / ``quat_wxyz`` → 4 dims, ``quat_x/y/z/w`` in declared order + """ + + format: RotationConvention = "rot6d" + prefix: str = "" + type: ClassVar[DimType] = DimType.ROT + + @property + def rotation_format(self) -> RotationConvention: + return self.format + + @property + def dim(self) -> int: + return _identity_rotation_vector(self.format).shape[0] + + def names(self) -> list[str]: + if self.format == "euler_xyz": + return [_join_prefix(self.prefix, c) for c in ("roll", "pitch", "yaw")] + if self.format == "axisangle": + return [_join_prefix(self.prefix, f"axang_{c}") for c in "xyz"] + if self.format.startswith("quat_"): + order = self.format.split("_", 1)[1] # "xyzw" or "wxyz" + return [_join_prefix(self.prefix, f"quat_{c}") for c in order] + return [_join_prefix(self.prefix, f"rot_{i}") for i in range(self.dim)] + + +@dataclass(frozen=True) +class Gripper: + """1D gripper command (binary 0/1 or continuous). Detected by frame-diff.""" + + prefix: str = "" + type: ClassVar[DimType] = DimType.GRIPPER + + @property + def dim(self) -> int: + return 1 + + def names(self) -> list[str]: + return [_join_prefix(self.prefix, "gripper")] + + +@dataclass(frozen=True) +class Joint: + """``n`` joint commands. Detected by frame-diff against ``joint_threshold``.""" + + n: int = 0 + label: str = "joint" + prefix: str = "" + type: ClassVar[DimType] = DimType.JOINT + + @property + def dim(self) -> int: + return self.n + + def names(self) -> list[str]: + return [_join_prefix(self.prefix, f"{self.label}_{i}") for i in range(self.n)] + + +@dataclass(frozen=True) +class Reserved: + """``n`` dims counted in ``action_dim`` but ignored by idle detection.""" + + n: int = 0 + label: str = "reserved" + prefix: str = "" + type: ClassVar[DimType] = DimType.RESERVED + + @property + def dim(self) -> int: + return self.n + + def names(self) -> list[str]: + return [_join_prefix(self.prefix, f"{self.label}_{i}") for i in range(self.n)] + + +# --------------------------------------------------------------------------- +# Builder +# --------------------------------------------------------------------------- + + +# Type alias for any DSL component. Not a runtime check — only annotation hint. +Component = Pos | Rot | Gripper | Joint | Reserved + + +def build_action_spec(*components: Component) -> ActionSpec: + """Compose ``components`` into an :class:`ActionSpec`. + + Each component contributes its ``names()`` and replicates its ``type`` for + every column it occupies. The first ROT component's ``rotation_format`` + is captured for the whole spec; mixing formats raises ``ValueError``. + """ + names: list[str] = [] + types: list[DimType] = [] + rotation_format: RotationConvention | None = None + + for c in components: + names.extend(c.names()) + types.extend([c.type] * c.dim) + if c.type == DimType.ROT: + fmt = c.rotation_format # type: ignore[union-attr] + if rotation_format is None: + rotation_format = fmt + elif rotation_format != fmt: + raise ValueError(f"Mixed rotation_format in one ActionSpec: {rotation_format!r} vs {fmt!r}") + + return ActionSpec( + names=names, + types=types, + rotation_format=rotation_format or "rot6d", + ) + + +__all__ = [ + "ActionSpec", + "Component", + "DimType", + "Gripper", + "Joint", + "Pos", + "Reserved", + "Rot", + "build_action_spec", +] diff --git a/cosmos_framework/data/vfm/action/agibotworld_beta_dataset.py b/cosmos_framework/data/vfm/action/agibotworld_beta_dataset.py new file mode 100644 index 0000000..efcc4bd --- /dev/null +++ b/cosmos_framework/data/vfm/action/agibotworld_beta_dataset.py @@ -0,0 +1,366 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""AgiBotWorld-Beta dataset with FK-pose actions and multi-view support. + +Uses the same calibrated G1/omnipicker URDF forward kinematics as Embodiment C to produce +relative SE(3) delta actions for head camera and left/right gripper-base wrists, +concatenated with gripper open-fraction values. + +Action layout (29 dims, same as GEAR gripper): + ``[head_cam_delta(9), right_wrist_delta(9), right_gripper(1), + left_wrist_delta(9), left_gripper(1)]`` + +View modes: + - **ego_view**: single ``observation.images.head`` camera. + - **concat_view**: head view on top, left/right wrist views resized and + concatenated horizontally on the bottom. +""" + +from __future__ import annotations + +from typing import Any, Literal, cast + +import numpy as np +import torch +import torch.nn.functional as F + +from cosmos_framework.data.vfm.action.embodiment_c_fk import ( + AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST, + apply_agibot_gripper_to_opencv, + apply_robot_base_motion_to_poses, + compute_fk_transforms_batch, + compute_link_poses_batch, + convert_gripper_state_to_open_fraction, + extract_fk_transforms_from_link_poses, +) +from cosmos_framework.data.vfm.action.embodiment_c_spec import AGIBOT_GEAR_GRIPPER_NORMALIZER_EMBODIMENT_TYPE +from cosmos_framework.data.vfm.action.agibotworld_beta_dataset_config import LEROBOT_ROOTS +from cosmos_framework.data.vfm.action.cosmos3_action_lerobot import ( + ActionNormalization, + ActionSpec, + BaseActionLeRobotDataset, + Gripper, + Pos, + Rot, + build_action_spec, +) +from cosmos_framework.data.vfm.action.pose_utils import PoseConvention, pose_abs_to_rel +from cosmos_framework.data.vfm.action.viewpoint_utils import Viewpoint + +# Video keys. +_HEAD_KEY = "observation.images.head" +_HAND_LEFT_KEY = "observation.images.hand_left" +_HAND_RIGHT_KEY = "observation.images.hand_right" + +# State observation keys needed for FK. +_ROBOT_POSITION_KEY = "observation.states.robot.position" +_ROBOT_ORIENTATION_KEY = "observation.states.robot.orientation" +_STATE_KEYS = [ + "observation.states.effector.position", + "observation.states.joint.position", + "observation.states.head.position", + "observation.states.waist.position", + _ROBOT_POSITION_KEY, + _ROBOT_ORIENTATION_KEY, +] + +_BASE_ROOT = "" + + +def _resolve_root_paths(root: str | list[str] | tuple[str, ...] | None) -> list[str]: + """Normalize a root argument into concrete AgiBotWorld-Beta LeRobot roots.""" + + if root is None: + return [f"{_BASE_ROOT}/{subpath}" for subpath in LEROBOT_ROOTS] + if not isinstance(root, str): + return [str(path) for path in root] + return [f"{root}/{subpath}" for subpath in LEROBOT_ROOTS] + + +def _split_task_for_caption(task: str) -> tuple[str, str]: + """Split AgiBotWorld task text into train caption and debug-only detail.""" + + ai_caption, separator, debug_caption = task.partition("|") + if not separator: + return task.strip(), "" + return ai_caption.strip(), debug_caption.strip() + + +def _assemble_embodiment_c_state( + effector_pos: np.ndarray, + joint_pos: np.ndarray, + head_pos: np.ndarray, + waist_pos: np.ndarray, +) -> np.ndarray: + """Assemble a GEAR-compatible flat state vector from Beta's decomposed fields. + + Args: + effector_pos: ``(N, 2)`` gripper positions as ``[left_gripper, right_gripper]``. + joint_pos: ``(N, 14)`` arm joint positions (7 left + 7 right). + head_pos: ``(N, 2)`` head joints as ``[head_yaw, head_pitch]``. + waist_pos: ``(N, 2)`` waist joints as ``[body_pitch, lift_body]``. + + Returns: + State array of shape ``(N, 20)`` with GEAR gripper layout: + ``[arm_joints(14), gripper(2), head_yaw, head_pitch, waist_pitch, waist_lift]`` + + Note: + Beta ``waist_pos = [body_pitch, lift_body]`` maps directly into the + standard GEAR body/head block: + ``state[16:20] = [head_yaw, head_pitch, body_pitch, lift_body]``. + """ + # Body/head block: [head_yaw, head_pitch, waist_pitch, waist_lift] + body_head = np.stack( + [head_pos[:, 0], head_pos[:, 1], waist_pos[:, 0], waist_pos[:, 1]], + axis=-1, + ) # [N,4] + + return np.concatenate([joint_pos, effector_pos, body_head], axis=-1).astype(np.float32) # [N,20] + + +class AgiBotWorldBetaDataset(BaseActionLeRobotDataset): + """AgiBotWorld-Beta dataset with FK-pose actions matching GEAR gripper format. + + Concat view layout: + ┌──────────────────┐ + │ head │ (H, W) + ├─────────┬────────┤ + │ hand_L │ hand_R │ (H/2, W/2) each + └─────────┴────────┘ + """ + + def __init__( + self, + root: str | list[str] | tuple[str, ...] | None = None, + fps: float = 10.0, + chunk_length: int = 16, + split_seed: int = 42, + split_val_ratio: float = 0.05, + split: str = "train", + action_normalization: ActionNormalization | None = None, + mode: str = "joint", + viewpoint: Viewpoint = "concat_view", + pose_convention: PoseConvention = "backward_framewise", + rotation_format: Literal["rot6d"] = "rot6d", + tolerance_s: float = 3e-4, + max_loaded_datasets: int = 32, + skip_video_loading: bool = False, + sample_stride: int = 1, + enable_fast_init: bool = False, + fast_init_max_workers: int = 64, + return_agibot_link_poses: bool = False, + ) -> None: + super().__init__( + fps=fps, + chunk_length=chunk_length, + split_seed=split_seed, + split_val_ratio=split_val_ratio, + split=split, + mode=mode, + embodiment_type="agibotworld", + viewpoint=viewpoint, + action_normalization=action_normalization, + pose_convention=pose_convention, + rotation_format=rotation_format, + tolerance_s=tolerance_s, + max_loaded_datasets=max_loaded_datasets, + skip_video_loading=skip_video_loading, + sample_stride=sample_stride, + enable_fast_init=enable_fast_init, + fast_init_max_workers=fast_init_max_workers, + ) + + self._is_concat_view = viewpoint == "concat_view" + self._to_opencv: dict[str, np.ndarray] = AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST + self._return_agibot_link_poses: bool = return_agibot_link_poses + + self._all_shard_roots = _resolve_root_paths(root) + + # T+1 frames for observations (states + video). Source action columns + # are intentionally not requested; emitted actions are state deltas. + frame_ts_obs = [i * self._dt for i in range(self._chunk_length + 1)] + + self._delta_timestamps = { + _HEAD_KEY: frame_ts_obs, + } + # State keys for FK (T+1 frames). + for key in _STATE_KEYS: + self._delta_timestamps[key] = frame_ts_obs + # Multi-view cameras. + if self._is_concat_view: + self._delta_timestamps[_HAND_LEFT_KEY] = frame_ts_obs + self._delta_timestamps[_HAND_RIGHT_KEY] = frame_ts_obs + + def _normalizer_filename(self) -> str: + """Use the shared Embodiment C gripper FK-pose stats for Beta data.""" + + return f"{AGIBOT_GEAR_GRIPPER_NORMALIZER_EMBODIMENT_TYPE}_{self._pose_convention}_{self._rotation_format}.json" + + # -- FK-based action construction ---------------------------------------- + + def _build_fk_action(self, sample: dict[str, Any]) -> tuple[torch.Tensor, dict[str, Any]]: + """Build relative FK-pose action plus absolute initial poses. + + Assembles GEAR-compatible flat state from Beta's decomposed observation + fields, runs calibrated G1/omnipicker URDF forward kinematics, and converts to framewise + relative SE(3) deltas with rot6d rotation blocks. + + Returns: + ``(action, extras)`` where ``action`` is ``(T, 29)`` and ``extras`` + carries absolute initial poses for reconstruction. + + Beta ``observation.states.effector.position`` stores left/right scalar + gripper positions. Before these are concatenated into the + GEAR-compatible 29D action, they are normalized to the shared + viewer/action convention: + ``0.0`` means closed and ``1.0`` means open. AgiBot actuator-close + state values use ``0=open`` and ``120=closed``; angle-valued gripper + states are scaled by ``convert_gripper_state_to_open_fraction``. + """ + + # Assemble GEAR-compatible flat state from decomposed Beta fields. + effector_pos = sample["observation.states.effector.position"].detach().cpu().numpy() # [T+1,2] + joint_pos = sample["observation.states.joint.position"].detach().cpu().numpy() # [T+1,14] + head_pos = sample["observation.states.head.position"].detach().cpu().numpy() # [T+1,2] + waist_pos = sample["observation.states.waist.position"].detach().cpu().numpy() # [T+1,2] + robot_pos = sample[_ROBOT_POSITION_KEY].detach().cpu().numpy() # [T+1,3] + robot_quat = sample[_ROBOT_ORIENTATION_KEY].detach().cpu().numpy() # [T+1,4] + num_state_steps = min( + effector_pos.shape[0], + joint_pos.shape[0], + head_pos.shape[0], + waist_pos.shape[0], + robot_pos.shape[0], + robot_quat.shape[0], + ) + if num_state_steps < 2: + raise ValueError(f"{self.__class__.__name__}: state observations must contain at least 2 frames.") + effector_pos = effector_pos[:num_state_steps].astype(np.float32, copy=False) # [T+1,2] + joint_pos = joint_pos[:num_state_steps].astype(np.float32, copy=False) # [T+1,14] + head_pos = head_pos[:num_state_steps].astype(np.float32, copy=False) # [T+1,2] + waist_pos = waist_pos[:num_state_steps].astype(np.float32, copy=False) # [T+1,2] + robot_pos = robot_pos[:num_state_steps].astype(np.float32, copy=False) # [T+1,3] + robot_quat = robot_quat[:num_state_steps].astype(np.float32, copy=False) # [T+1,4] + states_np = _assemble_embodiment_c_state(effector_pos, joint_pos, head_pos, waist_pos) # [T+1,20] + + # Forward kinematics → absolute 4×4 transforms; gripper rotations are + # first lifted by observed mobile-base motion, then converted through + # _to_opencv for action/viewer display. + link_poses = None + if self._return_agibot_link_poses: + link_poses = compute_link_poses_batch(states_np, "embodiment_c_gripper") # {name:[T+1,4,4]} + link_poses = apply_robot_base_motion_to_poses(link_poses, robot_pos, robot_quat) # {name:[T+1,4,4]} + native_fk = extract_fk_transforms_from_link_poses(link_poses) # {name:[T+1,4,4]} + else: + native_fk = compute_fk_transforms_batch(states_np, "embodiment_c_gripper") # {name:[T+1,4,4]} + native_fk = apply_robot_base_motion_to_poses(native_fk, robot_pos, robot_quat) # {name:[T+1,4,4]} + fk = apply_agibot_gripper_to_opencv(native_fk, self._to_opencv) # {name:[T+1,4,4]} + + # Relative SE(3) deltas. + pose_convention = cast(PoseConvention, self._pose_convention) + head_rel = pose_abs_to_rel(fk["head_camera"], rotation_format="rot6d", pose_convention=pose_convention) # [T,9] + right_rel = pose_abs_to_rel( + fk["right_wrist"], rotation_format="rot6d", pose_convention=pose_convention + ) # [T,9] + left_rel = pose_abs_to_rel(fk["left_wrist"], rotation_format="rot6d", pose_convention=pose_convention) # [T,9] + + # Gripper open fractions: Beta observed effector[0]=left, + # effector[1]=right. + # The converter standardizes URDF-angle and actuator-close-degree + # encodings to 0.0=closed, 1.0=open for viewer consistency. + right_gripper = convert_gripper_state_to_open_fraction(effector_pos[1:, 1:2]) # [T,1] + left_gripper = convert_gripper_state_to_open_fraction(effector_pos[1:, 0:1]) # [T,1] + + # Concatenate in GEAR action order. + action_np = np.concatenate( + [head_rel, right_rel, right_gripper, left_rel, left_gripper], + axis=-1, + ).astype(np.float32) # [T,29] + + extras = { + "initial_pose": torch.from_numpy(fk["head_camera"][0].copy()).float(), # [4,4] + "initial_pose_right": torch.from_numpy(fk["right_wrist"][0].copy()).float(), # [4,4] + "initial_pose_left": torch.from_numpy(fk["left_wrist"][0].copy()).float(), # [4,4] + } + if link_poses is not None: + extras["agibot_link_poses"] = { + link_name: torch.from_numpy(poses.copy()).float() for link_name, poses in link_poses.items() + } # {name:[T+1,4,4]} + return torch.from_numpy(action_np).float(), extras # [T,29] + + # -- Multi-view composition ---------------------------------------------- + + def _compose_multi_view(self, sample: dict[str, Any]) -> torch.Tensor: + """Compose head, left-hand, and right-hand views into a single frame. + + Returns: + Composited video tensor in raw LeRobot ``(T, C, H_out, W)`` float format. + """ + top = sample[_HEAD_KEY] # [T,C,H,W] + left = sample[_HAND_LEFT_KEY] # [T,C,H_l,W_l] + right = sample[_HAND_RIGHT_KEY] # [T,C,H_r,W_r] + + _, _, h_top, w_top = top.shape + half_h, half_w = h_top // 2, w_top // 2 + + left = F.interpolate(left, size=(half_h, half_w), mode="bilinear", align_corners=False) # [T,C,H/2,W/2] + right = F.interpolate(right, size=(half_h, half_w), mode="bilinear", align_corners=False) # [T,C,H/2,W/2] + bottom = torch.cat([left, right], dim=-1) # [T,C,H/2,W] + + composite = torch.cat([top, bottom], dim=-2) # [T,C,3H/2,W] + return composite # [T,C,3H/2,W] + + def __getitem__(self, idx: int) -> dict[str, Any]: + mode, _, _, sample = self._fetch_sample(idx) + + ai_caption, debug_caption = _split_task_for_caption(sample["task"]) + + if self._skip_video_loading: + video = None + elif self._is_concat_view: + video = self._compose_multi_view(sample) + else: + video = sample[_HEAD_KEY] # [T,C,H,W] + + # Action: FK-pose based (same layout as GEAR gripper). + action, action_extras = self._build_fk_action(sample) + + extras: dict[str, Any] = {**action_extras} + if self._is_concat_view: + extras["additional_view_description"] = ( + "The top row shows the head-mounted camera view looking down at the workspace. " + "The bottom row contains two horizontally concatenated wrist-mounted camera views: " + "the left hand camera on the left and the right hand camera on the right." + ) + if debug_caption: + extras["debug_caption"] = debug_caption + + return self._build_result(mode=mode, video=video, action=action, ai_caption=ai_caption, **extras) + + def _build_action_spec(self) -> ActionSpec: + """AgiBotWorld-Beta bimanual layout (29D, same as GEAR gripper). + + ``[head_pos+rot6d (9) | right_pos+rot6d (9) | right_gripper (1) + | left_pos+rot6d (9) | left_gripper (1)]`` + + All three SE(3) blocks (head camera + both wrists) participate in + idle-frame detection: a chunk only counts as idle if the head is + steady AND both arms are at rest AND both grippers are unchanged. + Override this method (or use ``Reserved`` for head dims) if you want + head motion to be ignored by idle detection. + """ + return build_action_spec( + Pos(prefix="head"), + Rot("rot6d", prefix="head"), + Pos(prefix="right"), + Rot("rot6d", prefix="right"), + Gripper(prefix="right"), + Pos(prefix="left"), + Rot("rot6d", prefix="left"), + Gripper(prefix="left"), + ) + + @property + def action_dim(self) -> int: + return 29 diff --git a/cosmos_framework/data/vfm/action/agibotworld_beta_dataset_config.py b/cosmos_framework/data/vfm/action/agibotworld_beta_dataset_config.py new file mode 100644 index 0000000..398be6b --- /dev/null +++ b/cosmos_framework/data/vfm/action/agibotworld_beta_dataset_config.py @@ -0,0 +1,203 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +LEROBOT_ROOTS = [ + "task_327", + "task_351", + "task_352", + "task_354", + "task_356", + "task_357", + "task_358", + "task_359", + "task_360", + "task_361", + "task_362", + "task_363", + "task_365", + "task_366", + "task_367", + "task_368", + "task_369", + "task_372", + "task_373", + "task_374", + "task_375", + "task_376", + "task_377", + "task_378", + "task_380", + "task_384", + "task_385", + "task_388", + "task_389", + "task_390", + "task_392", + "task_398", + "task_410", + "task_414", + "task_421", + "task_422", + "task_424", + "task_425", + "task_428", + "task_429", + "task_431", + "task_433", + "task_434", + "task_438", + "task_440", + "task_444", + "task_445", + "task_446", + "task_451", + "task_452", + "task_453", + "task_454", + "task_455", + "task_460", + "task_462", + "task_463", + "task_464", + "task_465", + "task_466", + "task_468", + "task_470", + "task_471", + "task_474", + "task_477", + "task_478", + "task_480", + "task_483", + "task_485", + "task_486", + "task_487", + "task_491", + "task_492", + "task_494", + "task_497", + "task_498", + "task_501", + "task_503", + "task_504", + "task_505", + "task_506", + "task_507", + "task_508", + "task_509", + "task_510", + "task_511", + "task_512", + "task_515", + "task_520", + "task_521", + "task_522", + "task_524", + "task_525", + "task_527", + "task_528", + "task_529", + "task_532", + "task_533", + "task_534", + "task_535", + "task_537", + "task_540", + "task_541", + "task_542", + "task_543", + "task_544", + "task_545", + "task_550", + "task_551", + "task_555", + "task_556", + "task_558", + "task_561", + "task_563", + "task_566", + "task_567", + "task_568", + "task_570", + "task_573", + "task_574", + "task_575", + "task_580", + "task_582", + "task_584", + "task_587", + "task_588", + "task_589", + "task_590", + "task_593", + "task_596", + "task_597", + "task_598", + "task_599", + "task_600", + "task_602", + "task_603", + "task_604", + "task_607", + "task_609", + "task_613", + "task_616", + "task_619", + "task_621", + "task_658", + "task_664", + "task_681", + "task_682", + "task_683", + "task_688", + "task_689", + "task_692", + "task_695", + "task_698", + "task_707", + "task_708", + "task_709", + "task_711", + "task_712", + "task_714", + "task_715", + "task_716", + "task_717", + "task_719", + "task_722", + "task_725", + "task_726", + "task_729", + "task_732", + "task_734", + "task_735", + "task_739", + "task_740", + "task_741", + "task_744", + "task_748", + "task_751", + "task_761", + "task_762", + "task_764", + "task_765", + "task_773", + "task_779", + "task_781", + "task_782", + "task_783", + "task_785", + "task_786", + "task_787", + "task_790", +] + +OBSERVATION_FEATURES = [ + "observation.images.head", + "observation.states.effector.position", + "observation.states.head.position", + "observation.states.joint.position", + "observation.states.robot.orientation", + "observation.states.robot.position", + "observation.states.waist.position", +] diff --git a/cosmos_framework/data/vfm/action/av_dataset.py b/cosmos_framework/data/vfm/action/av_dataset.py new file mode 100644 index 0000000..1c9d410 --- /dev/null +++ b/cosmos_framework/data/vfm/action/av_dataset.py @@ -0,0 +1,1009 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""AV Dataset for Action training. + +This module provides an IterableDataset for AV data +that loads S3 storage containing tar files with video, action trajectories, +and route waypoints. + +Data format expected: + s3://bucket/path/*.tar -> pkl files containing: + - video: mp4 bytes + - action: pickled dict with: + - history_xyz: (history_length, 3) tensor - position history + - history_quat: (history_length, 4) tensor - quaternion history + - future_xyz: (future_length, 3) tensor - position future + - future_quat: (future_length, 4) tensor - quaternion future + - route: pickled numpy array of shape (num_waypoints, 3) - route waypoints in ego frame + +Action format: 7D pose per timestep [x, y, z, qw, qx, qy, qz] (3 position + 4 quaternion) +""" + +import io +import json +import math +import pickle +import random +import tarfile +from typing import Iterator, Literal + +import numpy as np +import torch +import torchvision +import torchvision.transforms.functional as F +from scipy.spatial.transform import Rotation +from torch.utils.data import IterableDataset + +# import torch.multiprocessing +# torch.multiprocessing.set_sharing_strategy("file_system") +from cosmos_framework.utils import log +from cosmos_framework.utils.easy_io import easy_io +from cosmos_framework.data.vfm.action.camera_dataset import get_target_size_and_crop +from cosmos_framework.data.vfm.action.domain_utils import get_domain_id +from cosmos_framework.data.vfm.action.pose_utils import ( + RotationConvention, + build_abs_pose_from_components, + pose_abs_to_rel, +) + + +def decode_video_bytes( + video_bytes: bytes, + resolution: str | None = None, + history_len: float | None = None, + future_len: float | None = None, + original_history_steps: int | None = None, +) -> tuple[torch.Tensor, float]: + """Decode video from mp4 bytes using torchvision.io. + + Args: + video_bytes: Raw mp4 video bytes. + resolution: Target resolution for video frames (e.g. "256", "480"). If None, keeps original resolution. + history_len: Desired history length in seconds. Used with future_len to slice video. + future_len: Desired future length in seconds. Used with history_len to slice video. + original_history_steps: Number of frames in the original history portion of the video. + + Returns: + Tuple of (video tensor in (C, T, H, W) uint8 format, original fps). + + Note: + The video structure is [history_frames | future_frames]. When slicing: + - History portion: take last (history_len * fps) frames from video[:original_history_steps] + - Future portion: take first (future_len * fps) frames from video[original_history_steps:] + This mirrors the slicing in process_action_trajectory. + """ + # Write bytes to a temporary file for torchvision.io.read_video + import tempfile + + with tempfile.NamedTemporaryFile(suffix=".mp4", delete=True) as tmp_file: + tmp_file.write(video_bytes) + tmp_file.flush() + + # Read video using torchvision.io + # Returns: (video_frames, audio_frames, info) + # video_frames shape: (T, H, W, C) uint8 + video_frames, _, info = torchvision.io.read_video(tmp_file.name, pts_unit="sec") + + original_fps = info.get("video_fps", 30.0) + + # Slice video to match history_len and future_len + # Video structure: [history_frames | future_frames] + if original_history_steps is None: + raise ValueError("original_history_steps is required to slice video") + + # Split video at history/future boundary + history_video = video_frames[:original_history_steps] + future_video = video_frames[original_history_steps:] + + # Slice history (take last N frames) + if history_len is not None: + history_steps = int(history_len * original_fps) + if history_steps > history_video.shape[0]: + raise ValueError( + f"Requested history_len={history_len}s ({history_steps} frames at {original_fps}Hz) " + f"exceeds available history video ({history_video.shape[0]} frames)" + ) + history_video = history_video[-history_steps:] + + # Slice future (take first N frames) + if future_len is not None: + future_steps = int(future_len * original_fps) + if future_steps > future_video.shape[0]: + raise ValueError( + f"Requested future_len={future_len}s ({future_steps} frames at {original_fps}Hz) " + f"exceeds available future video ({future_video.shape[0]} frames)" + ) + future_video = future_video[:future_steps] + + # Concatenate sliced portions + video_frames = torch.cat([history_video, future_video], dim=0) # [T,H,W,C] + + # Convert from (T, H, W, C) to (T, C, H, W) + video_tensor = video_frames.permute(0, 3, 1, 2) # [T,C,H,W] + + # Resize and Crop if resolution is provided + if resolution is not None: + T, C, H, W = video_tensor.shape + # get_target_size_and_crop expects (resolution, current_H, current_W) + new_H, new_W, target_canvas_H, target_canvas_W = get_target_size_and_crop(resolution, H, W) + + # Resize if needed + if new_H != H or new_W != W: + video_tensor = F.resize( + video_tensor, [new_H, new_W], interpolation=F.InterpolationMode.BICUBIC, antialias=True + ) + + # Center Crop + if new_H != target_canvas_H or new_W != target_canvas_W: + video_tensor = F.center_crop(video_tensor, [target_canvas_H, target_canvas_W]) + + # Convert to uint8 if not already + if video_tensor.dtype != torch.uint8: + video_tensor = video_tensor.to(torch.uint8) + + # Convert from (T, C, H, W) to (C, T, H, W) + video_tensor = video_tensor.permute(1, 0, 2, 3) # [C,T,H,W] + + return video_tensor, original_fps + + +# 3x3 rotation from car convention (x=forward, y=left, z=up) +# to OpenCV convention (x=right, y=down, z=forward). +# Mapping: new_x = -old_y, new_y = -old_z, new_z = old_x +CAR_TO_OPENCV_ROTATION = np.array( + [[0, -1, 0], [0, 0, -1], [1, 0, 0]], + dtype=np.float32, +) + + +def process_action_trajectory( + action_data: dict, + history_len: float | None = None, + future_len: float | None = None, + fps: int = 10, + rotation_format: Literal["9D", "rot6d", "quat_xyzw", "euler_xyz"] = "9D", + pose_convention: Literal["backward_anchored", "backward_framewise"] = ("backward_framewise"), + scale: float = 1.0, + rotation_scale: float = 1.0, + max_translation_norm: float | None = None, + align_opencv_pose: bool = False, +): + """Process action trajectories from action data dict. + + Args: + action_data: Dict with: + - history_xyz: (T_hist, 3) tensor - position history + - history_quat: (T_hist, 4) tensor - quaternion history + - future_xyz: (T_fut, 3) tensor - position future + - future_quat: (T_fut, 4) tensor - quaternion future + history_len: Desired history length in seconds. + future_len: Desired future length in seconds. + fps: Frames per second, used to compute number of steps from time durations. + align_opencv_pose: If True, transform poses from car convention + (x=forward, y=left, z=up) to OpenCV convention (x=right, y=down, z=forward). + NOTE: av_v2_* data is already in OpenCV convention, DO NOT apply this transformation! + + Returns: + Tuple of (history_action, future_action). + Both actions are torch.Tensor of shape (T, 7) in [x, y, z, qw, qx, qy, qz] format. + + Note: + History steps = history_len * fps, same as future steps = future_len * fps. + For example, with history_len=1.0s and fps=10, we get 10 history steps. + """ + # Extract and ensure tensors + history_xyz = action_data["history_xyz"] + history_quat = action_data["history_quat"] + future_xyz = action_data["future_xyz"] + future_quat = action_data["future_quat"] + + # Convert to tensors if needed + if not isinstance(history_xyz, torch.Tensor): + history_xyz = torch.tensor(history_xyz, dtype=torch.float32) # [T_hist,3] + if not isinstance(history_quat, torch.Tensor): + history_quat = torch.tensor(history_quat, dtype=torch.float32) # [T_hist,4] + if not isinstance(future_xyz, torch.Tensor): + future_xyz = torch.tensor(future_xyz, dtype=torch.float32) # [T_fut,3] + if not isinstance(future_quat, torch.Tensor): + future_quat = torch.tensor(future_quat, dtype=torch.float32) # [T_fut,4] + + # Slice history to desired length (take the last N steps) + if history_len is not None: + history_steps = int(history_len * fps) + available_history = history_xyz.shape[0] + if history_steps > available_history: + raise ValueError( + f"Requested history_len={history_len}s ({history_steps} steps at {fps}Hz) " + f"exceeds available history ({available_history} steps)" + ) + history_xyz = history_xyz[-history_steps:] + history_quat = history_quat[-history_steps:] + + # Slice future to desired length (take the first N steps) + if future_len is not None: + future_steps = int(future_len * fps) + available_future = future_xyz.shape[0] + if future_steps > available_future: + raise ValueError( + f"Requested future_len={future_len}s ({future_steps} steps at {fps}Hz) " + f"exceeds available future ({available_future} steps)" + ) + future_xyz = future_xyz[:future_steps] + future_quat = future_quat[:future_steps] + + # Concatenate to form full trajectory + # history_xyz: (T_hist, 3) + # history_quat: (T_hist, 4) [w, x, y, z] + all_xyz = torch.cat([history_xyz, future_xyz], dim=0) # [T,3] + all_quat = torch.cat([history_quat, future_quat], dim=0) # [T,4] + + poses_abs = build_abs_pose_from_components( + all_xyz, + all_quat, + "quat_wxyz", + ) + + if align_opencv_pose: + poses_abs[:, :3, :3] = poses_abs[:, :3, :3] @ CAR_TO_OPENCV_ROTATION.T + + actions = pose_abs_to_rel( + poses_abs, + rotation_format=rotation_format, + pose_convention=pose_convention, + translation_scale=scale, + rotation_scale=rotation_scale, + ) + + if max_translation_norm is not None: + trans_norms = np.linalg.norm(actions[:, :3], axis=1) + if trans_norms.max() > max_translation_norm: + return None + + actions = torch.from_numpy(actions) # [T-1,action_dim] + + # Split back + # history_action has one less action than history_xyz because the first action is the initial pose + history_action = actions[: len(history_xyz) - 1] + future_action = actions[len(history_xyz) - 1 :] + + return history_action, future_action + + +def add_route_noise( + route: torch.Tensor, + lat_noise_range: float = 0.0, + long_noise_range: float = 0.0, + point_wise_noise: float = 0.0, +) -> torch.Tensor: + """Add noise to route waypoints for data augmentation. + + Applies two types of noise: + 1. Uniform lateral/longitudinal shift (same offset for all waypoints in a sample) + 2. Per-point Gaussian noise (independent per waypoint) + + Both noise types leave the z-axis unchanged. NaN waypoints (padding) are preserved. + + Args: + route: (T, 3) tensor of route waypoints in XYZ. + lat_noise_range: Half-range for uniform lateral (Y) noise. + long_noise_range: Half-range for uniform longitudinal (X) noise. + point_wise_noise: Standard deviation of per-point Gaussian noise. + + Returns: + Noisy route tensor of shape (T, 3). + """ + if lat_noise_range > 0 or long_noise_range > 0: + shift = torch.rand(3) * torch.tensor([2 * long_noise_range, 2 * lat_noise_range, 0.0]) - torch.tensor( # [3] + [long_noise_range, lat_noise_range, 0.0] + ) + route = route + shift[None, :] + + if point_wise_noise > 0: + noise = torch.randn(route.shape[0], 3) * point_wise_noise # [T,3] + noise[..., -1] = 0.0 + route = route + noise + + return route + + +def apply_route_dropout( + route: torch.Tensor, + dropout_rate: float = 0.5, + tail_dropout_rate: float = 0.3, +) -> torch.Tensor: + """Apply dropout masking to route waypoints for data augmentation. + + Three dropout behaviours, applied sequentially: + 1. With probability ``dropout_rate``, mask **all** waypoints. + Otherwise, randomly mask the first K waypoints (K ~ Uniform(0, T)). + 2. With probability ``tail_dropout_rate``, additionally mask waypoints + from a random index in [T//2, T) onward. + + Masked waypoints are set to NaN so downstream code can detect padding + via ``torch.isnan``. + + Args: + route: (T, 3) tensor of route waypoints. + dropout_rate: Probability of fully disabling the route. + tail_dropout_rate: Probability of additional tail dropout. + + Returns: + Route tensor with dropout applied, shape (T, 3). + """ + T = route.shape[0] + mask = torch.isnan(route[..., 0]) # [T] existing padding + + if random.uniform(0, 1) < dropout_rate: + dropout_mask = torch.ones(T, dtype=torch.bool) # [T] + else: + dropout_idx = random.randint(0, T) + dropout_mask = torch.arange(T) < dropout_idx # [T] + + if random.uniform(0, 1) < tail_dropout_rate: + tail_idx = random.randint(T // 2, T - 1) if T > 1 else 0 + tail_dropout_mask = torch.arange(T) >= tail_idx # [T] + dropout_mask = dropout_mask | tail_dropout_mask + + mask = mask | dropout_mask + route = route.clone() + route[mask] = float("nan") + return route + + +def _classify_displacement(dx: float, dy: float, move_threshold: float = 0.1) -> str: + """Classify a 2D displacement vector into a direction label. + + Uses the angle of the displacement in ego frame (X=forward, Y=left) to + determine the driving direction. + + Args: + dx: Forward displacement (positive = forward). + dy: Lateral displacement (positive = left). + move_threshold: Minimum displacement magnitude (meters) to count as movement. + + Returns: + One of: "go forward", "turn left", "turn right", "go backward", "stay". + """ + dist = math.sqrt(dx * dx + dy * dy) + if dist < move_threshold: + return "stay" + + angle_deg = math.degrees(math.atan2(dy, dx)) + + if -45 <= angle_deg <= 45: + return "go forward" + elif 45 < angle_deg <= 135: + return "turn left" + elif -135 <= angle_deg < -45: + return "turn right" + else: + return "go backward" + + +def classify_trajectory_to_text( + trajectory: torch.Tensor, + move_threshold: float = 0.1, + min_segment_steps: int = 2, +) -> str: + """Classify a trajectory in ego frame into a brief semantic text description. + + Classifies each consecutive point pair independently, groups consecutive + identical labels, filters out noisy short groups, and joins distinct + phases with "then". + + Works with any (T, 3) path in ego frame — route waypoints or pose + positions returned by :func:`compute_future_trajectory_in_ego_frame`. + + Args: + trajectory: (T, 3) tensor of positions in ego frame (X=forward, Y=left, Z=up). + The first point is treated as the starting position. + move_threshold: Minimum per-step displacement (meters) to count as movement. + min_segment_steps: Minimum consecutive steps required for a direction label to + be kept; shorter runs are treated as noise and dropped. + + Returns: + A description such as "go forward", "stay then go forward", + "turn left then go forward", or "stay" when the trajectory is empty + or all NaN. + """ + valid_mask = ~torch.isnan(trajectory[:, 0]) + valid_pts = trajectory[valid_mask] + + if len(valid_pts) < 2: + return "stay" + + # Classify every consecutive point pair + step_labels: list[str] = [] + for i in range(len(valid_pts) - 1): + dx = valid_pts[i + 1, 0].item() - valid_pts[i, 0].item() + dy = valid_pts[i + 1, 1].item() - valid_pts[i, 1].item() + step_labels.append(_classify_displacement(dx, dy, move_threshold)) + + # Group consecutive identical labels with their counts + groups: list[tuple[str, int]] = [] + for label in step_labels: + if groups and label == groups[-1][0]: + groups[-1] = (label, groups[-1][1] + 1) + else: + groups.append((label, 1)) + + # Filter out groups shorter than min_segment_steps to suppress noise + if len(groups) > 1: + filtered = [(lbl, cnt) for lbl, cnt in groups if cnt >= min_segment_steps] + if not filtered: + # All groups are short — keep the longest one + filtered = [max(groups, key=lambda g: g[1])] + groups = filtered + + # Deduplicate consecutive identical labels (may arise after filtering) + result = [groups[0][0]] + for label, _ in groups[1:]: + if label != result[-1]: + result.append(label) + + return " then ".join(result) + + +def compute_future_trajectory_in_ego_frame( + action_data: dict, + history_len: float | None = None, + future_len: float | None = None, + fps: int = 10, +) -> torch.Tensor: + """Compute future trajectory positions in the ego coordinate frame. + + Transforms absolute future xyz positions so that the origin is the last + history pose and axes align with ego frame (X=forward, Y=left, Z=up). + + Args: + action_data: Dict with keys ``history_xyz``, ``history_quat``, + ``future_xyz`` (and optionally ``future_quat``). + history_len: History length in seconds for slicing. If *None*, uses all. + future_len: Future length in seconds for slicing. If *None*, uses all. + fps: Frames per second. + + Returns: + (T, 3) float tensor of future positions in ego frame. + """ + history_xyz = action_data["history_xyz"] + history_quat = action_data["history_quat"] + future_xyz = action_data["future_xyz"] + + if not isinstance(history_xyz, torch.Tensor): + history_xyz = torch.tensor(history_xyz, dtype=torch.float32) + if not isinstance(history_quat, torch.Tensor): + history_quat = torch.tensor(history_quat, dtype=torch.float32) + if not isinstance(future_xyz, torch.Tensor): + future_xyz = torch.tensor(future_xyz, dtype=torch.float32) + + # Slice to match the requested durations + if history_len is not None: + history_steps = int(history_len * fps) + history_xyz = history_xyz[-history_steps:] + history_quat = history_quat[-history_steps:] + if future_len is not None: + future_steps = int(future_len * fps) + future_xyz = future_xyz[:future_steps] + + # Current pose = last history frame + current_pos = history_xyz[-1] # (3,) + current_quat_wxyz = history_quat[-1] # (4,) [w, x, y, z] + + # Scipy expects [x, y, z, w] + quat_xyzw = current_quat_wxyz[[1, 2, 3, 0]].numpy() + rot_world_to_ego = Rotation.from_quat(quat_xyzw).inv() + + # Translate then rotate into ego frame + future_rel = (future_xyz - current_pos[None, :]).numpy() + future_ego = rot_world_to_ego.apply(future_rel).astype(np.float32) + + return torch.from_numpy(future_ego) + + +class AVDataset(IterableDataset): + """AV dataset that reads tar files from S3 using wdinfo.json.""" + + def __init__( + self, + root: str | list[str] = "s3://nv-00-10206-robot/cosmos3_action_data/av_v2_02182026_wdinfo/", + credential_path: str = "credentials/gcp_training.secret", + resolution: str | None = None, + fps: int = 10, + mode: str = "policy", + embodiment_type: str = "av", + split: str = "train", + seed: int = 0, + shuffle: bool = True, + history_len: float | None = None, + future_len: float | None = None, + rotation_format: RotationConvention = "rot9d", + pose_convention: Literal["backward_anchored", "backward_framewise"] = ("backward_framewise"), + route_lat_noise_range: float = 0.0, + route_long_noise_range: float = 0.0, + route_point_wise_noise: float = 0.0, + route_dropout: bool = False, + route_dropout_rate: float = 0.0, + route_tail_dropout_rate: float = 0.0, + include_route_in_prompt: bool = True, + use_semantic_route_prompt: bool = False, + translation_scale: float = 1.0, + rotation_scale: float = 1.0, + max_action_translation_norm: float | None = None, + align_opencv_pose: bool = False, + # When True, use a separate domain ID for inverse dynamics / policy modes + # so that DomainAwareLinear learns different projections for anchored (conditioning) + # vs framewise (generation) action representations. + mode_aware_domain: bool = False, + inv_embodiment_type: str = "av_inv", + ): + """Initialize AVDataset. + + Args: + root: S3 path (or list of S3 paths) to wdinfo directories containing train/val subdirectories with wdinfo.json files. + credential_path: Path to JSON file containing S3 credentials. + resolution: Target resolution for video frames (e.g. "256", "480"). If None, keeps original resolution. + fps: Target frames per second for video and actions. + mode: Training mode ('policy', 'forward_dynamics', 'inverse_dynamics', 'image2video', 'joint'). + embodiment_type: Embodiment type for domain ID. + split: Dataset split ('train', 'val', or 'full'). + seed: Random seed for shuffling. + shuffle: Whether to shuffle tar files during iteration (for training). + history_len: Desired history length in seconds. If None, uses all available history. + future_len: Desired future length in seconds. If None, uses all available future. + rotation_format: Rotation convention for actions (e.g. "rot9d", "rot6d", "euler_xyz"). + pose_convention: Pose format for actions (e.g. "backward_framewise", "backward_framewise"). + route_lat_noise_range: Half-range for uniform lateral (Y) noise on route waypoints. + route_long_noise_range: Half-range for uniform longitudinal (X) noise on route waypoints. + route_point_wise_noise: Std-dev of per-waypoint Gaussian noise on route. + route_dropout: Whether to apply random waypoint dropout on route during training. + route_dropout_rate: Probability of fully masking the route (used when route_dropout=True). + route_tail_dropout_rate: Probability of additional tail dropout (used when route_dropout=True). + include_route_in_prompt: Whether to include route waypoints as text in the prompt. + use_semantic_route_prompt: When True and include_route_in_prompt is True, replace raw + coordinate waypoints with a brief semantic description (e.g. "go forward then turn left"). + translation_scale: Scale factor applied to the translation block of the encoded action. + rotation_scale: Scale factor applied to the rotation block of the encoded action + (uniform scalar, preserves rotation-block geometry). Pass the same value to + `pose_rel_to_abs` when decoding. + max_action_translation_norm: If set, discard the sample when any per-frame + scaled translation L2 norm exceeds this value. Acts as an outlier + filter to prevent loss spikes from extreme camera motion. + align_opencv_pose: If True, transform pose rotations from car body-frame + convention (x=forward, y=left, z=up) to OpenCV camera convention + (x=right, y=down, z=forward) before computing relative actions. + mode_aware_domain: When True, inverse_dynamics/policy modes use a separate domain ID. + inv_embodiment_type: Embodiment type string for the inverse domain ID. + """ + super().__init__() + + if isinstance(root, str): + root = [root] + self.roots = [r.rstrip("/") for r in root] + self.credential_path = credential_path + self.resolution = resolution + self.fps = fps + self.mode = mode + self.split = split.lower().strip() + self.seed = seed + self.shuffle = shuffle + self._epoch = 0 + self.history_len = history_len + self.future_len = future_len + self.rotation_format: RotationConvention = rotation_format + self.pose_convention: Literal["absolute", "backward_anchored", "backward_framewise"] = pose_convention + self.route_lat_noise_range = route_lat_noise_range + self.route_long_noise_range = route_long_noise_range + self.route_point_wise_noise = route_point_wise_noise + self.route_dropout = route_dropout + self.route_dropout_rate = route_dropout_rate + self.route_tail_dropout_rate = route_tail_dropout_rate + self.include_route_in_prompt = include_route_in_prompt + self.use_semantic_route_prompt = use_semantic_route_prompt + self.translation_scale = translation_scale + self.rotation_scale = rotation_scale + self.max_action_translation_norm = max_action_translation_norm + self.align_opencv_pose = align_opencv_pose + # Get domain ID for this embodiment + self.domain_id = get_domain_id(embodiment_type) + self.mode_aware_domain = mode_aware_domain + self.domain_id_inv = get_domain_id(inv_embodiment_type) if mode_aware_domain else self.domain_id + + # Validate mode + valid_modes = ["joint", "forward_dynamics", "inverse_dynamics", "policy", "image2video"] + if mode not in valid_modes: + raise ValueError(f"mode must be one of {valid_modes}, got {mode}") + + # Validate split + if self.split not in {"train", "val", "valid", "validation", "eval", "test", "full"}: + raise ValueError(f"Unsupported {split=}. Use train/val/full.") + + # Validate S3 roots + for r in self.roots: + if not r.startswith("s3://"): + raise ValueError(f"root must be an S3 path starting with 's3://', got: {r}") + + # Configure S3 backend using easy_io + self._setup_s3_backend() + + # Load tar files from wdinfo.json + self._tar_files: list[str] = [] + self._total_key_count: int = 0 + self._chunk_size: int = 10 + + self._load_wdinfo() + + log.info( + f"Initialized AVDataset: root={self.roots}, split={self.split}, " + f"resolution={resolution}, fps={fps}, mode={mode}, " + f"num_tar_files={len(self._tar_files)}, " + f"total_samples={self._total_key_count}" + ) + + def _setup_s3_backend(self) -> None: + """Configure the easy_io S3 backend. Called in __init__ and __iter__ for worker processes.""" + easy_io.set_s3_backend( + backend_args={ + "backend": "s3", + "path_mapping": None, + "s3_credential_path": self.credential_path, + } + ) + + def _load_wdinfo(self) -> None: + """Load wdinfo.json for the current split from all roots and build tar file list. + + Supports two directory layouts per root: + - Split-based: ``{root}/train/wdinfo.json``, ``{root}/val/wdinfo.json`` + - Flat: ``{root}/wdinfo.json`` (treated as train-only) + + Split-based paths are tried first; the flat path is used as a fallback + only when no split-based wdinfo was found and the requested split + includes "train". + """ + self._tar_files = [] + self._total_key_count = 0 + + for root in self.roots: + bucket = root.replace("s3://", "").split("/")[0] + + # Determine which splits we need + if self.split in {"val", "valid", "validation", "eval", "test"}: + target_splits = ["val"] + elif self.split == "train": + target_splits = ["train"] + elif self.split == "full": + target_splits = ["train", "val"] + else: + raise ValueError(f"Unsupported split: {self.split}") + + # Try split-based layout first ({root}/train/wdinfo.json, {root}/val/wdinfo.json) + wdinfo_entries: list[tuple[str, dict]] = [] + for split_name in target_splits: + split_path = f"{root}/{split_name}/wdinfo.json" + try: + wdinfo_entries.append((split_path, json.loads(easy_io.get(split_path)))) + except Exception: + pass + + # Fall back to flat layout ({root}/wdinfo.json, treated as train-only) + if not wdinfo_entries and "train" in target_splits: + flat_path = f"{root}/wdinfo.json" + try: + wdinfo_entries.append((flat_path, json.loads(easy_io.get(flat_path)))) + except Exception: + pass + + if not wdinfo_entries: + log.warning(f"No wdinfo.json found for root={root}, split={self.split}") + + for wdinfo_path, wdinfo in wdinfo_entries: + log.info(f"Loading wdinfo from: {wdinfo_path}") + + # Extract metadata + self._chunk_size = wdinfo.get("chunk_size", 10) + data_root = wdinfo.get("root", "") + data_list = wdinfo.get("data_list", []) + + if not data_list: + log.warning(f"No tar files found in wdinfo: {wdinfo_path}") + continue + + # Reconstruct full S3 paths for tar files + tar_root = f"s3://{bucket}/{data_root}".rstrip("/") + tar_paths = [f"{tar_root}/{filename}" for filename in data_list] + self._tar_files.extend(tar_paths) + + # Accumulate total sample count + self._total_key_count += wdinfo.get("total_key_count", len(data_list) * self._chunk_size) + + log.info( + f"Loaded {len(data_list)} tar files from wdinfo, " + f"with {wdinfo.get('total_key_count', len(data_list) * self._chunk_size)} samples" + ) + + if not self._tar_files: + raise RuntimeError(f"No tar files found in wdinfo at {self.roots}") + + def __len__(self) -> int: + """Return the estimated number of samples in the current split.""" + return self._total_key_count + + def _process_sample(self, pkl_data: dict, key: str, global_idx: int) -> dict | None: + """Process a single sample from pkl data. + + Args: + pkl_data: Dictionary with 'video' (bytes) and 'action' (pickled dict). + key: Sample key (basename without .pkl). + global_idx: Global sample index for __key__. + + Returns: + Processed sample dictionary, or None if the sample should be discarded. + """ + # Extract video bytes + video_bytes = pkl_data.get("video") + if video_bytes is None: + raise RuntimeError(f"No video found for key {key}") + + # Extract action data + action_bytes = pkl_data.get("action") + if action_bytes is None: + raise RuntimeError(f"Missing action data for key {key}") + + action_data = pickle.loads(action_bytes) + + # Extract route data + route_bytes = pkl_data.get("route") + if route_bytes is not None: + route_data = pickle.loads(route_bytes) + if not isinstance(route_data, torch.Tensor): + route = torch.tensor(route_data, dtype=torch.float32) # [num_waypoints,3] + else: + route = route_data.float() # [num_waypoints,3] + else: + log.warning(f"No route found for key {key}") + route = torch.full((20, 3), float("nan")) # [20,3] + + # Apply route augmentations during training + if self.split == "train": + route = add_route_noise( + route, + lat_noise_range=self.route_lat_noise_range, + long_noise_range=self.route_long_noise_range, + point_wise_noise=self.route_point_wise_noise, + ) + if self.route_dropout: + route = apply_route_dropout( + route, + dropout_rate=self.route_dropout_rate, + tail_dropout_rate=self.route_tail_dropout_rate, + ) + + # Get original history frame count for video slicing + original_history_steps = len(action_data["history_xyz"]) + + # Decode video + video, _ = decode_video_bytes( + video_bytes, + resolution=self.resolution, + history_len=self.history_len, + future_len=self.future_len, + original_history_steps=original_history_steps, + ) + + # Determine mode for this sample + if self.mode == "joint": + mode = random.choice(["forward_dynamics", "inverse_dynamics", "policy"]) + # mode = random.choice(["policy", "image2video"]) + else: + mode = self.mode + + # Process actions + action_result = process_action_trajectory( + action_data, + history_len=self.history_len, + future_len=self.future_len, + fps=self.fps, + rotation_format=self.rotation_format, + pose_convention=self.pose_convention, + scale=self.translation_scale, + rotation_scale=self.rotation_scale, + max_translation_norm=self.max_action_translation_norm, + align_opencv_pose=self.align_opencv_pose, + ) + if action_result is None: + return None + history_action, future_action = action_result + + # Combine and pad actions + combined_action = torch.cat([history_action, future_action], dim=0) # [T_hist+T_fut,action_dim] + + # FPS as tensor + fps_tensor = torch.tensor(self.fps, dtype=torch.long) # scalar + + # Key as tensor + key_tensor = torch.tensor([global_idx], dtype=torch.long) # [1] + + # Compute actual history/future lengths from data + actual_history_length = history_action.shape[0] + actual_future_length = future_action.shape[0] + + # Generate prompt based on actual data lengths + history_duration = actual_history_length / self.fps + future_duration = actual_future_length / self.fps + + prompt = "You are an autonomous vehicle planning system. " + if self.include_route_in_prompt and mode == "policy": # only include route in prompt for policy mode + if self.use_semantic_route_prompt: + future_ego = compute_future_trajectory_in_ego_frame( + action_data, self.history_len, self.future_len, self.fps + ) + trajectory_desc = classify_trajectory_to_text(future_ego) + prompt += f"Please {trajectory_desc}. " + else: + num_waypoints = route.shape[0] + waypoints_str = ", ".join( + "nan" if torch.isnan(wp[0]) else f"({wp[0]:.2f}, {wp[1]:.2f}, {wp[2]:.2f})" for wp in route + ) + prompt += ( + f"The navigation route has {num_waypoints} waypoints " + f"(XYZ in ego frame with X=forward, Y=left, Z=up): " + f"[{waypoints_str}]. A nan waypoint means that waypoint is not available. " + ) + # prompt += f"Predict the future {future_duration:.1f}s action trajectory at {self.fps}Hz." + + # Select domain ID: use inverse domain for generation modes when mode_aware_domain is on + if self.mode_aware_domain and mode in ["inverse_dynamics", "policy"]: + domain_id = self.domain_id_inv + else: + domain_id = self.domain_id + + sample = { + "video": video, + "action": combined_action, + "action_history": history_action, + "action_future": future_action, + "route": route, + "conditioning_fps": fps_tensor, + "prompt": prompt, + "ai_caption": prompt, + "mode": mode, + "__key__": key_tensor, + "domain_id": torch.tensor(domain_id, dtype=torch.long), + "history_length": actual_history_length, + "future_length": actual_future_length, + "viewpoint": "ego_view", + } + return sample + + def __iter__(self) -> Iterator[dict[str, torch.Tensor | str | int]]: + """Iterate over the dataset, loading tar files from S3.""" + # Re-configure S3 backend in case this is running in a worker process after unpickling + self._setup_s3_backend() + + # Optionally shuffle tar files for training + tar_files = list(self._tar_files) + if self.shuffle: + rng = random.Random(self.seed + self._epoch) + rng.shuffle(tar_files) + self._epoch += 1 + + global_idx = 0 + + for tar_path in tar_files: + try: + # Read tar file bytes using easy_io + tar_bytes = easy_io.get(tar_path) + + with tarfile.open(fileobj=io.BytesIO(tar_bytes), mode="r:*") as tar: + for member in tar.getmembers(): + if not member.name.endswith(".pkl"): + continue + + try: + # Extract and process the sample + f_member = tar.extractfile(member) + if f_member is None: + log.warning(f"Failed to extract {member.name} from {tar_path}") + continue + + try: + pkl_data = pickle.load(f_member) + finally: + f_member.close() + + key = member.name.rsplit(".", 1)[0] + + sample = self._process_sample(pkl_data, key, global_idx) + if sample is not None: + yield sample + global_idx += 1 + + except Exception as e: + log.warning(f"Failed to process sample {member.name} from {tar_path}: {e}") + continue + + except Exception as e: + log.warning(f"Failed to read tar file {tar_path}: {e}") + continue + + +# PYTHONPATH=. python cosmos_framework/data/vfm/action/av_dataset.py +if __name__ == "__main__": + import json as _json + import os + import time + + import torchvision + + from cosmos_framework.data.vfm.action.pose_utils import pose_rel_to_abs + + _ACTION_SCALE = 1.35 + _ROTATION_SCALE = 1.0 + _ROTATION_FORMAT = "rot6d" + _POSE_CONVENTION = "backward_framewise" + + dataset = AVDataset( + root=[ + # "s3://nv-00-10206-robot/cosmos3_action_data/av_02182026_wdinfo/", + # "s3://nv-00-10206-robot/cosmos3_action_data/av_03292026_wdinfo/", + "s3://nv-00-10206-robot/cosmos3_action_data/av_v2_02182026_wdinfo/", + "s3://nv-00-10206-robot/cosmos3_action_data/av_v2_03292026_wdinfo/", + ], + split="train", + shuffle=True, + fps=10, + mode="inverse_dynamics", + history_len=0.1, + future_len=6.0, + rotation_format=_ROTATION_FORMAT, + pose_convention=_POSE_CONVENTION, + translation_scale=_ACTION_SCALE, + rotation_scale=_ROTATION_SCALE, + resolution="480", + include_route_in_prompt=False, + use_semantic_route_prompt=False, + # align_opencv_pose=False, + ) + dataset_iter = iter(dataset) + os.makedirs("temp", exist_ok=True) + + for i in range(5): + print(f"==================== Sample {i} ====================") + _t0 = time.time() + data = next(dataset_iter) + _t1 = time.time() + print(f"{'Loading time':<25}: {_t1 - _t0:.2f}s") + + print(f"{'video shape':<25}: {data['video'].shape}") # [C,T,H,W] + print(f"{'action shape':<25}: {data['action'].shape}") # [T,action_dim] + print(f"{'action_history shape':<25}: {data['action_history'].shape}") + print(f"{'action_future shape':<25}: {data['action_future'].shape}") + print(f"{'route shape':<25}: {data['route'].shape}") + print(f"{'history_length':<25}: {data['history_length']}") + print(f"{'future_length':<25}: {data['future_length']}") + print(f"{'conditioning_fps':<25}: {data['conditioning_fps'].item()}") + print(f"{'mode':<25}: {data['mode']}") + print(f"{'domain_id':<25}: {data['domain_id'].item()}") + print(f"{'prompt':<25}: {data['prompt']}") + + # save video + video = data["video"].permute(1, 0, 2, 3) # [C,T,H,W] -> [T,C,H,W] + video_path = f"temp/av_sample_{i}.mp4" + torchvision.io.write_video( + video_path, video.permute(0, 2, 3, 1).numpy(), fps=data["conditioning_fps"].item() + ) # expects (T, H, W, C) + print(f"Saved video to {video_path}") + + # reconstruct absolute poses from relative actions and save as json + camera_poses = pose_rel_to_abs( + data["action"].float().numpy(), + rotation_format=_ROTATION_FORMAT, + pose_convention=_POSE_CONVENTION, + translation_scale=_ACTION_SCALE, + rotation_scale=_ROTATION_SCALE, + ) + pose_path = f"temp/av_sample_{i}_camera.json" + with open(pose_path, "w") as f: + _json.dump(camera_poses.tolist(), f) + print(f"Saved camera poses to {pose_path}") diff --git a/cosmos_framework/data/vfm/action/bridge_orig_lerobot_dataset.py b/cosmos_framework/data/vfm/action/bridge_orig_lerobot_dataset.py new file mode 100644 index 0000000..4cc0a30 --- /dev/null +++ b/cosmos_framework/data/vfm/action/bridge_orig_lerobot_dataset.py @@ -0,0 +1,272 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# * https://github.com/2toinf/X-VLA/blob/30090f81cf91b15da73af234ce2b098fe20590f8/datasets/domain_handler/simulations.py#L70-L93 +# * https://github.com/2toinf/X-VLA/issues/11 +# * https://github.com/2toinf/X-VLA/issues/33 +# * https://github.com/2toinf/X-VLA/issues/67 +# + +# uses identity stats (q01=-1, q99=1) on the 6D rotation dims 3..8, while +# ``"quantile_rot"`` uses the raw stats and normalizes those columns too. + +from typing import Any + +import numpy as np +import torch +from lerobot.datasets.lerobot_dataset import LeRobotDatasetMetadata + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.cosmos3_action_lerobot import ( + ActionNormalization, + ActionSpec, + BaseActionLeRobotDataset, + Gripper, + Pos, + Rot, + build_action_spec, +) +from cosmos_framework.data.vfm.action.pose_utils import ( + PoseConvention, + build_abs_pose_from_components, + convert_rotation, + pose_abs_to_rel, +) +from cosmos_framework.data.vfm.action.viewpoint_utils import Viewpoint + +# Bridge rotation decomposition: +# 1) _DEFAULT_ROTATION: raw bridge state → kinematics (MJCF/URDF) frame. +# The WidowX controller records ``R_state = R_fk @ DEFAULT_ROTATION.T``, +# so ``R_fk = R_state @ DEFAULT_ROTATION``. +# 2) _TCP_TO_FLANGE: re-reference from ee_gripper_link to gripper_link +# (pure translation in kinematics frame). See block below. +# 3) _KIN_TO_OPENCV: kinematics → OpenCV convention (for training/vis). +# The viewer undoes this before IK to recover the kinematic frame. +_DEFAULT_ROTATION = np.array( + [[0, 0, 1], [0, 1, 0], [-1, 0, 0]], + dtype=np.float32, +) +_BRIDGE_TO_OPENCV = np.array( + [[0, 0, 1], [-1, 0, 0], [0, -1, 0]], + dtype=np.float32, +) + +# --------------------------------------------------------------------------- +# TCP → flange (gripper body) offset +# --------------------------------------------------------------------------- +# The bridge dataset records EE poses at ``ee_gripper_link`` — the Interbotix +# SDK's end-effector reference, 93.6 mm past the wrist rotate body +# (``gripper_link``), roughly at the grasp center between the finger pads. +# For action learning we re-reference poses to the *wrist rotate body* +# (``gripper_link``) because: +# 1. It is the last actuated link — its pose is fully determined by joint +# angles, with no dependence on finger opening. +# 2. The ~10 cm offset reduces the lever-arm effect of small rotation +# errors on position accuracy. +# 3. Consistent with Google Robot, where we also target the gripper body. +# +# The constant below is the SE(3) transform from ``ee_gripper_link`` to +# ``gripper_link``, computed from the SimplerEnv URDF via pinocchio FK at the +# neutral configuration: +# T = oMf[ee_gripper_link]⁻¹ · oMf[gripper_link] +# It is pure translation (identity rotation) — the two frames share the +# same orientation by construction (connected via fixed joints with no +# rotational offset). +# +# Source URDF: https://github.com/simpler-env/ManiSkill2_real2sim +# → mani_skill2_real2sim/assets/descriptions/widowx_description/ +# + +# so the translation is expressed in the kinematic (MJCF) frame. +# fmt: off +_TCP_TO_FLANGE = np.array([ + [+1.0000000000, +0.0000000000, +0.0000000000, -0.0935750000], + [+0.0000000000, +1.0000000000, +0.0000000000, +0.0000000000], + [+0.0000000000, +0.0000000000, +1.0000000000, +0.0000000000], + [+0.0000000000, +0.0000000000, +0.0000000000, +1.0000000000], +], dtype=np.float32) +# fmt: on + + +class BridgeOrigLeRobotDataset(BaseActionLeRobotDataset): + """ """ + + def __init__( + self, + root: str = "", + fps: float = 5.0, + chunk_length: int = 16, + split_seed: int = 42, + split_val_ratio: float = 0.05, + split: str = "train", + mode: str = "policy", + pose_convention: PoseConvention = "backward_framewise", + action_normalization: ActionNormalization | None = None, + viewpoint: Viewpoint = "ego_view", + enable_fast_init: bool = False, + ) -> None: + """ """ + super().__init__( + fps=fps, + chunk_length=chunk_length, + split_seed=split_seed, + split_val_ratio=split_val_ratio, + split=split, + mode=mode, + embodiment_type="bridge_orig_lerobot", + viewpoint=viewpoint, + pose_convention=pose_convention, + rotation_format="rot6d", + action_normalization=action_normalization, + tolerance_s=1e-4, + enable_fast_init=enable_fast_init, + ) + # _to_opencv is the kinematics→OpenCV part only. + # The viewer undoes this before IK → recovers kinematic frame directly. + self._to_opencv = _BRIDGE_TO_OPENCV + + self._all_shard_roots = [root] + + self._delta_timestamps = { + "observation.images.image_0": [i * self._dt for i in range(0, self._chunk_length + 1)], + "observation.state": [i * self._dt for i in range(0, self._chunk_length + 1)], + "action": [i * self._dt for i in range(0, self._chunk_length)], + } + + # ------------------------------------------------------------------ + # Action computation + # ------------------------------------------------------------------ + + def _compute_absolute_action(self, sample: dict[str, Any]) -> tuple[torch.Tensor, torch.Tensor]: + """Absolute action from state + gripper from action. + + EEF xyz+rotation come from observation.state[1:]; gripper from action[:, 6]. + + Returns: + (action_tensor, initial_pose) — initial_pose is the first-frame + absolute EE pose (4×4, in the corrected OpenCV frame). + """ + state = sample["observation.state"][1:] # [T, 8] + poses_abs = build_abs_pose_from_components( + state[:, 0:3], + state[:, 3:6], + "euler_xyz", + ) + + # 1. Raw → kinematics: apply DEFAULT_ROTATION + # 2. TCP → flange: shift from ee_gripper_link to gripper_link + # 3. Kinematics → OpenCV convention (rotation only) + poses_abs[:, :3, :3] = poses_abs[:, :3, :3] @ _DEFAULT_ROTATION.astype(poses_abs.dtype) + poses_abs = poses_abs @ _TCP_TO_FLANGE.astype(poses_abs.dtype) + poses_abs[:, :3, :3] = poses_abs[:, :3, :3] @ self._to_opencv.astype(poses_abs.dtype) + + initial_pose = torch.from_numpy(poses_abs[0].copy()).float() + + translation = torch.from_numpy(poses_abs[:, :3, 3]).float() + rotation_matrix = torch.from_numpy(poses_abs[:, :3, :3]).float() + rotation = convert_rotation(rotation_matrix, input_format="matrix", output_format="rot6d").float() + + pose = torch.cat([translation, rotation], dim=-1) # [T, 9] + return torch.cat([pose, sample["action"][:, [6]]], dim=-1), initial_pose # [T, 10] + + def _compute_backward_framewise_action(self, sample: dict[str, Any]) -> tuple[torch.Tensor, torch.Tensor]: + """Body-frame (ego-frame) delta: ``T_curr^{-1} @ T_next``. + + Matches Camera/AV ``backward_framewise`` convention. Translation is in + the current frame's local coordinate system; rotation is + ``R_curr^{-1} @ R_next``. + + Returns: + (action_tensor, initial_pose) — initial_pose is the first-frame + absolute EE pose (4×4, in the corrected OpenCV frame). + """ + states = sample["observation.state"] # (chunk_length + 1, 8) + poses_abs = build_abs_pose_from_components( + states[:, 0:3], + states[:, 3:6], + "euler_xyz", + ) + + # 1. Raw → kinematics: apply DEFAULT_ROTATION + # 2. TCP → flange: shift from ee_gripper_link to gripper_link + # 3. Kinematics → OpenCV convention (rotation only) + poses_abs[:, :3, :3] = poses_abs[:, :3, :3] @ _DEFAULT_ROTATION.astype(poses_abs.dtype) + poses_abs = poses_abs @ _TCP_TO_FLANGE.astype(poses_abs.dtype) + poses_abs[:, :3, :3] = poses_abs[:, :3, :3] @ self._to_opencv.astype(poses_abs.dtype) + + initial_pose = torch.from_numpy(poses_abs[0].copy()).float() + + poses_rel = pose_abs_to_rel( + poses_abs=poses_abs, + rotation_format="rot6d", + pose_convention="backward_framewise", + ) + poses_rel_tensor = torch.from_numpy(poses_rel).float() + + return torch.cat([poses_rel_tensor, sample["action"][:, [6]]], dim=-1), initial_pose + + # ------------------------------------------------------------------ + # Normalization is handled by BaseActionLeRobotDataset. + # Stats are loaded from: + # cosmos_framework/data/vfm/action/normalizers/ + # bridge_orig_lerobot__.json + # Regenerate via ``compute_action_stats.py`` + ``debug/stats_all.sh``. + # ------------------------------------------------------------------ + + # ------------------------------------------------------------------ + # Episode filtering + # ------------------------------------------------------------------ + def _filter_valid_episodes(self, meta: LeRobotDatasetMetadata, episode_ids: list[int]) -> list[int]: + """Drop episodes whose ``tasks`` metadata is empty/whitespace. + + Narrower than the offline + ``projects/cosmos3/vfm/datasets/action/filter_bridge_dataset.py`` + (which also flags gibberish/question/non-English/patterns via + ``classify_task``). + """ + kept: list[int] = [] + dropped = 0 + for ep_id in episode_ids: + ep = meta.episodes[ep_id] + tasks = ep.get("tasks", []) + if isinstance(tasks, str): + tasks = [tasks] + has_prompt = any(t and str(t).strip() for t in (tasks or [])) + if has_prompt: + kept.append(ep_id) + else: + dropped += 1 + if dropped: + log.info(f"BridgeOrigLeRobotDataset: dropped {dropped} / {len(episode_ids)} episodes with empty prompt") + return kept + + # ------------------------------------------------------------------ + # __getitem__ + # ------------------------------------------------------------------ + + def _build_action_spec(self) -> ActionSpec: + """Bridge: 10D = ``[Pos, Rot6d, Gripper]``.""" + return build_action_spec(Pos(), Rot("rot6d"), Gripper()) + + def __getitem__(self, idx: int) -> dict[str, Any]: + """ """ + mode, _, _, sample = self._fetch_sample(idx) + + ai_caption = sample["task"] + + video = sample["observation.images.image_0"] # [T,C,H,W] + if self._pose_convention == "absolute": + action, initial_pose = self._compute_absolute_action(sample) + elif self._pose_convention == "backward_framewise": + action, initial_pose = self._compute_backward_framewise_action(sample) + else: + raise ValueError(f"Unknown pose_convention: {self._pose_convention}") + + return self._build_result( + mode=mode, video=video, action=action, ai_caption=ai_caption, initial_pose=initial_pose + ) + + @property + def action_dim(self) -> int: + """Action dimensionality: position(3) + 6D rotation(6) + gripper(1) = 10.""" + return 10 diff --git a/cosmos_framework/data/vfm/action/camera_dataset.py b/cosmos_framework/data/vfm/action/camera_dataset.py new file mode 100644 index 0000000..46acab9 --- /dev/null +++ b/cosmos_framework/data/vfm/action/camera_dataset.py @@ -0,0 +1,596 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import os +import random +import time +from dataclasses import dataclass +from functools import partial +from typing import Callable, Literal + +import numpy as np +import torch +import torch.distributed as dist +import torchvision.transforms.functional as F +from torch.utils.data import DataLoader, IterableDataset +from torchcodec.decoders import VideoDecoder + +from imaginaire.modules.camera import Camera +from cosmos_framework.utils import log +from cosmos_framework.utils.easy_io import easy_io +from cosmos_framework.data.vfm.action.domain_utils import get_domain_id +from cosmos_framework.data.vfm.action.pose_utils import ( + RotationConvention, + pose_abs_to_rel, +) +from cosmos_framework.data.vfm.action.unified_dataset import wrap_dataset +from cosmos_framework.data.vfm.joint_dataloader import custom_collate_fn +from cosmos_framework.data.vfm.utils import VIDEO_RES_SIZE_INFO + +""" This load the cosmos3 camera-depth data from s3. +File structure: + +s3://nv-00-10206-robot/cosmos3_action_data//v3/ +- videos/ + - .mp4 + ... +- captions/ + - .json + ... +- cameras/ + - .json + ... +meta.json + +Each video contains N = 150 frames at 30fps. + +For multi-view dataset, the structure is slightly different: +- videos/ + - / + - .mp4 + ... + +The meta.json is like: +{ + "scenes": [ + , + ... + ] +} + +The caption json is like: +{ + "Qwen3-VL-30B-A3B-Instruct": { + "long": "...", + "short": "...", + "medium": "..." + } +} + +The camera json is like: +{ + "camera": { + "focal_length": [fx, fy] * N, + "principal_point": [cx, cy] * N, + "pose_world2cam": [qx, qy, qz, qw, tx, ty, tz] * N + } +} + +Training datasets: +- tartanair: 2245 clips at 480x640 +- endeavor_forever: 40555 clips at 720x1364 +- synhuman_20251218: 24000 clips at 1080x1920 +- drivesim: 41639 scenes (each 7 clips) at 1080x1920 +- pretrain_camera_260131_10k: 42697 clips, various resolution (most are 720x1280), various fps (most are 30). ~half of the camera is static. +- MultiCamVideo: 13600*10 clips at 1280x1280, 30fps +- SyncCamVideo: 6800*10 clips at 1280x1280, 30fps + + +Testing datasets: +- camera_benchmark: 60 clips (various resolution) +- videos_camera_benchmark: 100 clips (1280x720, various fps) +""" + + +@dataclass +class DataConfig: + bucket: str = "nv-00-10206-robot" + credential_path: str = "credentials/gcp_training.secret" + validate: bool = False + validate_num: int = 100 # for each dataset + caption_model: str = "Qwen3-VL-30B-A3B-Instruct" + + + dataset_names: str = "tartanair" + + # Total video frames including the context frame (must be 4n+1 for VAE). + # E.g. 93 = 1 context + 92 action frames. + num_frames: int = 93 + + # resolution group + resolution: str = "256" + + # action mode + mode: str = "forward_dynamics" # camera-control is forward_dynamics + embodiment_type: str = "camera_pose" + # ablation caption + + fix_caption: bool = False + fix_caption_text: str = "The camera moves in a scene." + + # ablation camera action format + rotation_format: RotationConvention = "rot9d" + pose_convention: Literal["backward_anchored", "backward_framewise"] = "backward_framewise" + + translation_scale: float = 1.0 + rotation_scale: float = 1.0 + + # If set, downsample the video to this fps. Must be <= video fps (upsampling is forbidden). + target_fps: float | None = None + + +def get_target_size_and_crop(resolution: str, current_H: int, current_W: int) -> tuple[int, int, int, int]: + """Calculates resize dimensions and crop size for smallest-side resize + center crop.""" + target_resolutions = VIDEO_RES_SIZE_INFO[resolution] + + # Find closest supported aspect ratio to minimize cropping + current_ar = current_W / current_H + best_key = "1,1" + min_diff = float("inf") + + for key in target_resolutions: + w_r, h_r = map(int, key.split(",")) + target_ar = w_r / h_r + diff = abs(current_ar - target_ar) + if diff < min_diff: + min_diff = diff + best_key = key + + target_canvas_W, target_canvas_H = target_resolutions[best_key] + + # Resize logic (ResizeSmallestSideAspectPreserving) + # We want the image to cover the target canvas completely. + scaling_ratio = max(target_canvas_W / current_W, target_canvas_H / current_H) + + new_H = int(scaling_ratio * current_H + 0.5) + new_W = int(scaling_ratio * current_W + 0.5) + + return new_H, new_W, target_canvas_H, target_canvas_W + + +class CameraDataset(IterableDataset): + def __init__(self, conf: DataConfig): + super().__init__() + + self.conf = conf + self.domain_id = get_domain_id(conf.embodiment_type) + + easy_io.set_s3_backend( + backend_args={ + "backend": "s3", + "path_mapping": None, + "s3_credential_path": self.conf.credential_path, + } + ) + + self.uids = [] + dataset_names = conf.dataset_names.split(",") + for dataset_name in dataset_names: + path_uids = f"s3://{conf.bucket}/cosmos3_action_data/{dataset_name}/v3/meta.json" + uids = easy_io.load(path_uids)["scenes"] # list of uids + + # for benchmark dataset, do not split + if dataset_name not in ["camera_benchmark", "videos_camera_benchmark"]: + # train/test split + assert self.conf.validate_num > 0 and len(uids) >= self.conf.validate_num + stride = len(uids) // self.conf.validate_num + val_indices = {i * stride for i in range(self.conf.validate_num)} + if self.conf.validate: + uids = [uids[i] for i in sorted(val_indices)] + else: + uids = [uids[i] for i in range(len(uids)) if i not in val_indices] + + for uid in uids: + self.uids.append((dataset_name, uid)) + + log.warning(f"Loaded {len(self.uids)} uids from {conf.dataset_names}") + + def __len__(self): + return len(self.uids) + + def __iter__(self): + if self.conf.validate: + for i in range(len(self.uids)): + sample = self.load_data(self.uids[i]) + if sample is None: + continue + else: + yield sample + else: + # infinite random loop for training + while True: + indices = np.random.permutation(len(self.uids)) + for i in indices: + sample = self.load_data(self.uids[i]) + if sample is None: + continue + else: + yield sample + + def load_data(self, uid: str) -> dict | None: + """Load and preprocess a single sample. + For multi-view dataset, we randomly load one view at each iteration. + + Args: + uid: Unique identifier for the sample (e.g., 'westerndesert_Hard_P007') + + Returns: + Dictionary with video, action, and metadata, or None if loading fails. + """ + try: + dataset_name, sample_name = uid + + if dataset_name == "drivesim": + # multi-view dataset + view_names = [ + "camera_bev_from_behind_ego", + "camera_bev_looking_back_at_ego", + "camera_crossleft", + "camera_crossright", + "camera_frontwide", + "camera_rearleft", + "camera_rearright", + ] + view_name = random.choice(view_names) if not self.conf.validate else view_names[0] + sample_name = os.path.join(sample_name, view_name) + elif dataset_name in ["MultiCamVideo", "SyncCamVideo"]: + # multi-view dataset + view_names = [ + "cam01", + "cam02", + "cam03", + "cam04", + "cam05", + "cam06", + "cam07", + "cam08", + "cam09", + "cam10", + ] + view_name = random.choice(view_names) if not self.conf.validate else view_names[0] + sample_name = os.path.join(sample_name, view_name) + + video_path = f"s3://{self.conf.bucket}/cosmos3_action_data/{dataset_name}/v3/videos/{sample_name}.mp4" + camera_path = f"s3://{self.conf.bucket}/cosmos3_action_data/{dataset_name}/v3/cameras/{sample_name}.json" + caption_path = f"s3://{self.conf.bucket}/cosmos3_action_data/{dataset_name}/v3/captions/{sample_name}.json" + + # Load video bytes from S3 + video_bytes = easy_io.get(video_path) + + decoder = VideoDecoder(video_bytes, num_ffmpeg_threads=4) + del video_bytes + total_frames = decoder.metadata.num_frames + video_fps = decoder.metadata.average_fps + + # Determine temporal stride for fps downsampling + if self.conf.target_fps is not None: + if self.conf.target_fps > video_fps: + raise ValueError( + f"target_fps ({self.conf.target_fps}) > video_fps ({video_fps}). Upsampling is not supported." + ) + stride = round(video_fps / self.conf.target_fps) + effective_fps = video_fps / stride + else: + stride = 1 + effective_fps = video_fps + + # Sample consecutive frames + if self.conf.num_frames == -1: + # Load all available frames, aligned to VAE constraint (1 + 4*N total frames) + num_strided_frames = (total_frames - 1) // stride + 1 + N = (num_strided_frames - 1) // 4 + if N < 1: + log.warning(f"Not enough frames for {uid}, total_frames: {total_frames}, stride: {stride}") + return None + num_frames_to_load = 1 + 4 * N + start_idx = 0 + else: + num_frames_to_load = self.conf.num_frames + num_raw_frames_needed = (num_frames_to_load - 1) * stride + 1 + + if total_frames < num_raw_frames_needed: + log.warning( + f"Not enough frames to load for {uid}, total_frames: {total_frames}, num_raw_frames_needed: {num_raw_frames_needed}" + ) + return None + + # Random start index for consecutive frame sampling + if self.conf.validate: + start_idx = 0 + else: + start_idx = random.randint(0, total_frames - num_raw_frames_needed) + + num_raw_frames_needed = (num_frames_to_load - 1) * stride + 1 + frame_indices = list(range(start_idx, start_idx + num_raw_frames_needed, stride)) + + # torchcodec returns [T,C,H,W] tensor + frame_batch = decoder.get_frames_at(frame_indices) + video_frames = frame_batch.data # [T,C,H,W] uint8 + del decoder + + # Get target size and crop params + T, C, H, W = video_frames.shape + + # temp: for pretrain_camera_260131_10k, we only use videos with at 720x1280 to enable batched training + if dataset_name == "pretrain_camera_260131_10k": + assert H == 720 and W == 1280, f"Expected resolution 720x1280, got {H}x{W}" + + new_H, new_W, target_canvas_H, target_canvas_W = get_target_size_and_crop(self.conf.resolution, H, W) + + # Resize if needed + if new_H != H or new_W != W: + video_frames = F.resize( + video_frames, [new_H, new_W], interpolation=F.InterpolationMode.BICUBIC, antialias=True + ) + + # Center Crop + if new_H != target_canvas_H or new_W != target_canvas_W: + video_frames = F.center_crop(video_frames, [target_canvas_H, target_canvas_W]) + + # Convert to (C, T, H, W) format expected by model + video = video_frames.permute(1, 0, 2, 3) # [C,T,H,W] + + # Load camera data + camera_data = easy_io.load(camera_path) + w2c = np.array(camera_data["camera"]["pose_world2cam"]).reshape(-1, 7) # [N,7] + assert w2c.shape[0] == total_frames, f"Expected {total_frames} poses, got {w2c.shape[0]}" + + # Get w2c for the sampled frames (same stride as video frames) + w2c = w2c[start_idx : start_idx + num_raw_frames_needed : stride] # [num_frames,7] + # Convert (qx,qy,qz,qw,tx,ty,tz) to [R|t] matrices + w2c = Camera.extrinsic_params_to_matrices(w2c) # [num_frames,3,4] + w2c_homo = np.eye(4, dtype=np.float32)[None, :, :].repeat(w2c.shape[0], axis=0) # [num_frames,4,4] + w2c_homo[:, :3, :] = w2c + c2w_homo = np.linalg.inv(w2c_homo) + + # Determine mode + if self.conf.mode == "joint": + mode = random.choice(["forward_dynamics", "inverse_dynamics", "policy"]) + else: + mode = self.conf.mode + + # Using c2w_homo because pose_abs_to_rel expects c2w poses + # The function handles w2c/c2w/anchored logic based on pose_convention + action = pose_abs_to_rel( + c2w_homo, + rotation_format=self.conf.rotation_format, + pose_convention=self.conf.pose_convention, + translation_scale=self.conf.translation_scale, + rotation_scale=self.conf.rotation_scale, + ) + action = torch.from_numpy(action) # [num_frames-1,action_dim] + + # Load caption data + if self.conf.fix_caption: + caption = self.conf.fix_caption_text + else: + caption_data = easy_io.load(caption_path) + captions = caption_data[self.conf.caption_model] + # Randomly select from long/short/medium captions + if self.conf.validate: + caption = captions["long"] + else: + selected_type = random.choice(["long", "short", "medium"]) + caption = captions[selected_type] + + # FPS (must be scalar fps, not [fps]) + fps = torch.tensor(effective_fps, dtype=torch.float32) # scalar + + # Build sample dict matching dummy_dataset format + sample = { + "video": video, # [C,num_frames,H,W] uint8 + "action": action, # [num_frames-1,max_action_dim] float32 + "conditioning_fps": fps, # scalar float32 + "ai_caption": caption, + "mode": mode, + "__key__": torch.tensor([hash(uid) % (2**31)], dtype=torch.long), + "domain_id": torch.tensor(self.domain_id, dtype=torch.long), + "viewpoint": "ego_view", + } + return sample + + except Exception as e: + log.warning(f"Error loading {uid}: {e}") + return None # skip this sample + + +# to align with other robot datasets interface. +class CameraDatasetPatched(CameraDataset): + def __init__( + self, + dataset_names: str, + split: Literal["train", "val"] = "train", + num_frames: int = 93, + resolution: str = "256", + mode: str = "forward_dynamics", + fix_caption: bool = False, + rotation_format: RotationConvention = "rot9d", + pose_convention: Literal["backward_anchored", "backward_framewise"] = ("backward_framewise"), + translation_scale: float = 1.0, + rotation_scale: float = 1.0, + shuffle: bool = False, # not used, decided by split + ): + super().__init__( + conf=DataConfig( + dataset_names=dataset_names, + validate=split == "val", + num_frames=num_frames, + resolution=resolution, + mode=mode, + fix_caption=fix_caption, + rotation_format=rotation_format, + pose_convention=pose_convention, + translation_scale=translation_scale, + rotation_scale=rotation_scale, + ) + ) + self.dataset_names = dataset_names + self.split = split + self.num_frames = num_frames + self.resolution = resolution + self.mode = mode + self.fix_caption = fix_caption + self.rotation_format = rotation_format + self.pose_convention = pose_convention + self.translation_scale = translation_scale + self.rotation_scale = rotation_scale + + +def worker_init_fn(worker_id: int, seed: int = 0): + try: + rank = dist.get_rank() + except Exception: + rank = 0 + seed = seed + rank * 100000 + worker_id + np.random.seed(seed) + torch.manual_seed(seed) + random.seed(seed) + + +class MixedDataLoader: + def __init__(self, dataloaders: list[DataLoader], ratios: list[float]): + self.dataloaders = dataloaders + self.ratios = ratios + + def __len__(self) -> int: + return sum(len(dl.dataset) for dl in self.dataloaders) + + def __iter__(self): + iterators = [iter(dl) for dl in self.dataloaders] + while True: + choice = np.random.choice(len(self.dataloaders), p=self.ratios) + yield next(iterators[choice]) + + +def get_camera_dataloader( + conf: dict[str, DataConfig], + ratios: list[float] | None = None, + batch_size: int = 1, + num_workers: int = 8, + seed: int | None = None, + # wrap_dataset parameters for Action transform pipeline + resolution: str | None = None, + tokenizer_config: dict | None = None, + cfg_dropout_rate: float = 0.0, + max_action_dim: int = 32, + collate_fn: Callable = custom_collate_fn, +): + # random seed to make sure each training has different data order + if seed is None: + seed = random.randint(0, 1000000) + + # num_workers == 0 will not call worker_init_fn, so we need to seed the main process directly + if num_workers == 0: + try: + rank = dist.get_rank() + except Exception: + rank = 0 + rank_seed = seed + rank * 100000 + np.random.seed(rank_seed) + torch.manual_seed(rank_seed) + random.seed(rank_seed) + + # filter out None configs (e.g. when overriding) + conf = {k: v for k, v in conf.items() if v is not None} + + if len(conf) > 1: + dataloaders = [] + dataset_lengths = [] + + for i, dataset_name in enumerate(conf.keys()): + # dataset wrapped with Action transform pipeline + dataset = CameraDataset(conf=conf[dataset_name]) + dataset_lengths.append(len(dataset)) + wrapped = wrap_dataset( + dataset, + resolution=resolution, + tokenizer_config=tokenizer_config, + cfg_dropout_rate=cfg_dropout_rate, + max_action_dim=max_action_dim, + ) + + # loader + loader = DataLoader( + dataset=wrapped, + batch_size=batch_size, + num_workers=num_workers, + pin_memory=True, + worker_init_fn=partial(worker_init_fn, seed=seed), + collate_fn=collate_fn, + ) + dataloaders.append(loader) + + # calculate ratios if not provided + if ratios is None: + total_len = sum(dataset_lengths) + ratios = [l / total_len for l in dataset_lengths] + + log.info(f"MixedDataset sizes: {dataset_lengths}, ratios: {ratios}") + + result = MixedDataLoader(dataloaders, ratios) + else: + # dataset wrapped with Action transform pipeline + dataset = CameraDataset(conf=list(conf.values())[0]) + wrapped = wrap_dataset( + dataset, + resolution=resolution, + tokenizer_config=tokenizer_config, + cfg_dropout_rate=cfg_dropout_rate, + max_action_dim=max_action_dim, + ) + + # loader + result = DataLoader( + dataset=wrapped, + batch_size=batch_size, + num_workers=num_workers, + pin_memory=True, + worker_init_fn=partial(worker_init_fn, seed=seed), + collate_fn=collate_fn, + ) + + log.info(f"Dataset size: {len(dataset)}") + + return result + + +# PYTHONPATH=. python cosmos_framework/data/vfm/action/camera_dataset.py +if __name__ == "__main__": + import torchvision.io as io + + dataset = CameraDataset(conf=DataConfig(dataset_names="MultiCamVideo", validate=False, resolution="480")) + dataset_iter = iter(dataset) + + for i in range(3): + print(f"==================== Sample {i} ====================") + _t0 = time.time() + data = next(dataset_iter) + _t1 = time.time() + print(f"{'Loading time':<25}: {_t1 - _t0:.2f}s") + + print(f"==================== Sample {i} ====================") + print(f"{'video shape':<25}: {data['video'].shape}") # [C,T,H,W] + print(f"{'action shape':<25}: {data['action'].shape}") # [T,max_action_dim] + print(f"{'conditioning_fps':<25}: {data['conditioning_fps'].item()}") + print(f"{'mode':<25}: {data['mode']}") + print(f"{'domain_id':<25}: {data['domain_id'].item()}") + print(f"{'caption':<25}: {data['ai_caption']}...") + + # save video to local for debugging + video = data["video"] + video = video.permute(1, 0, 2, 3) # [T,C,H,W] + video_path = f"temp/camera_sample_{i}.mp4" + io.write_video( + video_path, video.permute(0, 2, 3, 1).numpy(), fps=data["conditioning_fps"].item() + ) # expects (T, H, W, C) + print(f"Saved video to {video_path}") diff --git a/cosmos_framework/data/vfm/action/camera_dataset_sharded.py b/cosmos_framework/data/vfm/action/camera_dataset_sharded.py new file mode 100644 index 0000000..82e8fb9 --- /dev/null +++ b/cosmos_framework/data/vfm/action/camera_dataset_sharded.py @@ -0,0 +1,653 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Sharded Camera Dataset for Action training. + +Example data structure: + +wdinfo: +s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/ + tartanair/v3/resolution_480/aspect_ratio_5_4/duration_150/wdinfo_02132026.json + +tarfiles: +s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset/ + tartanair/v3/resolution_480/aspect_ratio_5_4/duration_150/ + video/ + part_000000.tar + part_000001.tar + ... + camera/ + part_000000.tar + part_000001.tar + ... + metas/ + part_000000.tar + part_000001.tar + ... +""" + +import io +import json +import os +import random +import re +import tarfile +from typing import Iterator, Literal + +import numpy as np +import torch +from torch.utils.data import IterableDataset +from torchcodec.decoders import VideoDecoder + +from imaginaire.modules.camera import Camera +from cosmos_framework.utils import log +from cosmos_framework.utils.easy_io import easy_io +from cosmos_framework.data.vfm.action.domain_utils import get_domain_id +from cosmos_framework.data.vfm.action.pose_utils import ( + RotationConvention, + pose_abs_to_rel, +) + +# all ready-to-use wdinfos +CAMERA_WDINFOS = { + # ----- 5s datasets ----- + "tartanair_256": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/tartanair/v1/resolution_256/aspect_ratio_5_4/duration_150/wdinfo_03022026.json", + "tartanair_480": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/tartanair/v1/resolution_544/aspect_ratio_23_17/duration_150/wdinfo_03022026.json", + "endeavor_forever_256": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/endeavor_forever/v1/resolution_192/aspect_ratio_5_3/duration_150/wdinfo_03032026.json", + "endeavor_forever_480": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/endeavor_forever/v1/resolution_480/aspect_ratio_26_15/duration_150/wdinfo_03032026.json", + "synhuman_20251218_256": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/synhuman_20251218/v1/resolution_192/aspect_ratio_5_3/duration_150/wdinfo_03032026.json", + "synhuman_20251218_480": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/synhuman_20251218/v1/resolution_480/aspect_ratio_26_15/duration_150/wdinfo_03032026.json", + "pretrained_clips_260131_10k_256": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/pretrained_clips_260131_10k/v3/resolution_192/aspect_ratio_5_3/duration_150/wdinfo_02172026.json", + "pretrained_clips_260131_10k_480": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/pretrained_clips_260131_10k/v3/resolution_480/aspect_ratio_26_15/duration_150/wdinfo_02172026.json", + "synhuman_20260223_480": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/synhuman_20260223/v1/resolution_480/aspect_ratio_26_15/duration_150/wdinfo_03032026.json", + # ----- 10s datasets ----- + # pretrain video 100k (mixed resolution, 193k samples) + "pretrained_clips_260307_100k": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/pretrained_clips_260307_100k/v1p1/", + # pretrain video 100k filtered (mixed resolution, 72k samples) + "pretrained_clips_260307_100k_filtered": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/pretrained_clips_260307_100k_filtered/v1p1/", + "pretrained_clips_260325_500k_01_filtered": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/pretrained_clips_260325_500k_01_filtered/v1p1/", + "pretrained_clips_260325_500k_02_filtered": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/pretrained_clips_260325_500k_02_filtered/v1p1/", + "pretrained_clips_260313_10s_100k_filtered": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/pretrained_clips_260313_10s_100k_filtered/v1p1/", + # ----- 60s datasets ----- + # 60s duration + "endeavor_forever_480_60s": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/endeavor_forever/v1p2/resolution_480/aspect_ratio_26_15/duration_1800/wdinfo_03032026.json", # only 1798 frames actually! + "drivesim_480_60s": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/drivesim/v1p2/resolution_480/aspect_ratio_26_15/duration_1800/wdinfo_03042026.json", # 1749 frames + # pretrain video 100k 59s filtered (mixed resolution, 16k samples) + "pretrained_clips_260313_59s_100k_filtered": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/pretrained_clips_260313_59s_100k_filtered/v1p2/", + "pretrained_clips_260313_59s_100k_filtered_480": "s3://nv-00-10206-robot/cosmos3_action_data/camera_webdataset_wdinfo/pretrained_clips_260313_59s_100k_filtered/v1p2/resolution_480/aspect_ratio_16_9/frames_1700_1800/wdinfo_03242026.json", +} + + +class CameraDatasetSharded(IterableDataset): + def __init__( + self, + wdinfo_paths: list[str] | None = None, + bucket: str = "nv-00-10206-robot", + credential_path: str = "credentials/gcp_training.secret", + mode: str = "forward_dynamics", + embodiment_type: str = "camera_pose", + split: str = "train", + seed: int = 0, + shuffle: bool = True, + rotation_format: RotationConvention = "rot6d", + pose_convention: Literal["backward_anchored", "backward_framewise"] = ("backward_framewise"), + fix_caption: bool = True, # fix caption by default + fix_caption_text: str = "The camera moves in a scene.", + translation_scale: float = 1.0, + rotation_scale: float = 1.0, + discard_varying_intrinsics: bool = False, + # wdinfo filtering + wdinfo_resolution: Literal["all", "gt480", "gt720"] = "all", + max_frames: int = -1, # only truncate if exceed max_frames + num_frames: int = -1, # always truncate to num_frames + # When True, use a separate domain ID for inverse dynamics / policy modes + # so that DomainAwareLinear learns different projections for anchored (conditioning) + # vs framewise (generation) action representations. + mode_aware_domain: bool = False, + inv_embodiment_type: str = "camera_pose_inv", + max_action_translation_norm: float | None = None, + # When True, decode the full video in one pass (used by KV-cache segment loaders). + whole_video: bool = False, + # Benchmark datasets: list of S3 dataset names for loading individual files + # instead of tarfiles. Activated automatically when split is val-like. + benchmark_datasets: list[str] | None = ("videos_camera_benchmark",), + # Caption model key used in the benchmark caption JSON files. + benchmark_caption_model: str = "Qwen3-VL-30B-A3B-Instruct", + ): + super().__init__() + + self.wdinfo_paths = wdinfo_paths or [] + self.bucket = bucket + self.credential_path = credential_path + self.mode = mode + self.split = split.lower().strip() + self.seed = seed + self.shuffle = shuffle + self.rotation_format: RotationConvention = rotation_format + self.pose_convention: Literal["absolute", "backward_anchored", "backward_framewise"] = pose_convention + + # hard-coded caption keys and weights for now + caption_keys = { + "qwen2p5_7b_caption": 0.7, + "qwen2p5_7b_caption_short": 0.1, + "qwen2p5_7b_caption_medium": 0.2, + } + self.caption_keys = list(caption_keys.keys()) + self.caption_weights = list(caption_keys.values()) + + self.fix_caption = fix_caption + self.fix_caption_text = fix_caption_text + self.translation_scale = translation_scale + self.rotation_scale = rotation_scale + self.discard_varying_intrinsics = discard_varying_intrinsics + self.max_action_translation_norm = max_action_translation_norm + self.wdinfo_resolution = wdinfo_resolution + self.max_frames = max_frames + self.num_frames = num_frames + self.whole_video = whole_video + # Get domain ID for this embodiment + self.domain_id = get_domain_id(embodiment_type) + # When mode_aware_domain is True, inverse_dynamics/policy modes use a separate domain ID + self.mode_aware_domain = mode_aware_domain + self.domain_id_inv = get_domain_id(inv_embodiment_type) if mode_aware_domain else self.domain_id + self.benchmark_caption_model = benchmark_caption_model + + # Validate mode + valid_modes = ["joint", "forward_dynamics", "inverse_dynamics", "policy", "image2video"] + if mode not in valid_modes: + raise ValueError(f"mode must be one of {valid_modes}, got {mode}") + + # Validate split + if self.split not in {"train", "val", "valid", "validation", "full"}: + raise ValueError(f"Unsupported {split=}. Use train/val/full.") + + # Configure S3 backend using easy_io + self._setup_s3_backend() + + # Validation always uses benchmark datasets + self._benchmark_uids: list[tuple[str, str]] = [] + if self.split in {"val", "valid", "validation"}: + if not benchmark_datasets: + raise ValueError("benchmark_datasets is required for validation splits.") + self._load_benchmark_uids(benchmark_datasets) + + # Load tar files from wdinfo.json (training / full splits only) + self._tar_files: list[str] = [] + self._key_count: list[int] = [] + self._total_key_count: int = 0 + + if self._benchmark_uids: + log.info( + f"Initialized CameraDatasetSharded (benchmark): " + f"benchmark_datasets={benchmark_datasets}, " + f"mode={mode}, split={self.split}, " + f"num_samples={len(self._benchmark_uids)}" + ) + else: + if not self.wdinfo_paths: + raise ValueError("Must provide wdinfo_paths for training.") + + # Resolve prefix paths into individual wdinfo JSON paths + self.wdinfo_paths = self._resolve_wdinfo_paths(self.wdinfo_paths) + + # Filter wdinfo paths by resolution + self.wdinfo_paths = self._filter_wdinfo_by_resolution(self.wdinfo_paths) + + self._load_wdinfo() + + log.info( + f"Initialized CameraDatasetSharded: wdinfo_paths={self.wdinfo_paths}, " + f"mode={mode}, split={self.split}, " + f"num_tar_files={len(self._tar_files)}, " + f"total_samples={self._total_key_count}" + ) + + def _setup_s3_backend(self) -> None: + """Configure the easy_io S3 backend. Called in __init__ and __iter__ for worker processes.""" + easy_io.set_s3_backend( + backend_args={ + "backend": "s3", + "path_mapping": None, + "s3_credential_path": self.credential_path, + } + ) + + def _load_benchmark_uids(self, benchmark_datasets: list[str]) -> None: + """Load UIDs from benchmark datasets stored as individual S3 files. + + Each benchmark dataset lives at ``s3:///cosmos3_action_data//v3/`` + with a ``meta.json`` listing scene UIDs and per-UID ``videos/``, ``cameras/``, + ``captions/`` directories. + """ + for dataset_name in benchmark_datasets: + meta_path = f"s3://{self.bucket}/cosmos3_action_data/{dataset_name}/v3/meta.json" + try: + uids = json.loads(easy_io.get(meta_path))["scenes"] + except Exception as e: + raise RuntimeError(f"Failed to load benchmark meta from {meta_path}: {e}") from e + for uid in uids: + self._benchmark_uids.append((dataset_name, uid)) + log.info(f"Loaded {len(uids)} benchmark UIDs from {dataset_name}") + + def _resolve_wdinfo_paths(self, wdinfo_paths: list[str]) -> list[str]: + """Resolve wdinfo paths: direct .json paths are kept as-is, + prefix/directory paths are walked recursively to find all .json files.""" + resolved: list[str] = [] + for path in wdinfo_paths: + if path.endswith(".json"): + resolved.append(path) + else: + log.info(f"Walking S3 prefix to discover wdinfo JSONs: {path}") + relative_json_files = list( + easy_io.list_dir_or_file(path, list_dir=False, list_file=True, suffix=".json", recursive=True) + ) + if not relative_json_files: + raise RuntimeError(f"No .json wdinfo files found under prefix: {path}") + prefix = path.rstrip("/") + "/" + for rel_path in sorted(relative_json_files): + resolved.append(prefix + rel_path) + log.info(f"Discovered {len(relative_json_files)} wdinfo files under {path}") + return resolved + + def _filter_wdinfo_by_resolution(self, wdinfo_paths: list[str]) -> list[str]: + """Filter resolved wdinfo paths by minimum resolution. + + Resolution is extracted from the path pattern ``resolution_``. + For ``wdinfo_resolution="gtXXX"``, only paths with resolution >= XXX are kept. + """ + if self.wdinfo_resolution == "all": + return wdinfo_paths + + threshold_match = re.match(r"gt(\d+)", self.wdinfo_resolution) + if not threshold_match: + raise ValueError( + f"Invalid wdinfo_resolution format: {self.wdinfo_resolution!r}. " + "Expected 'all' or 'gtNNN' (e.g. 'gt480')." + ) + min_resolution = int(threshold_match.group(1)) + + filtered: list[str] = [] + for path in wdinfo_paths: + res_match = re.search(r"resolution_(\d+)", path) + if res_match: + resolution = int(res_match.group(1)) + if resolution >= min_resolution: + filtered.append(path) + else: + log.warning(f"No resolution found in wdinfo path, including by default: {path}") + filtered.append(path) + + log.info(f"Filtered wdinfo by resolution >= {min_resolution}: {len(filtered)}/{len(wdinfo_paths)} paths kept") + if not filtered: + raise RuntimeError(f"All wdinfo paths were filtered out by wdinfo_resolution={self.wdinfo_resolution!r}") + return filtered + + def _load_wdinfo(self) -> None: + self._tar_files = [] + self._key_count = [] + self._total_key_count = 0 + + for wdinfo_path in self.wdinfo_paths: + # log.info(f"Loading wdinfo from: {wdinfo_path}") + + try: + wdinfo_bytes = easy_io.get(wdinfo_path) + wdinfo = json.loads(wdinfo_bytes) + except Exception as e: + raise RuntimeError(f"Failed to load wdinfo from {wdinfo_path}: {e}") from e + + # Extract metadata + data_root = wdinfo["root"].rstrip( + "/" + ) # e.g. cosmos3_action_data/camera_webdataset/tartanair/v3/resolution_256/aspect_ratio_5_4/duration_150/ + data_list = wdinfo["data_list"] # ["part_000000.tar", "part_000001.tar", ...] + key_count = wdinfo["data_list_key_count"] # [20, 20, ...] + + # only for video tar files + tar_paths = [f"s3://{self.bucket}/{data_root}/video/{tar_path}" for tar_path in data_list] + + # Reconstruct full S3 paths for tar files + self._tar_files.extend(tar_paths) + self._key_count.extend(key_count) + + # Accumulate total sample count + self._total_key_count += sum(key_count) + + # log.info(f"Loaded {len(data_list)} tar files from wdinfo with {sum(key_count)} samples") + + if not self._tar_files: + raise RuntimeError(f"No tar files found in wdinfo at {self.wdinfo_paths}") + + def __len__(self) -> int: + if self._benchmark_uids: + return len(self._benchmark_uids) + return self._total_key_count + + def __iter__(self) -> Iterator[dict]: + """Iterate over the dataset.""" + # Re-configure S3 backend in case this is running in a worker process after unpickling + self._setup_s3_backend() + + if self._benchmark_uids: + yield from self._iter_benchmark() + else: + yield from self._iter_tar_files() + + def _iter_benchmark(self) -> Iterator[dict]: + """Iterate over benchmark datasets (individual S3 files).""" + uids = list(self._benchmark_uids) + if self.shuffle: + random.shuffle(uids) + + skipped: list[dict[str, str]] = [] + yielded = 0 + + for dataset_name, uid in uids: + try: + base = f"s3://{self.bucket}/cosmos3_action_data/{dataset_name}/v3" + video_bytes = easy_io.get(f"{base}/videos/{uid}.mp4") + camera_bytes = easy_io.get(f"{base}/cameras/{uid}.json") + + # Load caption from per-sample JSON (different format from tarfile metas) + caption = None + if not self.fix_caption: + caption_data = json.loads(easy_io.get(f"{base}/captions/{uid}.json")) + captions = caption_data[self.benchmark_caption_model] + caption = random.choice([captions["long"], captions["short"], captions["medium"]]) + + sample = self._process_sample(video_bytes, camera_bytes, uid=uid, caption_override=caption) + if sample: + yielded += 1 + yield sample + else: + log.warning(f"SKIPPED benchmark sample {dataset_name}/{uid}: _process_sample returned None") + skipped.append({"dataset": dataset_name, "uid": uid, "reason": "process_sample_returned_none"}) + + except Exception as e: + log.warning(f"SKIPPED benchmark sample {dataset_name}/{uid}: S3 load failed: {e}") + skipped.append({"dataset": dataset_name, "uid": uid, "reason": f"s3_load_error: {e}"}) + continue + + log.info(f"Benchmark iteration complete: {yielded}/{len(uids)} samples yielded, {len(skipped)} skipped") + if skipped: + log.warning(f"Skipped benchmark samples: {skipped}") + self._last_skipped_benchmark_samples = skipped + + def _iter_tar_files(self) -> Iterator[dict]: + """Iterate over sharded tar files from S3.""" + tar_files = list(self._tar_files) + if self.shuffle: + random.shuffle(tar_files) + + for tar_path in tar_files: + try: + # Read tar file bytes using easy_io + metas_path = tar_path.replace("/video/", "/metas/") + camera_path = tar_path.replace("/video/", "/camera/") + + video_bytes = easy_io.get(tar_path) + metas_bytes = easy_io.get(metas_path) + camera_bytes = easy_io.get(camera_path) + + with ( + tarfile.open(fileobj=io.BytesIO(video_bytes), mode="r") as video_tar, + tarfile.open(fileobj=io.BytesIO(metas_bytes), mode="r") as metas_tar, + tarfile.open(fileobj=io.BytesIO(camera_bytes), mode="r") as camera_tar, + ): + # Map filenames to members for fast lookup + camera_members = {m.name: m for m in camera_tar.getmembers() if m.isfile()} + metas_members = {m.name: m for m in metas_tar.getmembers() if m.isfile()} + + # Iterate over video files + for video_member in video_tar.getmembers(): + uuid = os.path.splitext(os.path.basename(video_member.name))[0] + json_name = f"{uuid}.json" + + if json_name not in camera_members or json_name not in metas_members: + raise ValueError(f"Missing metadata or camera for {uuid} in {tar_path}") + + # Extract data + video_file_bytes = video_tar.extractfile(video_member).read() + camera_file_bytes = camera_tar.extractfile(camera_members[json_name]).read() + metas_file_bytes = metas_tar.extractfile(metas_members[json_name]).read() + + sample = self._process_sample( + video_file_bytes, + camera_file_bytes, + uid=uuid, + metas_bytes=metas_file_bytes, + ) + if sample: + yield sample + # DEBUG: only use 1 video per tar + # break + + except Exception as e: + log.warning(f"Failed to read tar file {tar_path}: {e}") + continue + + def _process_sample( + self, + video_bytes: bytes, + camera_bytes: bytes, + uid: str = "", + *, + metas_bytes: bytes | None = None, + caption_override: str | None = None, + ) -> dict | None: + """Decode video frames, compute relative actions, and return a sample dict. + + When ``whole_video`` is True or ``num_frames == -1``, loads the full video from frame 0. + Otherwise applies ``max_frames`` / ``num_frames`` clipping and sampling as in the base + sharded loader. For the validation split, random clips use start index 0. + + Args: + caption_override: When provided, use this caption directly instead of + extracting from *metas_bytes*. Used by the benchmark data path where + captions live in a different JSON format. + """ + try: + decoder = VideoDecoder(video_bytes, num_ffmpeg_threads=4) + total_frames = decoder.metadata.num_frames or 0 + video_fps = decoder.metadata.average_fps or 0.0 + whole_video = self.whole_video + + if whole_video: + if total_frames == 0: + del decoder + log.warning(f"SKIPPED {uid}: whole_video=True but total_frames=0") + return None + frame_indices = list(range(total_frames)) + elif self.max_frames > 0: + # Use all frames at native fps (stride=1), capped by max_frames. + # VAE compresses temporal dim by 4x with 1 condition frame, + # so total video frames must be 1 + 4*N. + num_video_frames = min(total_frames, self.max_frames) + N = (num_video_frames - 1) // 4 + num_video_frames = 1 + 4 * N + if num_video_frames < 2: + del decoder + log.warning( + f"SKIPPED {uid}: too few frames after VAE alignment " + f"(total_frames={total_frames}, max_frames={self.max_frames}, aligned={num_video_frames})" + ) + return None + frame_indices = list(range(num_video_frames)) + else: + # No max_frames cap — use all frames, but still VAE-align to 1 + 4*N. + N = (total_frames - 1) // 4 + num_video_frames = 1 + 4 * N + if num_video_frames < 2: + del decoder + log.warning( + f"SKIPPED {uid}: too few frames after VAE alignment " + f"(total_frames={total_frames}, aligned={num_video_frames})" + ) + return None + frame_indices = list(range(num_video_frames)) + + # If num_frames is set, always sample exactly num_frames from the available frames. + if self.num_frames > 0 and not whole_video: + available = len(frame_indices) + N = (self.num_frames - 1) // 4 + target_frames = 1 + 4 * N + if target_frames < 2 or available < target_frames: + del decoder + log.warning( + f"SKIPPED {uid}: not enough frames for num_frames={self.num_frames} " + f"(available={available}, target={target_frames})" + ) + return None + max_start = available - target_frames + start = 0 if self.split != "train" else (random.randint(0, max_start) if max_start > 0 else 0) + frame_indices = frame_indices[start : start + target_frames] + + # torchcodec returns [T,C,H,W] tensor + frame_batch = decoder.get_frames_at(frame_indices) + video_frames = frame_batch.data # [T,C,H,W] uint8 + del decoder + + # Convert to [C,T,H,W] format expected by model + video = video_frames.permute(1, 0, 2, 3) # [T,C,H,W] -> [C,T,H,W] + + # Load camera data + camera_data = json.loads(camera_bytes) + + # Special check for varying intrinsics (e.g., in synhuman_20260223 dataset) + # If the intrinsic (fx, fy, cx, cy) changes during different frames, discard this sample. + fl = np.array(camera_data["camera"]["focal_length"]) + pp = np.array(camera_data["camera"]["principal_point"]) + if self.discard_varying_intrinsics and ( + not np.allclose(fl, fl[0], atol=1e-5) or not np.allclose(pp, pp[0], atol=1e-5) + ): + log.warning(f"SKIPPED {uid}: varying intrinsics detected (discard_varying_intrinsics=True)") + return None + + w2c = np.array(camera_data["camera"]["pose_world2cam"]).reshape(-1, 7) # [N,7] + if w2c.shape[0] <= frame_indices[-1]: + log.warning( + f"SKIPPED {uid}: not enough camera poses " + f"(num_poses={w2c.shape[0]}, last_frame_idx={frame_indices[-1]})" + ) + return None + + # Get w2c for the sampled frames + w2c = w2c[frame_indices] # [T,7] + + # Convert (qx,qy,qz,qw,tx,ty,tz) to [R|t] matrices + w2c = Camera.extrinsic_params_to_matrices(w2c) # [T,3,4] + w2c_homo = np.eye(4, dtype=np.float32)[None, :, :].repeat(w2c.shape[0], axis=0) # [T,4,4] + w2c_homo[:, :3, :] = w2c + c2w_homo = np.linalg.inv(w2c_homo) + + # Determine mode + if self.mode == "joint": + mode = random.choices( + ["forward_dynamics", "inverse_dynamics", "policy"], + weights=[0.8, 0.1, 0.1], + k=1, + )[0] + else: + mode = self.mode + + action = pose_abs_to_rel( + c2w_homo, + rotation_format=self.rotation_format, + pose_convention=self.pose_convention, + translation_scale=self.translation_scale, + rotation_scale=self.rotation_scale, + ) + + if self.max_action_translation_norm is not None and self.split == "train": + trans_norms = np.linalg.norm(action[:, :3], axis=1) + if trans_norms.max() > self.max_action_translation_norm: + log.warning( + f"SKIPPED {uid}: action translation norm too large " + f"(max={trans_norms.max():.4f}, threshold={self.max_action_translation_norm})" + ) + return None + + action = torch.from_numpy(action) # [num_frames,action_dim] + + # Load caption data + if self.fix_caption: + caption = self.fix_caption_text + elif caption_override is not None: + caption = caption_override + else: + metas_data = json.loads(metas_bytes) + + # Example: "t2w_windows": [{"end_frame": 150, "qwen2p5_7b_caption": "...", ...}] + t2w_windows = metas_data["t2w_windows"] + window = t2w_windows[0] + caption_key = random.choices(self.caption_keys, weights=self.caption_weights, k=1)[0] + caption = window[caption_key] + + # FPS + fps = torch.tensor(video_fps, dtype=torch.float32) # scalar + + # Select domain ID: use inverse domain for generation modes when mode_aware_domain is on + if self.mode_aware_domain and mode in ["inverse_dynamics", "policy"]: + domain_id = self.domain_id_inv + else: + domain_id = self.domain_id + + # Build sample dict + sample = { + "video": video, # [C,T,H,W] uint8 + "action": action, # [T-1,action_dim] float32 + "conditioning_fps": fps, # scalar float32 + "ai_caption": caption, + "mode": mode, + "__key__": torch.tensor([hash(uid) % (2**31)], dtype=torch.long), + "domain_id": torch.tensor(domain_id, dtype=torch.long), + "viewpoint": "ego_view", + } + return sample + + except Exception as e: + log.warning(f"Error processing sample {uid}: {e}") + return None + + +# PYTHONPATH=. python cosmos_framework/data/vfm/action/camera_dataset_sharded.py +if __name__ == "__main__": + import time + + import torchvision + + dataset = CameraDatasetSharded( + wdinfo_paths=[ + CAMERA_WDINFOS["pretrained_clips_260313_10s_100k_filtered"], + ], + split="train", + shuffle=True, + fix_caption=False, + # wdinfo_resolution="gt720", + # max_frames=200, + ) + dataset_iter = iter(dataset) + + for i in range(10): + print(f"==================== Sample {i} ====================") + _t0 = time.time() + data = next(dataset_iter) + _t1 = time.time() + print(f"{'Loading time':<25}: {_t1 - _t0:.2f}s") + + print(f"==================== Sample {i} ====================") + print(f"{'video shape':<25}: {data['video'].shape}") # [C,T,H,W] + print(f"{'action shape':<25}: {data['action'].shape}") # [T,max_action_dim] + print(f"{'conditioning_fps':<25}: {data['conditioning_fps'].item()}") + print(f"{'mode':<25}: {data['mode']}") + print(f"{'domain_id':<25}: {data['domain_id'].item()}") + print(f"{'caption':<25}: {data['ai_caption']}") + + # save video to local for debugging + video = data["video"] + video = video.permute(1, 0, 2, 3) # [T,C,H,W] + video_path = f"temp/camera_sample_{i}.mp4" + torchvision.io.write_video( + video_path, video.permute(0, 2, 3, 1).numpy(), fps=data["conditioning_fps"].item() + ) # expects (T, H, W, C) + print(f"Saved video to {video_path}") diff --git a/cosmos_framework/data/vfm/action/compute_action_stats.py b/cosmos_framework/data/vfm/action/compute_action_stats.py new file mode 100644 index 0000000..3b88dda --- /dev/null +++ b/cosmos_framework/data/vfm/action/compute_action_stats.py @@ -0,0 +1,1416 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Script to compute mean and std for action datasets. + +This script iterates through a dataset and computes action statistics +using Welford's online algorithm for numerical stability. + +Usage: + # Compute stats for LIBERO training dataset + PYTHONPATH=. python cosmos_framework/data/vfm/action/compute_action_stats.py \ + --config cosmos_framework/configs/base/config.py \ + --split train \ + --max-samples 10000 \ + --output cosmos_framework/data/vfm/action/libero_action_stats_10k.json \ + -- experiment=libero_exp + + # Compute stats for PushT + PYTHONPATH=. python cosmos_framework/data/vfm/action/compute_action_stats.py \ + --config cosmos_framework/configs/base/config.py \ + --split train \ + --output cosmos_framework/data/vfm/action/pusht_action_stats.json \ + -- experiment=pusht_exp + + # Quick test with limited samples + PYTHONPATH=. python cosmos_framework/data/vfm/action/compute_action_stats.py \ + --config cosmos_framework/configs/base/config.py \ + --split train \ + --max-samples 1000 \ + --output test_stats.json \ + -- experiment=libero_exp + + # Override dataset parameters + PYTHONPATH=. python cosmos_framework/data/vfm/action/compute_action_stats.py \ + --config cosmos_framework/configs/base/config.py \ + --split train \ + --output stats.json \ + -- experiment=libero_exp \ + dataloader_train.dataloaders.libero_data.dataloader.dataset.use_rotation_9d=False +""" + +import argparse +import copy +import importlib +import inspect +import json +import os as _os +import time +from pathlib import Path +from typing import Any, Callable + +import numpy as np +import torch +import torch.distributed as dist +from omegaconf import open_dict +from torch.utils.data import ConcatDataset, DataLoader, IterableDataset, Sampler +from tqdm import tqdm + +from cosmos_framework.utils.config import load_config +from cosmos_framework.utils.lazy_config import instantiate +from cosmos_framework.utils.context_managers import data_loader_init + +# cosmos_framework/data/vfm/action/ => 5 levels up to repo root +IMAGINAIRE4_ROOT = Path(__file__).resolve().parents[5] +# Bundled normalizers directory. When ``--output`` is not given, stats are +# written to ``/__.json`` so that +# loaders (e.g. ``BaseActionLeRobotDataset._load_norm_stats``) can resolve +# them from repo paths without hitting lustre. +DEFAULT_OUTPUT_DIR = IMAGINAIRE4_ROOT / "projects" / "cosmos3" / "vfm" / "datasets" / "action" / "normalizers" + + +# --------------------------------------------------------------------------- +# JSON pretty-printer +# --------------------------------------------------------------------------- +# ``json.dump(indent=2)`` puts every array element on its own line, which for +# stats payloads (action_dim vectors, 50-element quantile arrays) explodes +# what should be a ~17-line file into hundreds of lines. Keep dicts multi-line +# (2-space indent) but collapse numeric/scalar arrays to one line, and +# format floats at 6-decimal fixed precision (no scientific notation). +_FLOAT_DECIMALS = 6 + + +def _fmt_primitive(value: Any) -> str: + if isinstance(value, bool): + return "true" if value else "false" + if isinstance(value, float): + return f"{value:.{_FLOAT_DECIMALS}f}" + return json.dumps(value, ensure_ascii=False) + + +def _fmt_aligned(value: Any) -> str: + """Format a primitive with leading-space padding for non-negative numerics. + + Used inside numeric arrays so that negative (``-0.000094``) and non-negative + (`` 0.001623``) values line up column-wise. Still emits valid JSON — the + extra leading whitespace is insignificant per RFC 8259. + """ + if isinstance(value, bool): + return "true" if value else "false" + if isinstance(value, float): + return f"{value: .{_FLOAT_DECIMALS}f}" + if isinstance(value, int): + return f" {value}" if value >= 0 else f"{value}" + return json.dumps(value, ensure_ascii=False) + + +def _is_numeric(x: Any) -> bool: + """Real numeric (excludes bool, which is a subclass of int in Python).""" + return isinstance(x, (int, float)) and not isinstance(x, bool) + + +def _serialize(value: Any, indent: int = 0) -> str: + pad = " " * indent + inner = " " * (indent + 1) + if isinstance(value, dict): + if not value: + return "{}" + items = list(value.items()) + # Align value columns when any sibling value is a flat list (e.g., the + # ``global`` / ``global_raw`` blocks have mean/std/min/max... lists of + # different-length keys). Pad each key's trailing whitespace so all + # opening brackets land in the same column. + key_strs = [json.dumps(k, ensure_ascii=False) for k in value] + align = any(isinstance(v, list) and all(not isinstance(x, (dict, list)) for x in v) for v in value.values()) + max_key_len = max(len(s) for s in key_strs) if align else 0 + body = [] + for i, ((k, v), ks) in enumerate(zip(items, key_strs)): + gap = " " * (max_key_len - len(ks) + 1) if align else " " + sep = "," if i < len(items) - 1 else "" + body.append(f"{inner}{ks}:{gap}{_serialize(v, indent + 1)}{sep}") + return "{\n" + "\n".join(body) + f"\n{pad}}}" + if isinstance(value, list): + if not value: + return "[]" + if all(not isinstance(x, (dict, list)) for x in value): + # Numeric-array path: pad non-negatives with a leading space so + # ``-X.YYY`` and `` X.YYY`` columns line up vertically. + if any(isinstance(x, float) for x in value) and all(_is_numeric(x) for x in value): + return "[" + ", ".join(_fmt_aligned(x) for x in value) + "]" + return "[" + ", ".join(_fmt_primitive(x) for x in value) + "]" + body = [f"{inner}{_serialize(v, indent + 1)}{',' if i < len(value) - 1 else ''}" for i, v in enumerate(value)] + return "[\n" + "\n".join(body) + f"\n{pad}]" + return _fmt_primitive(value) + + +def _pretty_dump(obj: Any) -> str: + return _serialize(obj) + "\n" + + +class WelfordAccumulator: + """Welford's online algorithm for computing mean and variance. + + This is numerically stable for large datasets and works with streaming data. + + Reference: https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Welford's_online_algorithm + """ + + def __init__(self, dim: int): + self.dim = dim + self.count = 0 + self.mean = np.zeros(dim, dtype=np.float64) + self.m2 = np.zeros(dim, dtype=np.float64) # Sum of squared differences + self.min_val = np.full(dim, np.inf, dtype=np.float64) + self.max_val = np.full(dim, -np.inf, dtype=np.float64) + + def update(self, x: np.ndarray) -> None: + """Update statistics with a new sample or batch of samples. + + Uses Chan et al.'s parallel algorithm to merge a new batch in O(D) + numpy ops (no Python per-sample loop). + + Args: + x: Array of shape (D,) for single sample or (N, D) for batch. + """ + if x.ndim == 1: + x = x.reshape(1, -1) + x = x.astype(np.float64, copy=False) + + n_b = x.shape[0] + if n_b == 0: + return + + mean_b = x.mean(axis=0) + # sum of squared diffs within batch + m2_b = ((x - mean_b) ** 2).sum(axis=0) + + n_a = self.count + n_new = n_a + n_b + delta = mean_b - self.mean + + # Merge + self.mean = self.mean + delta * (n_b / n_new) + self.m2 = self.m2 + m2_b + (delta * delta) * (n_a * n_b / n_new) + self.count = n_new + + # Track per-dim min/max + np.minimum(self.min_val, x.min(axis=0), out=self.min_val) + np.maximum(self.max_val, x.max(axis=0), out=self.max_val) + + def get_mean(self) -> np.ndarray: + """Return the current mean.""" + return self.mean.copy() + + def get_variance(self) -> np.ndarray: + """Return the current sample variance.""" + if self.count < 2: + return np.zeros(self.dim, dtype=np.float64) + return self.m2 / (self.count - 1) + + def get_std(self) -> np.ndarray: + """Return the current standard deviation.""" + return np.sqrt(self.get_variance()) + + def get_min(self) -> np.ndarray: + """Return the minimum values seen.""" + return self.min_val.copy() + + def get_max(self) -> np.ndarray: + """Return the maximum values seen.""" + return self.max_val.copy() + + +class ReservoirAccumulator: + """Reservoir sampling (Algorithm R) for streaming quantile estimation. + + Maintains a fixed-size random sample of all observations seen so far. + After the stream finishes, exact quantiles are computed on the reservoir. + """ + + def __init__(self, dim: int, max_size: int = 50_000): + self.dim = dim + self.max_size = max_size + self.buffer = np.empty((max_size, dim), dtype=np.float32) + self.count = 0 + self._rng = np.random.RandomState(42) + + def update(self, x: np.ndarray) -> None: + """Update reservoir with a new sample or batch of samples.""" + if x.ndim == 1: + x = x.reshape(1, -1) + n = x.shape[0] + + if self.count < self.max_size: + space = self.max_size - self.count + fill_n = min(n, space) + self.buffer[self.count : self.count + fill_n] = x[:fill_n] + self.count += fill_n + x = x[fill_n:] + n = x.shape[0] + if n == 0: + return + + stream_indices = np.arange(self.count, self.count + n, dtype=np.int64) + j = (self._rng.random(n) * (stream_indices + 1)).astype(np.int64) + accept = j < self.max_size + if accept.any(): + self.buffer[j[accept]] = x[accept] + self.count += n + + def quantile(self, q: float) -> np.ndarray: + """Return the q-th quantile from the reservoir.""" + n = min(self.count, self.max_size) + if n == 0: + return np.zeros(self.dim, dtype=np.float32) + return np.quantile(self.buffer[:n], q, axis=0).astype(np.float32) + + +class SortedSubsetSampler(Sampler[int]): + """Yield a fixed list of indices in ascending order (no shuffling). + + Used to combine *unbiased* random subset sampling with *sequential* access + order: we draw a random subset once (seeded) and then iterate it sorted, + so that datasets with internal LRU caches (e.g. ``BaseActionLeRobotDataset`` + with 100+ inner LeRobotDatasets) don't thrash their cache on every batch. + + Unlike :class:`torch.utils.data.SubsetRandomSampler`, indices are **not** + reshuffled per iteration. + """ + + def __init__(self, indices: list[int]) -> None: + self._indices = sorted(indices) + + def __iter__(self): + return iter(self._indices) + + def __len__(self) -> int: + return len(self._indices) + + +ROTATION_FORMAT_DIM: dict[str, int] = {"rot9d": 9, "rot6d": 6, "euler_xyz": 3} + + +def get_rotation_dims(action_dim: int, rotation_format: str = "rot6d") -> list[int]: + """Infer rotation dim indices from ``action_dim`` and ``rotation_format``. + + Assumes the canonical layouts used in ``projects/cosmos3/vfm/datasets/action/``: + + - 9D (camera/AV): ``[pos(3) + rot(R)]`` + - 10D (single arm): ``[pos(3) + rot(R) + grip(1)]`` + - 2*(3+R+1) (dual arm): ``[L_pos + L_rot + L_grip | R_pos + R_rot + R_grip]`` + - 3*(3+R)+2 (AgiBot): ``[head_pos + head_rot | R_pos + R_rot + R_grip | + L_pos + L_rot + L_grip]`` + - 57D (HandPose rot6d, wrist+fingertips): + ``[cam(9) | R_wrist(9) + R_fingers(15) | L_wrist(9) + L_fingers(15)]`` + + For other configurations (e.g. rot9d HandPose or all-finger variants), + specify ``--skip-rotation-dims`` manually. + """ + if rotation_format not in ROTATION_FORMAT_DIM: + raise ValueError(f"rotation_format must be one of {list(ROTATION_FORMAT_DIM)}, got {rotation_format!r}") + rot_dim = ROTATION_FORMAT_DIM[rotation_format] + pos_dim = 3 + slot = pos_dim + rot_dim # 9 for rot6d + + # 9D: camera / AV — [pos + rot] + if action_dim == slot: + return list(range(pos_dim, slot)) + + # 10D: single arm + gripper — [pos + rot + grip] + if action_dim == slot + 1: + return list(range(pos_dim, slot)) + + # 20D: dual arm — [L_pos + L_rot + L_grip | R_pos + R_rot + R_grip] + if action_dim == 2 * (slot + 1): + right_rot_start = (slot + 1) + pos_dim + return list(range(pos_dim, slot)) + list(range(right_rot_start, right_rot_start + rot_dim)) + + # 29D: Embodiment C/Beta gripper FK-pose layout: + # [head_pos + head_rot | right_pos + right_rot + right_grip | left_pos + left_rot + left_grip] + if action_dim == 3 * slot + 2: + head_rot = list(range(pos_dim, slot)) + right_start = slot + right_rot = list(range(right_start + pos_dim, right_start + slot)) + left_start = slot + slot + 1 + left_rot = list(range(left_start + pos_dim, left_start + slot)) + return head_rot + right_rot + left_rot + + # 57D (rot6d) HandPose wrist+fingertips: + # [cam(3+R) | R_wrist(3+R) | R_fingers(5*3) | L_wrist(3+R) | L_fingers(5*3)] + if action_dim == slot + 2 * (slot + 5 * 3): + cam_rot = list(range(pos_dim, slot)) + r_wrist_rot = list(range(slot + pos_dim, slot + slot)) + l_wrist_rot = list(range(slot + slot + 15 + pos_dim, slot + slot + 15 + slot)) + return cam_rot + r_wrist_rot + l_wrist_rot + + # Unknown layout — most likely joint-space action (e.g. Embodiment_b 30D = + # arm(14) + end(14) + effector(2), or robomind joint-space). No SE(3) + # rotation dims to skip. Warn and return empty list rather than raising, + # so the stats get saved and the user can inspect the json. + print( + f" [warning] Cannot auto-detect SE(3) rotation dims for " + f"action_dim={action_dim}, rotation_format={rotation_format!r}. " + f"Assuming joint-space or custom layout — no dims will be skipped." + ) + return [] + + +def parse_dim_spec(spec: str | None) -> list[int]: + """Parse a dim index spec like ``"3-8,12-17,36-41"`` into a sorted unique list. + + Supports comma-separated tokens; each token is either a single integer or a + closed range ``a-b`` (both endpoints included). + """ + if not spec: + return [] + result: set[int] = set() + for token in spec.split(","): + token = token.strip() + if not token: + continue + if "-" in token: + a, b = token.split("-", 1) + lo, hi = int(a), int(b) + if lo > hi: + lo, hi = hi, lo + result.update(range(lo, hi + 1)) + else: + result.add(int(token)) + return sorted(result) + + +def _apply_skip_rotation(stats: dict[str, Any], skip_dims: list[int], action_dim: int) -> None: + """In-place overwrite stats at ``skip_dims`` with identity values. + + After this, downstream ``normalize_action`` (both ``meanstd`` and ``quantile``) + is the identity for those dims. Useful for rot6d/rot9d rotation dims, which + must not be per-dim normalized. + """ + if not skip_dims: + return + valid = [d for d in skip_dims if 0 <= d < action_dim] + if not valid: + return + + for key, identity in ( + ("mean", 0.0), + ("std", 1.0), + ("min", -1.0), + ("max", 1.0), + ("q01", -1.0), + ("q99", 1.0), + ): + arr = stats["global"][key] + for d in valid: + arr[d] = identity + stats["global"][key] = arr + + +def compute_action_stats( + dataloader: Any, + action_key: str = "action", + max_samples: int | None = None, + action_dim: int | None = None, + reservoir_size: int = 50_000, + return_accumulators: bool = False, +) -> dict[str, Any]: + """Compute mean and std for actions in a dataloader. + + Args: + dataloader: DataLoader that yields batches with action tensors. + action_key: Key for action tensor in batch output. + max_samples: If set, limit the number of samples to process. + action_dim: If set, only compute stats for first action_dim dimensions. + If None, auto-detect from first sample. + reservoir_size: Number of action frames to keep for quantile estimation. + + Returns: + Dictionary containing mean, std, quantiles, and metadata. + """ + print(f"Computing action statistics...") + print(f" action_key: {action_key}") + print(f" max_samples: {max_samples}") + + # Initialize accumulators (will be created after seeing first sample) + global_accumulator: WelfordAccumulator | None = None + reservoir: ReservoirAccumulator | None = None + detected_action_dim: int | None = None + chunk_length: int | None = None + + sample_count = 0 + start_time = time.time() + + data_iter = iter(dataloader) + pbar = tqdm(desc="Computing action stats", unit="samples") + + while True: + try: + batch = next(data_iter) + except StopIteration: + print("End of dataset reached.") + break + + if isinstance(batch, torch.Tensor): + batch = batch.numpy() + + if batch.ndim == 2: + batch = batch[:, None, :] + + B, chunk_len_batch, dim = batch.shape + + # Initialize accumulators on first batch + if global_accumulator is None: + chunk_length = chunk_len_batch + detected_action_dim = action_dim if action_dim is not None else dim + print(f" Detected action shape: {batch.shape[1:]}, using dim={detected_action_dim}") + global_accumulator = WelfordAccumulator(detected_action_dim) + reservoir = ReservoirAccumulator(detected_action_dim, max_size=reservoir_size) + + action = batch[:, :, :detected_action_dim].astype(np.float64) + + # Update global accumulator with all timesteps + flat_actions = action.reshape(-1, detected_action_dim) + global_accumulator.update(flat_actions) + reservoir.update(flat_actions.astype(np.float32)) + + sample_count += B + pbar.update(B) + + if sample_count % 1000 == 0: + elapsed = time.time() - start_time + rate = sample_count / elapsed + pbar.set_postfix({"rate": f"{rate:.1f} samples/s"}) + + if max_samples is not None and sample_count >= max_samples: + print(f"\nReached max_samples limit: {max_samples}") + break + pbar.close() + elapsed_time = time.time() - start_time + + if global_accumulator is None: + raise RuntimeError("No samples processed - dataset is empty or no actions found") + + print(f"\nProcessed {sample_count} samples in {elapsed_time:.1f}s ({sample_count / elapsed_time:.1f} samples/s)") + + # Compile results. Metadata is kept minimal; callers in main() attach + # dataset-descriptor fields after instantiation. + results: dict[str, Any] = { + "metadata": { + "action_dim": detected_action_dim, + "chunk_length": chunk_length, + "num_samples_stats": sample_count, + }, + "global": { + "mean": global_accumulator.get_mean().tolist(), + "std": global_accumulator.get_std().tolist(), + "min": global_accumulator.get_min().tolist(), + "max": global_accumulator.get_max().tolist(), + "q01": reservoir.quantile(0.01).tolist(), + "q99": reservoir.quantile(0.99).tolist(), + }, + } + + if return_accumulators: + # Multi-rank callers merge the raw accumulators via + # ``_merge_welford`` / ``_merge_reservoirs`` on rank 0 before + # rebuilding the final ``global`` block. Include the live + # reservoir buffer (only the populated prefix) so no samples are + # wasted in the merge. + results["_accumulators"] = { + "count": int(global_accumulator.count), + "mean": global_accumulator.mean.copy(), + "m2": global_accumulator.m2.copy(), + "min": global_accumulator.min_val.copy(), + "max": global_accumulator.max_val.copy(), + "reservoir_buffer": reservoir.buffer[: min(reservoir.count, reservoir.max_size)].copy(), + "reservoir_count": int(reservoir.count), + "reservoir_max_size": int(reservoir.max_size), + } + + return results + + +def get_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Compute action statistics for a dataset") + parser.add_argument("--config", help="Path to the config file", required=True) + parser.add_argument("--split", help="Dataset split (train/val)", default="train") + parser.add_argument("--output", help="Output JSON file path", default=None) + + # Processing options + parser.add_argument( + "--action-key", + type=str, + default="action", + help="Key for action tensor in dataset output (default: 'action')", + ) + parser.add_argument( + "--action-dim", + type=int, + default=None, + help="Number of action dimensions to use (default: auto-detect)", + ) + parser.add_argument( + "--max-samples", + type=int, + default=None, + help=( + "If set and smaller than the full dataset size, draw a random " + "subset of this size (without replacement, deterministic via " + "--sampling-seed) before iterating. Gives unbiased stats on " + "datasets too large to fully iterate. Default: None (full data)." + ), + ) + parser.add_argument( + "--sampling-seed", + type=int, + default=42, + help=( + "Seed for the random subset sampler used when --max-samples " + "truncates the dataset. Default: 42 (reproducible)." + ), + ) + parser.add_argument( + "--reservoir-size", + type=int, + default=50_000, + help="Reservoir size for quantile estimation (default: 50000)", + ) + parser.add_argument( + "--skip-rotation-dims", + type=str, + default="auto", + help=( + "Force identity stats (mean=0, std=1, min=-1, max=1, q01=-1, q99=1) " + "on given dim indices so downstream normalization is a pass-through. " + 'Default "auto" infers from action_dim + --rotation-format (supports ' + "9D camera, 10D single-arm, 20D dual-arm, 29D AgiBot FK-pose, " + "57D HandPose rot6d). " + 'Pass explicit indices/ranges to override, e.g. "3-8" or "3-8,12-17,36-41". ' + 'Pass "none" (or an empty string) to disable skipping.' + ), + ) + parser.add_argument( + "--rotation-format", + type=str, + default="rot6d", + choices=list(ROTATION_FORMAT_DIM), + help="Rotation format used by the dataset (for --skip-rotation-dims auto).", + ) + + # DataLoader tuning + parser.add_argument("--batch-size", type=int, default=256, help="DataLoader batch size") + parser.add_argument("--num-workers", type=int, default=48, help="DataLoader num_workers") + parser.add_argument("--prefetch-factor", type=int, default=2, help="DataLoader prefetch_factor") + parser.add_argument( + "--enable-fast-init", + action="store_true", + help=( + "Enable BaseActionLeRobotDataset fast shard metadata initialization when " + "the dataset supports it. Useful for multi-shard AgiBot/RoboMIND/HandPose stats." + ), + ) + parser.add_argument( + "--fast-init-max-workers", + type=int, + default=64, + help="Max metadata prefetch workers used with --enable-fast-init (default: 64)", + ) + + # Sample stride override. Training uses stride=1 (overlapping chunks); + # stats converge equally well on non-overlapping chunks and run ~stride× + # faster. Applied to each ``BaseActionLeRobotDataset`` instance after + # construction but before episode indices are built. Datasets that don't + # expose ``_sample_stride`` are left untouched. + parser.add_argument( + "--sample-stride", + type=int, + default=16, + help=( + "Sample stride to use for stats computation (default: 16 = " + "non-overlapping chunks of ``chunk_length=16``). Passed to each " + "LeRobot-backed dataset's ``_sample_stride`` attribute before the " + "episode index is built. Use ``--sample-stride 1`` to match " + "training semantics exactly (slower but identical numerics)." + ), + ) + + # Dataset filter (for experiments that bundle multiple datasets, e.g. + # ``action_midtrain_exp004_onlyActionData`` — pick one entry at a time). + parser.add_argument( + "--dataset-name", + type=str, + default=None, + help=( + "If set, keep only the entry in ``list_of_datasets`` whose ``name`` " + "starts with this string (case-sensitive prefix match). Used when " + "the experiment bundles several datasets (e.g. " + "``action_midtrain_exp004_onlyActionData``) and you want stats for " + "just one. Default: None (keep all entries)." + ), + ) + + # Hydra-style config overrides + parser.add_argument( + "opts", + help="Modify config options (e.g., experiment=libero_exp)", + default=None, + nargs=argparse.REMAINDER, + ) + + return parser.parse_args() + + +# --------------------------------------------------------------------------- +# Distributed (torchrun) helpers +# --------------------------------------------------------------------------- +# The heavy cost on large multi-shard datasets (e.g. ``HandPoseDataset`` with +# ~900 shards) is *index construction* inside ``_register_sources`` — each +# shard opens ``episodes.parquet`` / ``subtasks.parquet`` and builds a +# per-episode span table. All shards are processed sequentially in a single +# process, so the 3k-second ``init_data_loader`` time dominates the ~3-minute +# actual stats compute. +# +# The shard registration API already supports slicing (``_register_sources( +# indices)``) so each rank can register only its share of shards. We exploit +# this here: when launched via ``torchrun``, each rank registers +# ``shards[rank::world_size]``, computes partial Welford + reservoir on its +# slice, then ``all_gather_object`` merges into a global set of stats on +# rank 0 (which writes the file). Other ranks exit. + + +def _init_distributed() -> tuple[int, int, bool]: + """Initialize the default process group from ``torchrun`` env vars. + + Uses the ``gloo`` backend (pure CPU — this script has no GPU work). + Returns ``(rank, world_size, is_distributed)``; when ``WORLD_SIZE`` is 1 + or env vars are missing, returns ``(0, 1, False)`` and no process group + is created. + """ + world_size = int(_os.environ.get("WORLD_SIZE", "1")) + rank = int(_os.environ.get("RANK", "0")) + if world_size > 1 and not dist.is_initialized(): + dist.init_process_group(backend="gloo") + return rank, world_size, world_size > 1 + + +def _register_rank_shards(unified_dataset: Any, rank: int, world_size: int) -> None: + """Register each inner dataset's shards using a rank-sliced index set. + + Replaces the default ``_ensure_sources_registered()`` path with round-robin + shard assignment: each rank only calls ``_register_sources(shards[rank:: + world_size])`` on datasets that expose ``_all_shard_roots``. Datasets + without sharded roots (single-shard LeRobot, plain ``IterableDataset``) + fall back to a full registration on every rank — acceptable because + those are cheap. + """ + from cosmos_framework.data.vfm.action.unified_dataset import MapToIterableAdapter + + for entry in unified_dataset._datasets: + ds = entry["dataset"] + if isinstance(ds, MapToIterableAdapter): + ds = ds.dataset + if not hasattr(ds, "_register_sources"): + continue + shard_roots = getattr(ds, "_all_shard_roots", []) + if not shard_roots: + ds._register_sources() + continue + n = len(shard_roots) + my_indices = list(range(rank, n, world_size)) + print( + f" [rank {rank}/{world_size}] {ds.__class__.__name__}: " + f"registering {len(my_indices)}/{n} shards (round-robin)" + ) + ds._register_sources(my_indices) + unified_dataset._sources_initialized = True + + +def _merge_welford( + counts: list[int], + means: list[np.ndarray], + m2s: list[np.ndarray], + mins: list[np.ndarray], + maxs: list[np.ndarray], +) -> tuple[int, np.ndarray, np.ndarray, np.ndarray, np.ndarray]: + """Pairwise-merge Welford stats via Chan et al.'s parallel algorithm. + + All per-rank arrays must share the same ``dim``. ``min``/``max`` are + merged element-wise. Ranks with ``count == 0`` are skipped. + """ + total_count = 0 + mean = None + m2 = None + min_val = None + max_val = None + for c, mu, v, mn, mx in zip(counts, means, m2s, mins, maxs): + if c == 0: + continue + if total_count == 0: + total_count, mean, m2, min_val, max_val = c, mu.copy(), v.copy(), mn.copy(), mx.copy() + continue + n_a, n_b = total_count, c + n_new = n_a + n_b + delta = mu - mean + mean = mean + delta * (n_b / n_new) + m2 = m2 + v + (delta * delta) * (n_a * n_b / n_new) + total_count = n_new + np.minimum(min_val, mn, out=min_val) + np.maximum(max_val, mx, out=max_val) + return total_count, mean, m2, min_val, max_val + + +def _merge_reservoirs( + buffers: list[np.ndarray], + max_size: int, + seed: int = 42, +) -> np.ndarray: + """Merge per-rank reservoir buffers into a single reservoir. + + Each rank's buffer is assumed to be an unbiased uniform sample of its + local stream. We concatenate all buffers then uniformly sub-sample + ``max_size`` rows. This is strictly unbiased only when every rank + processed the same number of samples (the common case here, since we + split ``max_samples`` evenly across ranks) — accurate enough for + quantile estimation in all practical regimes. + """ + non_empty = [b for b in buffers if b.size > 0] + if not non_empty: + return np.empty((0, 0), dtype=np.float32) + concat = np.concatenate(non_empty, axis=0) + if concat.shape[0] <= max_size: + return concat + rng = np.random.RandomState(seed) + keep = rng.choice(concat.shape[0], size=max_size, replace=False) + keep.sort() + return concat[keep] + + +def _build_action_dataloader( + dl_config: Any, + action_key: str, + batch_size: int = 64, + num_workers: int = 16, + prefetch_factor: int = 2, + max_samples: int | None = None, + sampling_seed: int = 42, + dataset_name: str | None = None, + sample_stride: int = 1, + enable_fast_init: bool = False, + fast_init_max_workers: int = 64, + rank: int = 0, + world_size: int = 1, +) -> tuple[DataLoader, dict[str, Any]]: + """Instantiate the action dataset from config and wrap it in a DataLoader. + + Must be called inside a ``data_loader_init()`` context. + + When ``max_samples`` is set and smaller than the concatenated dataset size, + a random subset of size ``max_samples`` is drawn (without replacement, + seeded by ``sampling_seed``) and iterated in sorted order via + :class:`SortedSubsetSampler`. Random subset + sorted iteration gives + unbiased stats + cache-friendly access (important for datasets with many + inner LeRobotDatasets behind an LRU cache). Otherwise the full dataset is + iterated in natural order. + + When ``dataset_name`` is set, ``list_of_datasets`` is filtered in-place + to keep only the entry whose ``name`` field starts with that prefix. This + is useful when running against an experiment that bundles multiple + datasets (e.g. ``action_midtrain_exp004_onlyActionData``) — pick one at + a time. + + Returns: + (dataloader, dataset_params) where ``dataset_params`` describes each + inner dataset (class, pose_convention, rotation_format, mode, ...) + and is written into the stats JSON metadata. ``dataset_params`` also + carries ``total_len`` and, when sampling is enabled, + ``sampled_len`` and ``sampling_seed``. + """ + from cosmos_framework.data.vfm.action.unified_dataset import MapToIterableAdapter + + ds_config = dl_config.dataloaders.action_data.dataloader.dataset + ds_config.tokenizer_config = None + + # Optional filter: keep only the entry whose ``name`` starts with ``dataset_name``. + if dataset_name is not None: + all_names = [entry.get("name", "") for entry in ds_config.list_of_datasets] + matched = [entry for entry in ds_config.list_of_datasets if str(entry.get("name", "")).startswith(dataset_name)] + if not matched: + raise ValueError( + f"--dataset-name={dataset_name!r} did not match any entry. " + f"Available names in list_of_datasets: {all_names}" + ) + if len(matched) > 1: + matched_names = [entry.get("name") for entry in matched] + print( + f" WARNING: --dataset-name={dataset_name!r} matched {len(matched)} entries " + f"({matched_names}); keeping only the first." + ) + matched = matched[:1] + with open_dict(ds_config): + ds_config.list_of_datasets = matched + print( + f" Filtered list_of_datasets to 1 entry (name={matched[0].get('name')!r}) " + f"from {len(all_names)} candidates." + ) + + def _cfg_get(cfg: Any, key: str, default: Any = None) -> Any: + """Safely read ``key`` and unwrap OmegaConf containers to plain Python + types, so downstream metadata never leaks ``ListConfig`` / ``DictConfig`` + into the JSON pretty-printer or ``json.dumps`` fallback. + """ + try: + if key not in cfg: + return default + val = cfg[key] + except Exception: + return default + try: + from omegaconf import DictConfig, ListConfig, OmegaConf + + if isinstance(val, (DictConfig, ListConfig)): + return OmegaConf.to_container(val, resolve=True) + except ImportError: + pass + return val + + def _resolve_target(target: Any) -> Callable | None: + """Resolve a ``_target_`` field (string or class) to a callable; None on failure.""" + if callable(target): + return target + if isinstance(target, str): + module_path, _, attr = target.rpartition(".") + if not module_path: + return None + try: + mod = importlib.import_module(module_path) + return getattr(mod, attr, None) + except Exception: + return None + return None + + def _callable_accepts_kwarg(target: Any, kwarg: str) -> bool: + """Check whether ``target.__init__`` (or factory) accepts ``kwarg``. + + Returns True only if the signature **explicitly** lists ``kwarg`` as a + named parameter. A bare ``**kwargs`` is intentionally NOT treated as + acceptance, because factories like ``get_umi_dataset(..., **kwargs)`` + forward extras into a struct-mode OmegaConf config via + ``cfg.update(kwargs)``, which raises ``ConfigKeyError`` on unknown + keys. For those we rely on the post-instantiation + ``ds._skip_video_loading = True`` fallback below. + + Returns False on any introspection failure (safer to under-inject + than blow up at instantiation time). + """ + fn = _resolve_target(target) + if fn is None: + return False + try: + sig = inspect.signature(fn) + except (TypeError, ValueError): + return False + for p in sig.parameters.values(): + if p.name == kwarg and p.kind is not inspect.Parameter.VAR_KEYWORD: + return True + return False + + dataset_params: dict[str, Any] = {"datasets": []} + for entry in ds_config.list_of_datasets: + ds_dict = entry.dataset + # Only disable normalization if the dataset supports that kwarg. Some + # datasets (e.g. EmbodimentCGripperDataset) don't take ``action_normalization``. + if "action_normalization" in ds_dict: + with open_dict(ds_dict): + ds_dict.action_normalization = None + # Inject ``skip_video_loading=True`` only when the target class/factory + # explicitly lists it as a named kwarg. Everything else — subclasses + # that don't forward ``**kwargs`` to super (BridgeOrigLeRobotDataset, + # DROIDLeRobotDataset, RoboMINDFrankaDataset, ...) AND factories that + # forward ``**kwargs`` into a struct-mode OmegaConf config + # (``get_umi_dataset`` does ``cfg.update(kwargs)``) — falls back to + # the post-instantiation ``ds._skip_video_loading = True`` below, + # which the base class honors via attribute lookup. + target_cfg = _cfg_get(ds_dict, "_target_") + if _callable_accepts_kwarg(target_cfg, "skip_video_loading"): + with open_dict(ds_dict): + ds_dict.skip_video_loading = True + if enable_fast_init and _callable_accepts_kwarg(target_cfg, "enable_fast_init"): + with open_dict(ds_dict): + ds_dict.enable_fast_init = True + if enable_fast_init and _callable_accepts_kwarg(target_cfg, "fast_init_max_workers"): + with open_dict(ds_dict): + ds_dict.fast_init_max_workers = fast_init_max_workers + # Extract descriptive parameters into metadata (for naming stats files). + # ``embodiment_type`` / ``root`` / ``sample_stride`` may be absent from + # the config when the dataset class hard-codes them in ``__init__`` + # (e.g. Bridge / DROID / Fractal); we fill those in after instantiation + # from the instance attributes below. + target = _cfg_get(ds_dict, "_target_", "unknown") + dataset_params["datasets"].append( + { + "name": _cfg_get(entry, "name"), + # ``target`` is typically a class object whose ``repr`` is + # ``".'>"``. Prefer ``__name__`` + # when available; otherwise strip the ```` wrapper. + "class": getattr(target, "__name__", None) + or (str(target).rsplit(".", 1)[-1].rstrip("'>") if target else "unknown"), + "embodiment_type": _cfg_get(ds_dict, "embodiment_type"), + "pose_convention": _cfg_get(ds_dict, "pose_convention"), + "rotation_format": _cfg_get(ds_dict, "rotation_format"), + "mode": _cfg_get(ds_dict, "mode"), + "chunk_length": _cfg_get(ds_dict, "chunk_length"), + "root": _cfg_get(ds_dict, "root"), + "sample_stride": _cfg_get(ds_dict, "sample_stride"), + } + ) + + unified_dataset = instantiate(ds_config) + unified_dataset._transform = lambda data_dict, resolution=None: data_dict + + # Override ``_sample_stride`` on each inner BaseActionLeRobotDataset *before* + # ``_ensure_sources_registered()`` builds the episode index. This lets the + # stats run use non-overlapping chunks (stride=chunk_length) for speed + # without touching training-side defaults. Datasets that don't expose the + # attribute (e.g. UMI BaseDataset, IterableDatasets) are skipped silently. + if sample_stride != 1: + for entry in unified_dataset._datasets: + ds = entry["dataset"] + if isinstance(ds, MapToIterableAdapter): + ds = ds.dataset + if hasattr(ds, "_sample_stride"): + ds._sample_stride = sample_stride + if enable_fast_init: + for entry in unified_dataset._datasets: + ds = entry["dataset"] + if isinstance(ds, MapToIterableAdapter): + ds = ds.dataset + if hasattr(ds, "_enable_fast_init"): + ds._enable_fast_init = True + if hasattr(ds, "_fast_init_max_workers"): + ds._fast_init_max_workers = fast_init_max_workers + + if world_size > 1: + _register_rank_shards(unified_dataset, rank=rank, world_size=world_size) + else: + unified_dataset._ensure_sources_registered() + + inner_datasets: list[Any] = [] + total_len = 0 + for idx, entry in enumerate(unified_dataset._datasets): + ds = entry["dataset"] + if isinstance(ds, MapToIterableAdapter): + ds = ds.dataset + ds._skip_video_loading = True + inner_datasets.append(ds) + ds_len = len(ds) + total_len += ds_len + print(f" Inner dataset: {ds.__class__.__name__}, len={ds_len}") + + # Backfill fields that the dataset class hard-codes in ``__init__`` + # (not visible in the config). Falls through silently if the instance + # doesn't expose a matching attribute — we just keep the None from + # the pre-instantiation record. + if idx < len(dataset_params["datasets"]): + dp_entry = dataset_params["datasets"][idx] + if not dp_entry.get("embodiment_type"): + dp_entry["embodiment_type"] = getattr(ds, "_embodiment_type", None) + if not dp_entry.get("pose_convention"): + dp_entry["pose_convention"] = getattr(ds, "_pose_convention", None) + if not dp_entry.get("rotation_format"): + dp_entry["rotation_format"] = getattr(ds, "_rotation_format", None) + if not dp_entry.get("sample_stride"): + dp_entry["sample_stride"] = getattr(ds, "_sample_stride", None) + if not dp_entry.get("root"): + # Base class exposes ``_all_shard_roots`` (list of shard roots); + # for single-shard datasets use it verbatim, else the common + # parent directory across all shard roots. + shard_roots = getattr(ds, "_all_shard_roots", None) + if shard_roots: + if len(shard_roots) == 1: + dp_entry["root"] = shard_roots[0] + else: + dp_entry["root"] = _os.path.commonpath(shard_roots) + if total_len == 0: + raise RuntimeError( + "All inner datasets are empty (total len=0). Most likely the data " + "files are missing or inaccessible on this machine. Check the " + "dataset root paths and warnings printed above." + ) + + combined_ds = ConcatDataset(inner_datasets) if len(inner_datasets) > 1 else inner_datasets[0] + + def collate_actions(batch: list[dict]) -> torch.Tensor | np.ndarray: + actions = [sample[action_key] for sample in batch] + is_tensor = isinstance(actions[0], torch.Tensor) + # Fast path: uniform chunk length (the common case). Preserves the + # (B, T, D) shape so ``chunk_length`` metadata reflects the true + # sampling window. + shapes = {tuple(a.shape) for a in actions} + if len(shapes) == 1: + return torch.stack(actions) if is_tensor else np.stack(actions) + # Fallback: variable-length chunks (e.g. HandPoseDataset with + # ``snap_to_subtask``). Per-frame stats are order-/shape-invariant, + # so we flatten each chunk and concatenate along the time axis. + # Downstream compute_action_stats reshapes 2D input to (N, 1, D) and + # computes stats over all rows — numerics unchanged. The reported + # ``chunk_length`` metadata will be 1 in this mode; the true + # config-side chunk_length is still captured in dataset_params. + if is_tensor: + dim = actions[0].shape[-1] + return torch.cat([a.reshape(-1, dim) for a in actions], dim=0) + dim = actions[0].shape[-1] + return np.concatenate([np.asarray(a).reshape(-1, dim) for a in actions], axis=0) + + dataset_params["total_len"] = total_len + + + # then yield them **sorted ascending** via SortedSubsetSampler. Random + # iteration order (as SubsetRandomSampler would give) causes catastrophic + # LRU cache misses on datasets like RoboMINDFrankaDataset that hold + # O(100) inner LeRobotDatasets behind a size-32 LRU — observed ~170x + # slowdown (10K samples/s sorted vs ~60 samples/s random). Stats only + # depend on the *set* of samples, not on visit order, so sorting is free. + sampler: Sampler[int] | None = None + is_iterable = isinstance(combined_ds, IterableDataset) or any( + isinstance(ds, IterableDataset) for ds in inner_datasets + ) + # With multi-rank launches each rank only holds ``shards[rank::world_size]``, + # so ``total_len`` above is already this rank's slice. Split the global + # ``max_samples`` evenly across ranks; combined across all ranks we still + # process ~``max_samples`` rows total. Use a per-rank seed offset so + # different ranks draw different (non-overlapping) subsets from their + # disjoint shard slices. + if max_samples is not None and max_samples > 0 and world_size > 1: + per_rank_max = (max_samples + world_size - 1) // world_size + per_rank_seed = sampling_seed + rank + else: + per_rank_max = max_samples + per_rank_seed = sampling_seed + + if is_iterable and per_rank_max is not None and per_rank_max > 0: + # IterableDataset doesn't support DataLoader samplers. Rely on the + # ``max_samples`` hard-break inside ``collect_stats_loop`` instead. + # Note: stats may be biased toward whatever order the underlying + # iterable yields (e.g. early shards) — acceptable for stats-only use. + dataset_params["sampled_len"] = per_rank_max + print( + f" IterableDataset detected — skipping subset sampler. " + f"Will stop after {per_rank_max:,} samples in natural iteration " + "order (may be biased toward early shards)." + ) + elif per_rank_max is not None and per_rank_max > 0 and per_rank_max < total_len: + g = torch.Generator().manual_seed(per_rank_seed) + indices = torch.randperm(total_len, generator=g)[:per_rank_max].tolist() + sampler = SortedSubsetSampler(indices) + dataset_params["sampled_len"] = per_rank_max + dataset_params["sampling_seed"] = per_rank_seed + print( + f" [rank {rank}/{world_size}] Random subset sampling: " + f"{per_rank_max:,} / {total_len:,} " + f"(seed={per_rank_seed}, iterated in sorted order for cache locality)" + ) + elif per_rank_max is not None and per_rank_max > 0: + print( + f" [rank {rank}/{world_size}] per_rank_max={per_rank_max:,} >= " + f"total_len={total_len:,}; iterating this rank's full slice in natural order." + ) + + dataloader = DataLoader( + combined_ds, + batch_size=batch_size, + sampler=sampler, + shuffle=False, + num_workers=num_workers, + collate_fn=collate_actions, + prefetch_factor=prefetch_factor, + persistent_workers=True, + ) + return dataloader, dataset_params + + +def _resolve_default_output_path( + dataset_params: dict[str, Any], + split: str, + exp_name: str, + cli_rotation_format: str, +) -> Path: + """Compute the default output path under ``DEFAULT_OUTPUT_DIR``. + + Matches ``BaseActionLeRobotDataset._normalizer_filename`` exactly so the + file the loader will look up is the one we write here: + + - SE(3) pose datasets (``embodiment`` + ``pose`` + ``rot``): + ``normalizers/__.json`` + - joint-space datasets (``pose`` / ``rot`` both ``None``): + ``normalizers/.json`` + - ``_`` suffix is appended for non-train splits. + + Falls back to ``action_stats__.json`` only when even + ``embodiment`` is unknown. + """ + datasets = dataset_params.get("datasets") or [] + first = datasets[0] if datasets else {} + embodiment = first.get("embodiment_type") + pose = first.get("pose_convention") + # Datasets that don't expose ``rotation_format`` as a kwarg (e.g. DROID, + # Fractal, RoboMIND) still produce rot6d actions internally; fall back + # to the CLI ``--rotation-format`` which defaults to rot6d. For + # joint-space datasets (e.g. Embodiment_b) have ``pose`` set to ``None`` — keep + # ``rot`` ``None`` too so the filename drops both suffixes instead of + # synthesizing a bogus ``_rot6d``. FK-pose datasets such as Embodiment C + # and AgiBotWorld-Beta expose both ``pose`` and ``rotation_format``. + rot = first.get("rotation_format") + if rot is None and pose is not None: + rot = cli_rotation_format + + if embodiment and pose and rot: + stem = f"{embodiment}_{pose}_{rot}" + elif embodiment and pose is None and rot is None: + stem = str(embodiment) + else: + return DEFAULT_OUTPUT_DIR / f"action_stats_{exp_name}_{split}.json" + + if split != "train": + stem = f"{stem}_{split}" + return DEFAULT_OUTPUT_DIR / f"{stem}.json" + + +def main() -> None: + args = get_args() + + # Initialize distributed (torchrun) early so all the ``[rank R/W]`` log + # prefixes below have the right values. ``WORLD_SIZE == 1`` (plain + # ``python`` invocation) is a no-op — the code paths below are identical + # to the single-process pre-change behavior. + rank, world_size, is_distributed = _init_distributed() + if is_distributed: + print(f"[dist] rank={rank} world_size={world_size} backend=gloo") + + # Infer experiment name for default output path + exp_name = "default" + if args.opts: + for opt in args.opts: + if "experiment=" in opt: + exp_name = opt.split("=")[1] + exp_name = exp_name.strip("'\"") + break + + # Load config + if rank == 0: + print(f"Loading config: {args.config}") + print(f"Config overrides: {args.opts}") + try: + config = load_config(args.config, args.opts, enable_one_logger=False) + except Exception as e: + print(f"Failed to load config: {e}") + raise + + # Instantiate dataloader + with data_loader_init(): + if args.split == "train": + dl_config = config.dataloader_train + else: + dl_config = config.dataloader_val + + if rank == 0: + print(f"Instantiating {args.split} dataloader...") + try: + dataloader, dataset_params = _build_action_dataloader( + dl_config, + args.action_key, + batch_size=args.batch_size, + num_workers=args.num_workers, + prefetch_factor=args.prefetch_factor, + max_samples=args.max_samples, + sampling_seed=args.sampling_seed, + dataset_name=args.dataset_name, + sample_stride=args.sample_stride, + enable_fast_init=args.enable_fast_init, + fast_init_max_workers=args.fast_init_max_workers, + rank=rank, + world_size=world_size, + ) + except Exception as e: + print(f"Failed to instantiate dataloader: {e}") + raise + + # Resolve output path now that we know embodiment / pose / rotation. + if args.output: + output_path = Path(args.output) + else: + output_path = _resolve_default_output_path( + dataset_params=dataset_params, + split=args.split, + exp_name=exp_name, + cli_rotation_format=args.rotation_format, + ) + if rank == 0: + print(f" Default output path: {output_path}") + + # Compute statistics. In multi-rank mode each rank iterates its own + # shard slice and returns raw accumulators; we merge them below before + # re-deriving the final ``global`` block. The global cap (``max_samples``) + # stays as-is for single-process mode, but is split evenly across ranks + # inside ``_build_action_dataloader`` when ``world_size > 1``. + per_rank_cap = args.max_samples + if is_distributed and per_rank_cap is not None: + per_rank_cap = (per_rank_cap + world_size - 1) // world_size + results = compute_action_stats( + dataloader=dataloader, + action_key=args.action_key, + max_samples=per_rank_cap, + action_dim=args.action_dim, + reservoir_size=args.reservoir_size, + return_accumulators=is_distributed, + ) + + if is_distributed: + # Gather every rank's raw accumulators on rank 0, merge via Chan's + # parallel Welford algorithm + reservoir union, then rebuild the + # ``global`` block. ``all_gather_object`` is symmetric (all ranks + # receive the list) so we only have one collective call; the + # non-zero ranks simply discard their copy. + local_accum = results.pop("_accumulators") + gathered: list[dict[str, Any]] = [None] * world_size # type: ignore[list-item] + dist.all_gather_object(gathered, local_accum) + + # Sum the raw (pre-cap) sample counts for metadata reporting. + total_samples = int(sum(g["count"] for g in gathered)) + + if rank == 0: + total_count, mean, m2, min_val, max_val = _merge_welford( + counts=[g["count"] for g in gathered], + means=[g["mean"] for g in gathered], + m2s=[g["m2"] for g in gathered], + mins=[g["min"] for g in gathered], + maxs=[g["max"] for g in gathered], + ) + if total_count == 0: + raise RuntimeError("Distributed merge saw zero total samples across all ranks.") + + merged_reservoir = _merge_reservoirs( + buffers=[g["reservoir_buffer"] for g in gathered], + max_size=args.reservoir_size, + seed=args.sampling_seed, + ) + # ``std`` from Chan-merged ``m2``: sample variance = m2 / (N-1). + std = np.sqrt(m2 / max(total_count - 1, 1)) + q01 = np.quantile(merged_reservoir, 0.01, axis=0) if merged_reservoir.size else np.zeros_like(mean) + q99 = np.quantile(merged_reservoir, 0.99, axis=0) if merged_reservoir.size else np.zeros_like(mean) + + results["global"] = { + "mean": mean.tolist(), + "std": std.tolist(), + "min": min_val.tolist(), + "max": max_val.tolist(), + "q01": q01.astype(np.float32).tolist(), + "q99": q99.astype(np.float32).tolist(), + } + results["metadata"]["num_samples_stats"] = total_samples + results["metadata"]["world_size"] = world_size + print( + f"\n[dist] merged across world_size={world_size}: total_samples={total_samples:,}, " + f"reservoir_size={merged_reservoir.shape[0]:,}" + ) + # Non-rank-0 processes have nothing more to do. + dist.barrier() + if rank != 0: + dist.destroy_process_group() + return + + # Zero-out rotation dims so normalization is a pass-through for them. + # Keep this *before* the metadata re-assembly below so skip_rotation_dims + # is ready when we compose the final payload. + act_dim = results["metadata"]["action_dim"] + spec = (args.skip_rotation_dims or "").strip().lower() + if spec in ("", "none"): + skip_dims: list[int] = [] + print("\nSkip rotation: disabled (stats reflect raw data on all dims).") + elif spec == "auto": + skip_dims = get_rotation_dims(act_dim, args.rotation_format) + print( + f"\nAuto-detected rotation dims (action_dim={act_dim}, " + f"rotation_format={args.rotation_format}): {skip_dims or '[] (none)'}" + ) + else: + skip_dims = parse_dim_spec(args.skip_rotation_dims) + if skip_dims: + print(f"\nForcing identity stats for rotation dims: {skip_dims} (action_dim={act_dim})") + + if skip_dims: + # Preserve the raw (pre-replacement) stats under ``global_raw`` so + # consumers can inspect the data's true distribution on rotation dims + # even after ``global`` has been overwritten with identity values. + results["global_raw"] = copy.deepcopy(results["global"]) + _apply_skip_rotation(results, skip_dims, act_dim) + + # Assemble the final flat metadata block. Identity fields first (who/what + # these stats describe), then stats-run configuration (how they were + # computed). Source priority for ``rotation_format``: dataset config > + # CLI ``--rotation-format``. Fields with no source stay ``None`` rather + # than being silently dropped, so gaps are visible. + first = (dataset_params.get("datasets") or [{}])[0] + meta = results["metadata"] + ordered: dict[str, Any] = { + # Identity + "embodiment_type": first.get("embodiment_type"), + "pose_convention": first.get("pose_convention"), + # Leave ``rotation_format`` as ``None`` for joint-space datasets; do + # NOT fall back to the CLI default, which would silently claim + # "rot6d" on a dataset with no rotation. + "rotation_format": first.get("rotation_format"), + "action_dim": meta["action_dim"], + "skip_rotation_dims": skip_dims, + "chunk_length": meta["chunk_length"], + "sample_stride": first.get("sample_stride"), + # Dataset provenance + "dataset_name": first.get("name"), + "dataset_class": first.get("class"), + "dataset_root": first.get("root"), + "split": args.split, + # Stats-run configuration (how these numbers were computed) + "num_samples_stats": meta["num_samples_stats"], + "reservoir_size": args.reservoir_size, + } + if args.max_samples is not None: + ordered["max_samples"] = args.max_samples + ordered["sampling_seed"] = args.sampling_seed + + results["metadata"] = ordered + + # Save results. Use a compact pretty-printer so numeric arrays stay on + # one line (``json.dump(indent=2)`` puts every number on its own line, + # blowing a 17-line payload into ~900 lines). + # Serialize fully *before* touching disk: opening with ``"w"`` truncates + # the target, so a formatting crash would otherwise wipe any previously + # good file on disk. Stage to a sibling ``.tmp`` and atomically rename + # only after successful formatting. + output_path.parent.mkdir(parents=True, exist_ok=True) + try: + payload = _pretty_dump(results) + except Exception: + fallback = output_path.with_suffix(output_path.suffix + ".raw.json") + with open(fallback, "w") as f: + json.dump(results, f, indent=2, default=str) + print(f"\n[warning] pretty-print failed; raw results saved to: {fallback}") + raise + tmp_path = output_path.with_suffix(output_path.suffix + ".tmp") + with open(tmp_path, "w") as f: + f.write(payload) + _os.replace(tmp_path, output_path) + + print(f"\nResults saved to: {output_path}") + print("\nGlobal statistics:") + print(f" Mean: {results['global']['mean']}") + print(f" Std: {results['global']['std']}") + print(f" Min: {results['global']['min']}") + print(f" Max: {results['global']['max']}") + print(f" q01: {results['global']['q01']}") + print(f" q99: {results['global']['q99']}") + + if is_distributed: + dist.destroy_process_group() + + +if __name__ == "__main__": + main() diff --git a/cosmos_framework/data/vfm/action/compute_pose_stats.py b/cosmos_framework/data/vfm/action/compute_pose_stats.py new file mode 100644 index 0000000..a8c5060 --- /dev/null +++ b/cosmos_framework/data/vfm/action/compute_pose_stats.py @@ -0,0 +1,625 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Compute raw action translation and rotation statistics for camera and AV pose datasets. + +Both camera and AV datasets produce actions via ``pose_abs_to_rel`` with +layout ``[translation(3), rotation(...)]``. + +This script iterates dataset samples with ``translation_scale=1.0`` and +``rotation_scale=1.0`` to collect **raw** translation and rotation +values, then reports their distribution (mean, std, min, max, +percentiles) both globally and per-timestep for each block. The +reported per-dim std is what directly matches MSE loss scale, so ratios +like ``std_translation / std_rotation`` are what you want to use to +pick ``translation_scale`` and ``rotation_scale`` so the two loss blocks +contribute comparably. + +Usage: + # Camera – backward_framewise (used in inverse_dynamics / policy) + PYTHONPATH=. python cosmos_framework/data/vfm/action/compute_pose_stats.py \ + --dataset camera --split train --pose-convention backward_framewise --max-samples 1000 --max-frames 17 --rotation-format axisangle + + PYTHONPATH=. python cosmos_framework/data/vfm/action/compute_pose_stats.py \ + --dataset camera --split train --pose-convention backward_framewise --max-samples 1000 --max-frames 61 --rotation-format axisangle + + # AV – backward_framewise + PYTHONPATH=. python cosmos_framework/data/vfm/action/compute_pose_stats.py \ + --dataset av --split train --pose-convention backward_framewise --max-samples 1000 --max-frames 17 --rotation-format axisangle + + PYTHONPATH=. python cosmos_framework/data/vfm/action/compute_pose_stats.py \ + --dataset av --split train --pose-convention backward_framewise --max-samples 1000 --max-frames 61 --rotation-format axisangle + + # Clip per-frame outliers automatically at P99 of each block's L2 norm. + PYTHONPATH=. python cosmos_framework/data/vfm/action/compute_pose_stats.py \ + --dataset camera --split train --pose-convention backward_framewise --max-samples 1000 --max-frames 61 \ + --rotation-format axisangle --max-trans-norm 10 + + PYTHONPATH=. python cosmos_framework/data/vfm/action/compute_pose_stats.py \ + --dataset av --split train --pose-convention backward_framewise --max-samples 1000 --max-frames 61 \ + --rotation-format axisangle --max-trans-norm 10 +""" + +from __future__ import annotations + +import argparse +import json +import time +from pathlib import Path +from typing import Literal + +import numpy as np +import torch +from tqdm import tqdm + +from cosmos_framework.data.vfm.action.pose_utils import RotationConvention + +SCRIPT_DIR = Path(__file__).resolve().parent +DEFAULT_OUTPUT_DIR = SCRIPT_DIR / "stats" + +TRANSLATION_DIM = 3 +PoseConvention = Literal[ + "backward_framewise", + "backward_anchored", +] + + +# --------------------------------------------------------------------------- +# Welford accumulator +# --------------------------------------------------------------------------- +class WelfordAccumulator: + """Welford's online algorithm for numerically stable mean/variance.""" + + def __init__(self, dim: int) -> None: + self.dim = dim + self.count = 0 + self.mean = np.zeros(dim, dtype=np.float64) + self.m2 = np.zeros(dim, dtype=np.float64) + self.min_val = np.full(dim, np.inf, dtype=np.float64) + self.max_val = np.full(dim, -np.inf, dtype=np.float64) + + def update(self, x: np.ndarray) -> None: + """Update with a single sample (D,) or a batch (N, D).""" + if x.ndim == 1: + x = x.reshape(1, -1) + for sample in x: + self.count += 1 + delta = sample - self.mean + self.mean += delta / self.count + delta2 = sample - self.mean + self.m2 += delta * delta2 + self.min_val = np.minimum(self.min_val, sample) + self.max_val = np.maximum(self.max_val, sample) + + def get_std(self) -> np.ndarray: + if self.count < 2: + return np.zeros(self.dim, dtype=np.float64) + return np.sqrt(self.m2 / (self.count - 1)) + + def as_dict(self) -> dict: + return { + "mean": self.mean.tolist(), + "std": self.get_std().tolist(), + "min": self.min_val.tolist(), + "max": self.max_val.tolist(), + "count": self.count, + } + + +# --------------------------------------------------------------------------- +# Dataset creation helpers +# --------------------------------------------------------------------------- +def _create_camera_dataset( + split: str, + rotation_format: RotationConvention, + pose_convention: PoseConvention, + credential_path: str, + wdinfo_names: list[str] | None, +): + from cosmos_framework.data.vfm.action.camera_dataset_sharded import ( + CAMERA_WDINFOS, + CameraDatasetSharded, + ) + + if wdinfo_names: + wdinfo_paths = [CAMERA_WDINFOS[n] for n in wdinfo_names] + else: + wdinfo_paths = [CAMERA_WDINFOS["pretrained_clips_260307_100k_filtered"]] + + ds = CameraDatasetSharded( + wdinfo_paths=wdinfo_paths, + split=split, + shuffle=False, + fix_caption=True, + mode="forward_dynamics", + rotation_format=rotation_format, + pose_convention=pose_convention, + credential_path=credential_path, + translation_scale=1.0, + rotation_scale=1.0, + ) + print(f"CameraDatasetSharded wdinfos={wdinfo_names or ['pretrained_clips_260307_100k_filtered']}") + return ds + + +def _create_av_dataset( + split: str, + rotation_format: RotationConvention, + pose_convention: PoseConvention, + credential_path: str, + av_root: str, + av_history_len: float, + av_future_len: float, + av_fps: int, +): + from cosmos_framework.data.vfm.action.av_dataset import AVDataset + + ds = AVDataset( + root=av_root, + split=split, + fps=av_fps, + mode="policy", + history_len=av_history_len, + future_len=av_future_len, + rotation_format=rotation_format, + pose_convention=pose_convention, + credential_path=credential_path, + shuffle=False, + include_route_in_prompt=False, + translation_scale=1.0, + rotation_scale=1.0, + ) + print(f"AVDataset root={av_root}") + return ds + + +# --------------------------------------------------------------------------- +# Core computation +# --------------------------------------------------------------------------- +def _summarize_block( + kept_values: list[np.ndarray], + per_timestep_values: list[list[np.ndarray]], + chunk_length: int, + block_label: str, +) -> tuple[dict, dict]: + """Build the per-dim and L2-norm summary dicts for one (already-filtered) action block.""" + percentiles = [5, 10, 25, 50, 75, 90, 95] + + concat = np.concatenate(kept_values, axis=0) if kept_values else np.zeros((0, 0), dtype=np.float64) + if concat.size == 0: + raise RuntimeError(f"No {block_label} frames left after filtering") + + dim = concat.shape[1] + count = int(concat.shape[0]) + + global_mean = concat.mean(axis=0).tolist() + global_median = np.median(concat, axis=0).tolist() + global_std = (concat.std(axis=0, ddof=1) if count > 1 else np.zeros(dim)).tolist() + global_min = concat.min(axis=0).tolist() + global_max = concat.max(axis=0).tolist() + + # Single-scalar summary from the flattened pool. This is the RMS per element + # (about the pool mean) — the right quantity for picking one global scale + # factor that preserves the vector's internal geometry. + flat = concat.reshape(-1) + flat_mean = float(flat.mean()) + flat_std = float(flat.std(ddof=1)) if flat.size > 1 else 0.0 + + zero_dim = [0.0] * dim + per_timestep_means: list[list[float]] = [] + per_timestep_medians: list[list[float]] = [] + per_timestep_stds: list[list[float]] = [] + per_timestep_mins: list[list[float]] = [] + per_timestep_maxs: list[list[float]] = [] + for t in range(chunk_length): + vals = per_timestep_values[t] + if not vals: + per_timestep_means.append(zero_dim) + per_timestep_medians.append(zero_dim) + per_timestep_stds.append(zero_dim) + per_timestep_mins.append(zero_dim) + per_timestep_maxs.append(zero_dim) + continue + arr = np.stack(vals) + per_timestep_means.append(arr.mean(axis=0).tolist()) + per_timestep_medians.append(np.median(arr, axis=0).tolist()) + per_timestep_stds.append((arr.std(axis=0, ddof=1) if arr.shape[0] > 1 else np.zeros(dim)).tolist()) + per_timestep_mins.append(arr.min(axis=0).tolist()) + per_timestep_maxs.append(arr.max(axis=0).tolist()) + + l2_norms = np.linalg.norm(concat, axis=-1) # [N_kept] + + global_dict = { + "mean": global_mean, + "std": global_std, + "min": global_min, + "max": global_max, + "count": count, + "median": global_median, + "flat_mean": flat_mean, + "flat_std": flat_std, + } + + scale_name = "translation_scale" if block_label == "translation" else "rotation_scale" + raw_stats = { + "description": ( + f"Per-dim statistics on raw action {block_label} block (translation_scale=1.0, rotation_scale=1.0). " + f"Use flat_std (single scalar across all dims) to choose {scale_name} when you want a uniform " + f"scale that preserves the block's internal geometry; use per-dim std when per-dim rescaling is acceptable." + ), + "global": global_dict, + "per_timestep": { + "mean": per_timestep_means, + "median": per_timestep_medians, + "std": per_timestep_stds, + "min": per_timestep_mins, + "max": per_timestep_maxs, + }, + } + l2_stats = { + "description": f"L2 norm of raw {block_label} vectors across all frames.", + "median": float(np.median(l2_norms)), + "mean": float(np.mean(l2_norms)), + "std": float(np.std(l2_norms, ddof=1)) if len(l2_norms) > 1 else 0.0, + "min": float(np.min(l2_norms)), + "max": float(np.max(l2_norms)), + "percentiles": {str(p): float(np.percentile(l2_norms, p)) for p in percentiles}, + } + return raw_stats, l2_stats + + +def compute_action_stats( + dataset, + max_samples: int | None = None, + max_frames: int | None = None, + max_trans_norm: float | None = None, + max_rot_norm: float | None = None, + percentile_clip: float | None = None, +) -> dict: + """Iterate over *dataset* and collect raw action translation and rotation statistics. + + The dataset must be created with ``translation_scale=1.0`` and + ``rotation_scale=1.0`` so that the returned actions contain unmodified + translation and rotation values. + + If *max_frames* is given, each sample's action tensor is truncated to + the first *max_frames* frames before statistics are accumulated. + + Outlier filtering (applied per-frame, not per-sample): + * ``max_trans_norm``: drop frames whose translation L2 norm exceeds this value. + * ``max_rot_norm``: drop frames whose rotation L2 norm exceeds this value. + * ``percentile_clip``: if set (e.g. 99), the thresholds default to the P{n} + of the corresponding L2-norm distribution from the data actually seen. + Explicit ``max_*_norm`` arguments take precedence over this. + + Returns a dict ready for JSON serialisation. + """ + all_translations: list[np.ndarray] = [] + all_rotations: list[np.ndarray] = [] + chunk_length: int | None = None + rotation_dim: int | None = None + sample_count = 0 + start = time.time() + + pbar = tqdm(desc="Reading action tensors", unit="samples") + for sample in dataset: + action = sample["action"] + if isinstance(action, torch.Tensor): + action = action.detach().cpu().numpy() + if action.ndim == 1: + action = action.reshape(1, -1) + if max_frames is not None: + action = action[:max_frames] + + trans = action[:, :TRANSLATION_DIM].astype(np.float64) # [T, 3] + rot = action[:, TRANSLATION_DIM:].astype(np.float64) # [T, D_rot] + all_translations.append(trans) + all_rotations.append(rot) + + if rotation_dim is None: + rotation_dim = rot.shape[1] + if chunk_length is None: + chunk_length = trans.shape[0] + print( + f" action shape : {action.shape} (first {TRANSLATION_DIM} dims = translation, next {rotation_dim} = rotation)" + ) + print(f" chunk_length : {chunk_length}") + print(f" rotation_dim : {rotation_dim}") + + sample_count += 1 + pbar.update(1) + if sample_count % 1000 == 0: + elapsed = time.time() - start + pbar.set_postfix(rate=f"{sample_count / elapsed:.1f} s/s") + + if max_samples is not None and sample_count >= max_samples: + print(f"\nReached max_samples={max_samples}") + break + + pbar.close() + elapsed = time.time() - start + + if not all_translations: + raise RuntimeError("No samples processed – dataset is empty or no actions found") + assert chunk_length is not None and rotation_dim is not None + + print(f"\nProcessed {sample_count} samples in {elapsed:.1f}s ({sample_count / elapsed:.1f} samples/s)") + + # Per-frame L2 norms used for filtering and percentile-based threshold derivation. + trans_norms_per_sample = [np.linalg.norm(t, axis=-1) for t in all_translations] + rot_norms_per_sample = [np.linalg.norm(r, axis=-1) for r in all_rotations] + all_trans_norms = np.concatenate(trans_norms_per_sample) + all_rot_norms = np.concatenate(rot_norms_per_sample) + total_frames = int(all_trans_norms.shape[0]) + + if percentile_clip is not None: + if not (0.0 < percentile_clip <= 100.0): + raise ValueError(f"percentile_clip must be in (0, 100], got {percentile_clip}") + if max_trans_norm is None: + max_trans_norm = float(np.percentile(all_trans_norms, percentile_clip)) + print(f" Auto-derived max_trans_norm at P{percentile_clip:g} = {max_trans_norm:.6f}") + if max_rot_norm is None: + max_rot_norm = float(np.percentile(all_rot_norms, percentile_clip)) + print(f" Auto-derived max_rot_norm at P{percentile_clip:g} = {max_rot_norm:.6f}") + + kept_trans: list[np.ndarray] = [] + kept_rotations: list[np.ndarray] = [] + per_timestep_trans: list[list[np.ndarray]] = [[] for _ in range(chunk_length)] + per_timestep_rot: list[list[np.ndarray]] = [[] for _ in range(chunk_length)] + kept_frame_count = 0 + for trans, rot, tn, rn in zip(all_translations, all_rotations, trans_norms_per_sample, rot_norms_per_sample): + mask = np.ones(trans.shape[0], dtype=bool) + if max_trans_norm is not None: + mask &= tn <= max_trans_norm + if max_rot_norm is not None: + mask &= rn <= max_rot_norm + if mask.any(): + kept_trans.append(trans[mask]) + kept_rotations.append(rot[mask]) + kept_frame_count += int(mask.sum()) + upper = min(chunk_length, trans.shape[0]) + for t in range(upper): + if mask[t]: + per_timestep_trans[t].append(trans[t]) + per_timestep_rot[t].append(rot[t]) + + dropped = total_frames - kept_frame_count + filter_active = max_trans_norm is not None or max_rot_norm is not None + if filter_active: + pct = (100.0 * dropped / total_frames) if total_frames else 0.0 + print(f"\nOutlier filter: kept {kept_frame_count} / {total_frames} frames (dropped {dropped}, {pct:.2f}%)") + print(f" thresholds: max_trans_norm={max_trans_norm}, max_rot_norm={max_rot_norm}") + + raw_trans_stats, trans_l2_stats = _summarize_block(kept_trans, per_timestep_trans, chunk_length, "translation") + raw_rot_stats, rot_l2_stats = _summarize_block(kept_rotations, per_timestep_rot, chunk_length, "rotation") + + return { + "metadata": { + "translation_dim": TRANSLATION_DIM, + "rotation_dim": rotation_dim, + "chunk_length": chunk_length, + "num_samples": sample_count, + "total_frames": total_frames, + "kept_frames": kept_frame_count, + "dropped_frames": dropped, + "max_trans_norm": max_trans_norm, + "max_rot_norm": max_rot_norm, + "percentile_clip": percentile_clip, + "processing_time_s": round(elapsed, 2), + }, + "raw_translation_stats": raw_trans_stats, + "translation_l2_norm": trans_l2_stats, + "raw_rotation_stats": raw_rot_stats, + "rotation_l2_norm": rot_l2_stats, + } + + +# --------------------------------------------------------------------------- +# CLI +# --------------------------------------------------------------------------- +def parse_args() -> argparse.Namespace: + p = argparse.ArgumentParser( + description="Compute raw action translation statistics for camera / AV pose datasets.", + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + p.add_argument("--dataset", required=True, choices=["camera", "av"]) + p.add_argument("--split", default="train", choices=["train", "val", "full"]) + p.add_argument("--output", default=None, help="Output JSON path (auto-generated if omitted)") + p.add_argument("--max-samples", type=int, default=None) + p.add_argument( + "--max-frames", + type=int, + default=None, + help="Use only the first N frames per sample (e.g. 17). All frames used if omitted.", + ) + + # Pose options + p.add_argument( + "--rotation-format", default="rot6d", choices=["rot9d", "rot6d", "quat_xyzw", "euler_xyz", "axisangle"] + ) + p.add_argument( + "--pose-convention", + default="backward_framewise", + choices=["backward_anchored", "backward_framewise"], + ) + p.add_argument("--credential-path", default="credentials/gcp_training.secret") + + # AV-specific + p.add_argument("--av-root", default="s3://nv-00-10206-robot/cosmos3_action_data/av_v2_02182026_wdinfo/") + p.add_argument("--av-history-len", type=float, default=0.1) + p.add_argument("--av-future-len", type=float, default=6.0) + p.add_argument("--av-fps", type=int, default=10) + + # Camera-specific + p.add_argument( + "--camera-wdinfos", + nargs="*", + default=None, + help="Camera wdinfo keys (default: pretrained_clips_260307_100k). " + "See CAMERA_WDINFOS in camera_dataset_sharded.py for available keys.", + ) + + # Outlier filtering (applied per-frame, not per-sample). + p.add_argument( + "--max-trans-norm", + type=float, + default=None, + help="Drop frames whose translation L2 norm exceeds this value. Takes precedence over --percentile-clip.", + ) + p.add_argument( + "--max-rot-norm", + type=float, + default=None, + help="Drop frames whose rotation L2 norm exceeds this value. Takes precedence over --percentile-clip.", + ) + p.add_argument( + "--percentile-clip", + type=float, + default=None, + help="Auto-derive max_trans_norm / max_rot_norm from this percentile " + "(e.g. 99 or 99.5) of the observed L2-norm distributions. " + "Ignored for a block if --max-{trans,rot}-norm is given explicitly.", + ) + + return p.parse_args() + + +def main() -> None: + args = parse_args() + + tag = f"{args.dataset}_{args.rotation_format}_{args.pose_convention}" + if args.output: + output_path = Path(args.output) + else: + DEFAULT_OUTPUT_DIR.mkdir(parents=True, exist_ok=True) + output_path = DEFAULT_OUTPUT_DIR / f"pose_stats_{tag}_{args.split}.json" + + print(f"Dataset : {args.dataset}") + print(f"Split : {args.split}") + print(f"Rotation format: {args.rotation_format}") + print(f"Rel pose format: {args.pose_convention}") + print(f"Max samples : {args.max_samples}") + print(f"Max frames : {args.max_frames}") + print(f"Normalization : none (raw values, translation_scale=1.0, rotation_scale=1.0)") + filter_desc_parts: list[str] = [] + if args.max_trans_norm is not None: + filter_desc_parts.append(f"max_trans_norm={args.max_trans_norm}") + if args.max_rot_norm is not None: + filter_desc_parts.append(f"max_rot_norm={args.max_rot_norm}") + if args.percentile_clip is not None: + filter_desc_parts.append(f"percentile_clip=P{args.percentile_clip:g}") + print(f"Outlier filter : {', '.join(filter_desc_parts) if filter_desc_parts else 'off'}") + print() + + rotation_fmt: RotationConvention = args.rotation_format # type: ignore[assignment] + pose_conv: PoseConvention = args.pose_convention # type: ignore[assignment] + + if args.dataset == "camera": + dataset = _create_camera_dataset( + split=args.split, + rotation_format=rotation_fmt, + pose_convention=pose_conv, + credential_path=args.credential_path, + wdinfo_names=args.camera_wdinfos, + ) + else: + dataset = _create_av_dataset( + split=args.split, + rotation_format=rotation_fmt, + pose_convention=pose_conv, + credential_path=args.credential_path, + av_root=args.av_root, + av_history_len=args.av_history_len, + av_future_len=args.av_future_len, + av_fps=args.av_fps, + ) + + print(f"Dataset length : {len(dataset)}") + print() + + results = compute_action_stats( + dataset, + max_samples=args.max_samples, + max_frames=args.max_frames, + max_trans_norm=args.max_trans_norm, + max_rot_norm=args.max_rot_norm, + percentile_clip=args.percentile_clip, + ) + + results["metadata"]["dataset"] = args.dataset + results["metadata"]["split"] = args.split + results["metadata"]["max_frames"] = args.max_frames + results["metadata"]["rotation_format"] = args.rotation_format + results["metadata"]["pose_convention"] = args.pose_convention + if args.dataset == "av": + results["metadata"]["av_root"] = args.av_root + results["metadata"]["av_history_len"] = args.av_history_len + results["metadata"]["av_future_len"] = args.av_future_len + results["metadata"]["av_fps"] = args.av_fps + else: + results["metadata"]["camera_wdinfos"] = args.camera_wdinfos or ["pretrained_clips_260307_100k"] + + output_path.parent.mkdir(parents=True, exist_ok=True) + with open(output_path, "w") as f: + json.dump(results, f, indent=2) + + print(f"\nResults saved to: {output_path}") + + meta = results["metadata"] + if meta["dropped_frames"] > 0: + pct = 100.0 * meta["dropped_frames"] / meta["total_frames"] if meta["total_frames"] else 0.0 + print( + f"Filtered frames: kept {meta['kept_frames']} / {meta['total_frames']} " + f"(dropped {meta['dropped_frames']}, {pct:.2f}%) — " + f"thresholds max_trans_norm={meta['max_trans_norm']}, max_rot_norm={meta['max_rot_norm']}" + ) + + def _print_block(label: str, raw_key: str, l2_key: str) -> None: + g = results[raw_key]["global"] + n = results[l2_key] + print(f"\nRaw {label} per-dim statistics — translation_scale=1.0, rotation_scale=1.0:") + print(f" Mean : {g['mean']}") + print(f" Median : {g['median']}") + print(f" Std : {g['std']}") + print(f" Min : {g['min']}") + print(f" Max : {g['max']}") + print(f" Flat mean : {g['flat_mean']:.6f} (pooled across all dims)") + print(f" Flat std : {g['flat_std']:.6f} (single global scalar, preserves geometry)") + print(f" Count : {g['count']} frames from {results['metadata']['num_samples']} samples") + print(f"\n{label.capitalize()} L2 norm:") + print(f" Median : {n['median']:.6f}") + print(f" Mean : {n['mean']:.6f}") + print(f" Std : {n['std']:.6f}") + print(f" Min : {n['min']:.6f}") + print(f" Max : {n['max']:.6f}") + for pct, val in n["percentiles"].items(): + print(f" P{pct:<5}: {val:.6f}") + + _print_block("translation (tx, ty, tz)", "raw_translation_stats", "translation_l2_norm") + _print_block( + f"rotation ({args.rotation_format}, {results['metadata']['rotation_dim']} dims)", + "raw_rotation_stats", + "rotation_l2_norm", + ) + + trans_flat_std = results["raw_translation_stats"]["global"]["flat_std"] + rot_flat_std = results["raw_rotation_stats"]["global"]["flat_std"] + print("\nSuggested uniform scales (matches MSE-loss magnitude per dim, preserves block geometry):") + print( + f" translation_scale = 1 / trans_flat_std = {1.0 / trans_flat_std:.6f}" + if trans_flat_std > 0 + else " translation_scale : undefined (trans_flat_std=0)" + ) + print( + f" rotation_scale = 1 / rot_flat_std = {1.0 / rot_flat_std:.6f}" + if rot_flat_std > 0 + else " rotation_scale : undefined (rot_flat_std=0)" + ) + if trans_flat_std > 0 and rot_flat_std > 0: + print( + f" (equivalently, with translation_scale=1: rotation_scale = trans_flat_std / rot_flat_std = {trans_flat_std / rot_flat_std:.6f})" + ) + + +if __name__ == "__main__": + main() diff --git a/cosmos_framework/data/vfm/action/cosmos3_action_lerobot.py b/cosmos_framework/data/vfm/action/cosmos3_action_lerobot.py new file mode 100644 index 0000000..7b53ec3 --- /dev/null +++ b/cosmos_framework/data/vfm/action/cosmos3_action_lerobot.py @@ -0,0 +1,1003 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Shared LeRobot adapter utilities for Action datasets. + +These helpers centralize common behavior across Action wrappers: +- deterministic train/val episode splitting +- valid per-episode index range construction +- a reusable BaseActionLeRobotDataset class with lazy init, video formatting, + and common result building +""" + +from __future__ import annotations + +import importlib +import logging as _logging +import math +import os as _os +import random +from bisect import bisect_right +from collections import OrderedDict, defaultdict +from collections.abc import Callable, Sequence +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path +from threading import Lock +from typing import Any, ClassVar, Literal + +import huggingface_hub.constants as _hf_const +import numpy as np +import torch +from lerobot.datasets.lerobot_dataset import LeRobotDataset, LeRobotDatasetMetadata +from torch.utils.data import Dataset + +_hf_offline_applied = False + + +def _ensure_hf_hub_offline() -> None: + """Force HF Hub into offline mode for local-only datasets (repo_id="local"). + + Sets the ``HF_HUB_OFFLINE`` env var (for any future imports in worker + processes), patches the already-imported constant, and suppresses the + expected "Returning existing local_dir" fallback warning. + + Safe to call multiple times; only applies once per process. + """ + global _hf_offline_applied + if _hf_offline_applied: + return + if "HF_HUB_OFFLINE" not in _os.environ: + _os.environ["HF_HUB_OFFLINE"] = "1" + if not _hf_const.HF_HUB_OFFLINE: + _hf_const.HF_HUB_OFFLINE = True + _logging.getLogger("huggingface_hub._snapshot_download").setLevel(_logging.ERROR) + _hf_offline_applied = True + + +from functools import cached_property + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.action_normalization import ( + load_action_stats, + normalize_action, +) + +# Re-export the action_spec DSL from this module so that subclass datasets +# only need a single import block (alongside ``BaseActionLeRobotDataset``). +from cosmos_framework.data.vfm.action.action_spec import ( # noqa: F401 (re-export) + ActionSpec, + DimType, + Gripper, + Joint, + Pos, + Reserved, + Rot, + build_action_spec, +) +from cosmos_framework.data.vfm.action.domain_utils import get_domain_id +from cosmos_framework.data.vfm.action.pose_utils import compute_idle_frames +from cosmos_framework.data.vfm.action.viewpoint_utils import Viewpoint +from cosmos_framework.data.vfm.action_scripts.memprofile import ( + deep_size as _deep_size, +) +from cosmos_framework.data.vfm.action_scripts.memprofile import ( + fmt_mb as _fmt_mb, +) +from cosmos_framework.data.vfm.action_scripts.memprofile import ( + log_worker_memory_breakdown, + rss_tracker, +) +from cosmos_framework.data.vfm.action_scripts.memprofile import ( + memprofile_enabled as _memprofile_enabled, +) + +# --------------------------------------------------------------------------- +# LRU-capped VideoDecoderCache +# --------------------------------------------------------------------------- +_LRU_VIDEO_CACHE_MAX_SIZE: int = 64 +_LRU_DATASET_MAX_LOADED: int = 32 +ActionNormalization = Literal["quantile", "quantile_rot", "meanstd", "minmax"] +_ACTION_NORMALIZATION_CHOICES: tuple[str, ...] = ("quantile", "quantile_rot", "meanstd", "minmax") + +_decoder_cache_patched = False + + +class _LRUVideoDecoderCache: + """Drop-in replacement for ``lerobot.datasets.video_utils.VideoDecoderCache`` + with LRU eviction. When the cache exceeds *max_size* entries the + least-recently-used decoder (and its file handle) is evicted. + """ + + def __init__(self, max_size: int = _LRU_VIDEO_CACHE_MAX_SIZE) -> None: + self._max_size = max_size + self._cache: OrderedDict[str, tuple[Any, Any]] = OrderedDict() + self._lock = Lock() + self._hits = 0 + self._misses = 0 + self._evictions = 0 + + def get_decoder(self, video_path: str) -> Any: + if importlib.util.find_spec("torchcodec"): # type: ignore[attr-defined] + from torchcodec.decoders import VideoDecoder + else: + raise ImportError("torchcodec is required but not available.") + + import fsspec + + video_path = str(video_path) + + with self._lock: + if video_path in self._cache: + self._cache.move_to_end(video_path) + self._hits += 1 + return self._cache[video_path][0] + + self._misses += 1 + file_handle = fsspec.open(video_path).__enter__() + decoder = VideoDecoder(file_handle, seek_mode="approximate") # type: ignore[arg-type] + self._cache[video_path] = (decoder, file_handle) + + evicted = 0 + while len(self._cache) > self._max_size: + _, (_, old_fh) = self._cache.popitem(last=False) + try: + old_fh.close() + except Exception: + pass + evicted += 1 + self._evictions += evicted + + if evicted and self._evictions % 50 <= evicted: + log.debug( + f"[VideoDecoderCache pid={_os.getpid()}] " + f"evicted={self._evictions} total, size={len(self._cache)}/{self._max_size}, " + f"hits={self._hits}, misses={self._misses}, " + f"hit_rate={100 * self._hits / max(1, self._hits + self._misses):.1f}%" + ) + + return decoder + + def clear(self) -> None: + with self._lock: + for _, file_handle in self._cache.values(): + try: + file_handle.close() + except Exception: + pass + self._cache.clear() + + def size(self) -> int: + with self._lock: + return len(self._cache) + + +def _patch_decoder_cache(max_size: int = _LRU_VIDEO_CACHE_MAX_SIZE) -> None: + """Replace the module-level ``_default_decoder_cache`` in LeRobot with an + LRU-capped version to prevent unbounded memory growth in workers.""" + global _decoder_cache_patched + if _decoder_cache_patched: + return + + import lerobot.datasets.video_utils as _vu + + lru_cache = _LRUVideoDecoderCache(max_size=max_size) + _vu._default_decoder_cache = lru_cache + _decoder_cache_patched = True + log.debug(f"Patched LeRobot VideoDecoderCache with LRU max_size={max_size}") + + +def _parallel_map( + fn: Callable[[Any], Any], + items: list[Any], + *, + max_workers: int, + label: str, +) -> list[Any]: + """Thread-pool ``map`` — returns results in input order. + + Intended for IO-bound prefetch (``LeRobotDatasetMetadata`` loads, + parquet column reads). Preserves item-order so callers can ``zip`` + with their ``indices`` / ``roots`` list. Skips the thread pool + entirely when there is 0 or 1 task — avoids per-worker + ``ThreadPoolExecutor`` setup cost and log spam under + ``shard_across_workers=True`` where each worker typically gets + only 1-2 shards. + """ + if not items: + return [] + if len(items) == 1 or max_workers <= 1: + return [fn(items[0])] if len(items) == 1 else [fn(x) for x in items] + log.info(f"{label}: {len(items)} tasks (workers={max_workers})") + with ThreadPoolExecutor(max_workers=max_workers) as ex: + return list(ex.map(fn, items)) + + +def split_episode_ids(total_episodes: int, seed: int, val_ratio: float, split: str) -> list[int]: + """Create deterministic random episode ids for train/val/full splits.""" + num_val = int(round(total_episodes * val_ratio)) + g = torch.Generator().manual_seed(seed) + episode_ids = torch.randperm(total_episodes, generator=g).tolist() + + if split == "train": + return episode_ids[num_val:] + if split == "val": + return episode_ids[:num_val] + return episode_ids + + +def build_episode_spans( + episodes: Any, + episode_ids: Sequence[int], + chunk_length: int, + sample_stride: int = 1, +) -> tuple[list[tuple[int, int, int]], int, int]: + """Build valid episode spans for LeRobot frame queries. + + Returns: + - episode spans as ``(episode_id, sample_start, valid_len)`` + - total valid sample count across selected episodes + - total raw frame count across selected episodes + """ + assert sample_stride >= 1, f"sample_stride must be >= 1, got {sample_stride}" + + dataset_from_index = list(episodes["dataset_from_index"]) + dataset_to_index = list(episodes["dataset_to_index"]) + length = list(episodes["length"]) + + spans: list[tuple[int, int, int]] = [] + valid_count = 0 + sample_count = 0 + for episode_id in episode_ids: + start = dataset_from_index[episode_id] + stop = dataset_to_index[episode_id] + raw_valid_len = stop - start - chunk_length + if raw_valid_len > 0: + valid_len = (raw_valid_len + sample_stride - 1) // sample_stride + spans.append((episode_id, start, valid_len)) + valid_count += valid_len + sample_count += int(length[episode_id]) + + return spans, valid_count, sample_count + + +def _normalize_split(split: str) -> str: + """Normalize split name to one of ``'train'``, ``'val'``, ``'full'``.""" + s = split.lower().strip() + if s in {"val", "valid", "validation", "eval", "test"}: + return "val" + if s in {"train", "full"}: + return s + raise ValueError(f"Unsupported {split=}. Use train/val/full.") + + +class BaseActionLeRobotDataset(Dataset): + """Reusable base class for Action LeRobot-backed map-style datasets. + + Subclasses typically: + 1) call ``_register_source`` to register one or more LeRobot sources + 2) implement ``__getitem__`` for dataset-specific sample parsing + 3) call ``_build_result`` to assemble the return dict + """ + + # Applied as: R_opencv = R_native @ _to_opencv + # Subclasses override in __init__; default is identity (no correction). + + # Bundled normalization stats directory. Stats are committed at + # ``<_NORMALIZERS_DIR>/__.json`` (flat + # layout matching the existing UMI files) and produced by + # ``projects/cosmos3/vfm/datasets/action/compute_action_stats.py``. + # Subclasses that need a different filename scheme can override + # :meth:`_normalizer_filename`. + _NORMALIZERS_DIR: ClassVar[Path] = Path(__file__).parent / "normalizers" + + def __init__( + self, + *, + fps: float, + chunk_length: int, + split_seed: int, + split_val_ratio: float, + split: str, + mode: str, + embodiment_type: str, + viewpoint: Viewpoint, + pose_convention: str | None = None, + rotation_format: str | None = None, + action_normalization: ActionNormalization | None = None, + tolerance_s: float = 1e-4, + max_loaded_datasets: int = _LRU_DATASET_MAX_LOADED, + skip_video_loading: bool = False, + sample_stride: int = 1, + enable_fast_init: bool = False, + fast_init_max_workers: int = 64, + ) -> None: + super().__init__() + _ensure_hf_hub_offline() + _patch_decoder_cache() + self._memprofile = _memprofile_enabled() + + assert sample_stride >= 1, f"sample_stride must be >= 1, got {sample_stride}" + assert fast_init_max_workers >= 1, f"fast_init_max_workers must be >= 1, got {fast_init_max_workers}" + assert action_normalization is None or action_normalization in _ACTION_NORMALIZATION_CHOICES, ( + f"action_normalization must be None or one of {_ACTION_NORMALIZATION_CHOICES}, got {action_normalization!r}" + ) + + with rss_tracker(f"{self.__class__.__name__}.__init__", enabled=self._memprofile): + self._fps = fps + self._dt = 1.0 / fps + self._chunk_length = chunk_length + self._split_seed = split_seed + self._split_val_ratio = split_val_ratio + self._split = _normalize_split(split) + self._mode = mode + self._embodiment_type = embodiment_type + self._viewpoint: Viewpoint = viewpoint + self._pose_convention = pose_convention + self._rotation_format = rotation_format + self._action_normalization = action_normalization + # Lazy-loaded stats cache, populated on first call to + # :meth:`_normalize_action`. Per-process (workers get their own). + self._norm_stats: dict[str, torch.Tensor] | None = None + self._tolerance_s = tolerance_s + self._max_loaded_datasets = max_loaded_datasets + self._skip_video_loading = skip_video_loading + self._sample_stride = sample_stride + self._enable_fast_init = enable_fast_init + self._fast_init_max_workers = fast_init_max_workers + self._delta_timestamps: dict[str, list[float]] = {} + self._to_opencv: np.ndarray | dict[str, np.ndarray] = np.eye(3, dtype=np.float32) + + if pose_convention is None: + log.warning( + f"{self.__class__.__name__}: pose_convention is not set. " + "Consider specifying 'backward_framewise' or 'backward_anchored'." + ) + + self._datasets: list[LeRobotDataset | None] = [] + self._dataset_build_args: list[dict[str, Any] | None] = [] + self._loaded_lru: OrderedDict[int, None] = OrderedDict() + + # -- Flat index structures (populated by _append_index_records) -- + # Together these two lists form a searchable map from a flat + # global index to (dataset, row, episode, frame). One entry per + # episode span across *all* registered sources. + # + # _episode_records[i] = (ds_idx, sample_start, valid_len, episode_id) + # ds_idx – which source dataset (index into _datasets) + # sample_start – first row of this span in that dataset's table + # valid_len – number of usable frames in this span + # episode_id – the episode this span belongs to + # + # _episode_cum_ends[i] = running total of valid_len through span i + # Used for O(log N) lookup via bisect_right in _resolve_index. + self._episode_records: list[tuple[int, int, int, int]] = [] + self._episode_cum_ends: list[int] = [] + self._num_valid_indices = 0 + self._domain_id = get_domain_id(self._embodiment_type) + + # Deferred-init shard roots — a list of root paths. + # Subclasses populate this in __init__; _register_sources() + # reads _delta_timestamps and _tolerance_s from self (both + # initialised above, with _delta_timestamps overridden by + # each subclass). + # ActionUnifiedIterableDataset.assign_worker uses len() for + # round-robin shard distribution and _register_sources(indices) + # for deferred loading. When empty, shard distribution is + # skipped (every worker iterates the full dataset). + self._all_shard_roots: list[str] = [] + + # -- public properties --------------------------------------------------- + + @property + def fps(self) -> float: + return self._fps + + @property + def chunk_length(self) -> int: + return self._chunk_length + + @property + def split(self) -> str: + return self._split + + @property + def mode(self) -> str: + return self._mode + + @mode.setter + def mode(self, value: str) -> None: + self._mode = value + + @property + def domain_id(self) -> int: + return self._domain_id + + # -- source registration ------------------------------------------------- + + def _register_source( + self, + *, + delta_timestamps: dict[str, list[float]], + tolerance_s: float, + root: str | None = None, + repo_id: str = "local", + force_cache_sync: bool = False, + download_videos: bool = False, + video_backend: str | None = None, + revision: str | None = None, + dataset_label: str | None = None, + prefetched_meta: LeRobotDatasetMetadata | None = None, + ) -> LeRobotDatasetMetadata: + """Register a LeRobot dataset source lazily (metadata-only at init). + + ``prefetched_meta`` lets subclasses load metadata in a thread pool + (``LeRobotDatasetMetadata`` reads are pure I/O — ``info.json`` + + ``episodes.parquet`` + ``tasks.parquet``) and then hand the ready + object to the serial append-path below, which still manages the + order-sensitive shared state (``_datasets`` / ``_dataset_build_args`` + / ``_episode_records`` / ``_episode_cum_ends``). When ``None`` the + caller gets the original single-threaded behavior. + """ + label_str = f" [{dataset_label}]" if dataset_label else "" + cls = self.__class__.__name__ + # "local" is not a valid PEP 440 version, so LeRobot's + # is_valid_version() check skips the get_safe_version() HF API call. + if repo_id == "local" and revision is None: + revision = "local" + + with rss_tracker(f"{cls}{label_str} — metadata load", enabled=self._memprofile): + if prefetched_meta is not None: + meta = prefetched_meta + else: + meta = LeRobotDatasetMetadata( + repo_id=repo_id, + root=root, + revision=revision, + force_cache_sync=force_cache_sync, + ) + ds_idx = len(self._datasets) + self._datasets.append(None) + self._dataset_build_args.append( + { + "repo_id": repo_id, + "root": root, + "delta_timestamps": delta_timestamps, + "tolerance_s": tolerance_s, + "force_cache_sync": force_cache_sync, + "download_videos": download_videos, + "video_backend": video_backend, + "revision": revision, + } + ) + + with rss_tracker( + f"{cls}{label_str} — index records", + enabled=self._memprofile, + extras_fn=lambda: [ + f"episode_records so far: {len(self._episode_records)} entries, " + f"~{_fmt_mb(_deep_size(self._episode_records) / (1024 * 1024))}", + f"episode_cum_ends so far: {len(self._episode_cum_ends)} entries, " + f"~{_fmt_mb(_deep_size(self._episode_cum_ends) / (1024 * 1024))}", + ], + ): + self._append_index_records(meta=meta, ds_idx=ds_idx, dataset_label=dataset_label) + + return meta + + def _append_index_records( + self, + *, + meta: LeRobotDatasetMetadata, + ds_idx: int, + dataset_label: str | None = None, + ) -> None: + """Populate episode split / index records from dataset metadata.""" + episode_ids = split_episode_ids( + total_episodes=meta.total_episodes, + seed=self._split_seed, + val_ratio=self._split_val_ratio, + split=self._split, + ) + + if hasattr(self, "_filter_valid_episodes"): + episode_ids = self._filter_valid_episodes(meta, episode_ids) + episode_spans, valid_count, sample_count = build_episode_spans( + episodes=meta.episodes, + episode_ids=episode_ids, + chunk_length=self._chunk_length, + sample_stride=self._sample_stride, + ) + + class_name = self.__class__.__name__ + label = f" [{dataset_label}]" if dataset_label else "" + log.info(f"{class_name}{label}: split={self._split}, num episodes={len(episode_ids)}") + if sample_count > 0: + log.info( + f"{class_name}{label}: kept {valid_count} / {sample_count} " + f"({100 * valid_count / sample_count:.2f} %) samples" + ) + + for episode_id, sample_start, valid_len in episode_spans: + self._episode_records.append((ds_idx, sample_start, valid_len, episode_id)) + self._num_valid_indices += valid_len + self._episode_cum_ends.append(self._num_valid_indices) + + # -- deferred shard registration ----------------------------------------- + + def _register_sources(self, indices: list[int] | None = None) -> None: + """Register a subset (or all) of the shard roots in ``_all_shard_roots``. + + Called by ``ActionUnifiedIterableDataset.assign_worker`` during training, + or explicitly by eval/visualization scripts after construction. + + ``_all_shard_roots`` is a list of root paths. Per-shard args that are + shared across all shards (``delta_timestamps``, ``tolerance_s``) are + taken from ``self``. Subclasses may override this for extra per-shard + setup (e.g. loading instruction segments). + + When ``enable_fast_init=True``, ``LeRobotDatasetMetadata`` (a pure-IO + read of ``info.json`` + ``episodes.parquet`` + ``tasks.parquet``) is + prefetched in a thread pool and handed to the order-sensitive + serial register loop via ``prefetched_meta=``. Shard count scales + the speedup; for single-shard datasets the two paths are + equivalent. + + Args: + indices: Which entries of ``_all_shard_roots`` to register. + ``None`` means all. + """ + if indices is None: + indices = list(range(len(self._all_shard_roots))) + if not indices: + return + + roots = [self._all_shard_roots[i] for i in indices] + + if self._enable_fast_init: + # ``_ensure_hf_hub_offline`` already ran in ``__init__`` and is + # idempotent; no need to re-invoke here. + workers = max(1, min(self._fast_init_max_workers, len(roots))) + metas: list[LeRobotDatasetMetadata | None] = _parallel_map( + lambda root: LeRobotDatasetMetadata(repo_id="local", root=root, revision="local"), + roots, + max_workers=workers, + label=f"{type(self).__name__}: LeRobotDatasetMetadata prefetch", + ) + else: + metas = [None] * len(roots) + + for root, meta in zip(roots, metas): + label = root.rsplit("/", 1)[-1] if "/" in root else root + self._register_source( + root=root, + delta_timestamps=self._delta_timestamps, + tolerance_s=self._tolerance_s, + dataset_label=label, + prefetched_meta=meta, + ) + + # -- lazy dataset access ------------------------------------------------- + + def _get_dataset(self, ds_idx: int) -> LeRobotDataset: + """Get or lazily construct the LeRobot dataset for the given source index. + + Loaded datasets are tracked with LRU ordering. When the number of + loaded datasets exceeds ``_max_loaded_datasets`` the least-recently-used + dataset is evicted (set back to ``None``) so the GC can reclaim it. + """ + ds = self._datasets[ds_idx] + if ds is not None: + self._loaded_lru.move_to_end(ds_idx) + return ds + + _ensure_hf_hub_offline() + + build_args = self._dataset_build_args[ds_idx] + if build_args is None: + raise RuntimeError(f"Missing dataset build args for dataset index {ds_idx}") + + # Evict least-recently-used datasets before loading a new one. + while len(self._loaded_lru) >= self._max_loaded_datasets: + evict_idx, _ = self._loaded_lru.popitem(last=False) + self._datasets[evict_idx] = None + + with rss_tracker( + f"[WORKER {_os.getpid()}] Lazy-loaded ds[{ds_idx}]", + enabled=self._memprofile, + extras_fn=lambda: [f"total loaded={len(self._loaded_lru)}/{len(self._datasets)}"], + ): + delta_ts = build_args["delta_timestamps"] + if self._skip_video_loading: + # Covers both LeRobot v2 (``observation.images.``) and + # v3 (``observation.image.``) video-column conventions. + delta_ts = {k: v for k, v in delta_ts.items() if not k.startswith("observation.image")} + + log.info(f"Loading shard root={build_args['root']}") + ds = LeRobotDataset( + repo_id=build_args["repo_id"], + root=build_args["root"], + delta_timestamps=delta_ts, + tolerance_s=build_args["tolerance_s"], + force_cache_sync=build_args["force_cache_sync"], + download_videos=build_args["download_videos"], + video_backend=build_args["video_backend"], + revision=build_args["revision"], + episodes=None, + ) + if self._skip_video_loading: + ds.meta.info["features"] = { + k: v for k, v in ds.meta.info["features"].items() if v.get("dtype") != "video" + } + self._datasets[ds_idx] = ds + self._loaded_lru[ds_idx] = None + + return ds + + # -- index resolution ---------------------------------------------------- + + def _resolve_index(self, idx: int) -> tuple[int, int, int, int]: + """Map a flat global index to the source dataset, row, episode, and frame. + + Multiple datasets are concatenated into a single virtual sequence. + Each episode contributes a contiguous *span* of valid frames, and + ``_episode_cum_ends[i]`` stores the running total of valid frames + through the *i*-th span. For example, with two episodes of lengths + 5 and 3 the cum-ends are ``[5, 8]``, so global index 6 falls in the + second span at offset 1. + + The lookup is O(log N) via :func:`bisect_right`. + + Returns: + dataset_idx: Which source dataset this sample belongs to. + row_idx: Row index *within* that dataset's LeRobot table. + episode_id: The episode ID for this sample. + frame_offset: Frame offset from the start of the episode span + (0-based). + + Pure index math -- no I/O or dataset access. Higher-level helpers + like :meth:`_fetch_sample` build on this. + """ + # Support negative indexing (e.g. -1 → last sample). + if idx < 0: + idx += self._num_valid_indices + if idx < 0 or idx >= self._num_valid_indices: + raise IndexError(f"{self.__class__.__name__} index {idx} out of range for size {self._num_valid_indices}") + + # _episode_cum_ends is a monotonically increasing list where entry i + # holds the cumulative number of valid frames up to and including the + # i-th episode span. bisect_right finds the first span whose + # cumulative end is strictly greater than idx, i.e. the span that + # contains idx. + # + # Example: cum_ends = [5, 8, 20] + # idx=0 -> span_idx=0 (first span, frames 0..4) + # idx=4 -> span_idx=0 + # idx=5 -> span_idx=1 (second span, frames 5..7) + # idx=8 -> span_idx=2 (third span, frames 8..19) + span_idx = bisect_right(self._episode_cum_ends, idx) + + # The global index where this span begins is the previous span's + # cumulative end (or 0 for the very first span). The frame_offset + # is how far idx is into this particular episode. + span_start = 0 if span_idx == 0 else self._episode_cum_ends[span_idx - 1] + frame_offset = idx - span_start + + # _episode_records[span_idx] stores (dataset_idx, row_start, valid_len, + # episode_id). row_start is the absolute row in the LeRobot table + # where this episode begins. With sample_stride=k, consecutive + # valid indices map to rows k apart inside the episode, so the + # effective row is row_start + frame_offset * sample_stride. + dataset_idx, row_start, _, episode_id = self._episode_records[span_idx] + row_idx = row_start + frame_offset * self._sample_stride + return dataset_idx, row_idx, episode_id, frame_offset + + def _choose_mode(self) -> str: + """Resolve the active mode for one sample request.""" + if self._mode == "joint": + return random.choice(("forward_dynamics", "inverse_dynamics", "policy")) + return self._mode + + def _fetch_sample(self, idx: int) -> tuple[str, int, int, dict[str, Any]]: + """Resolve index, pick a mode, and load the sample from the dataset. + + Returns ``(mode, dataset_idx, row_idx, sample_dict)``. + """ + mode = self._choose_mode() + dataset_idx, row_idx, _, _ = self._resolve_index(idx) + + self._getitem_count = getattr(self, "_getitem_count", 0) + 1 + profile = self._memprofile and self._getitem_count % 50 == 1 + + with rss_tracker( + f"[WORKER {_os.getpid()}] __getitem__ transient (dataset_idx={dataset_idx})", + enabled=profile, + after_fn=lambda: log_worker_memory_breakdown(self), + ): + sample = self._get_dataset(dataset_idx)[row_idx] + + if self._skip_video_loading: + sample = defaultdict(lambda: None, sample) + + return mode, dataset_idx, row_idx, sample + + # -- action normalization ------------------------------------------------ + + def _normalizer_filename(self) -> str: + """Bundled stats filename for this dataset instance. + + Default convention (matches ``compute_action_stats.py`` output): + ``[_][_].json``. + + Pose/rotation suffixes are appended only when the instance actually + has them (SE(3) pose datasets like Bridge / DROID). Joint-space + datasets — where both are ``None`` — resolve to just + ``.json``. + + Subclasses may override when the bundled filename uses a different + scheme (e.g. UMI's ``uva_umi_single_task_normalizer.json``). + """ + if not self._embodiment_type: + raise RuntimeError( + f"{self.__class__.__name__}: embodiment_type is not set; cannot resolve normalizer filename." + ) + parts = [self._embodiment_type] + if self._pose_convention: + parts.append(self._pose_convention) + if self._rotation_format: + parts.append(self._rotation_format) + return "_".join(parts) + ".json" + + def _normalizer_path(self) -> Path: + """Full path to the bundled stats JSON for this dataset.""" + return self._NORMALIZERS_DIR / self._normalizer_filename() + + def _load_norm_stats(self) -> dict[str, torch.Tensor]: + """Lazy-load action normalization stats (once per worker process). + + Raises :class:`FileNotFoundError` if the stats file is missing. This + is intentional — silently falling back to identity normalization when + the user asked for ``quantile`` / ``quantile_rot`` / ``meanstd`` / + ``minmax`` would be a training bug. + """ + if self._norm_stats is not None: + return self._norm_stats + stats_key = "global_raw" if self._action_normalization == "quantile_rot" else "global" + raw = load_action_stats(str(self._normalizer_path()), stats_key=stats_key) + self._norm_stats = {} + for key, value in raw.items(): + self._norm_stats[key] = torch.from_numpy(value).float() # [D] + return self._norm_stats + + def _normalize_action(self, action: torch.Tensor) -> torch.Tensor: + """Apply the configured normalization, or return the raw action. + + - ``action_normalization=None`` → pass-through (used by viewer / debug) + - ``"quantile"`` → ``2·(x − q01) / (q99 − q01) − 1`` clamped to [-1, 1] + - ``"quantile_rot"`` → same as ``"quantile"``, but using ``global_raw`` + stats so rotation dimensions are normalized too. + - ``"meanstd"`` → ``(x − mean) / std`` + - ``"minmax"`` → ``2·(x − min) / (max − min) − 1`` clamped to [-1, 1] + """ + if self._action_normalization is None: + return action + method = "quantile" if self._action_normalization == "quantile_rot" else self._action_normalization + normalized_action = normalize_action( + action, + method, + self._load_norm_stats(), + ) # [T,D] + return normalized_action + + # -- video formatting ---------------------------------------------------- + + def _convert_video(self, video_tchw: torch.Tensor | None) -> torch.Tensor | None: + """Convert LeRobot ``(T,C,H,W)`` float video to Action ``(C,T,H,W)`` uint8. + + Args: + video_tchw: Raw floating-point video tensor in ``[0, 1]`` with + LeRobot layout, or ``None``. # [T,C,H,W] | None + + Returns: + Action-formatted video tensor, or ``None``. # [C,T,H,W] | None + """ + if self._skip_video_loading or video_tchw is None: + return None + if video_tchw.ndim != 4: + raise ValueError( + f"{self.__class__.__name__}._convert_video expected video with shape [T,C,H,W], " + f"got ndim={video_tchw.ndim}" + ) + if not torch.is_floating_point(video_tchw): + raise TypeError( + f"{self.__class__.__name__}._convert_video expected floating-point video in [0, 1], " + f"got dtype={video_tchw.dtype}" + ) + video_min = video_tchw.amin() # [] + video_max = video_tchw.amax() # [] + if video_min.item() < 0.0 or video_max.item() > 1.0: + raise ValueError( + f"{self.__class__.__name__}._convert_video expected floating-point video in [0, 1], " + f"got range=[{video_min.item():.6f}, {video_max.item():.6f}]" + ) + formatted_video = (video_tchw * 255.0).clamp(0.0, 255.0).to(torch.uint8).permute(1, 0, 2, 3) # [C,T,H,W] + return formatted_video + + # -- result building ----------------------------------------------------- + + def _build_action_spec(self) -> ActionSpec | None: + """Subclass override: declare this dataset's action layout. + + Called once per instance — the result is cached by ``self.action_spec``. + Return ``None`` to skip spec-driven idle detection; in that case + ``_compute_idle_frames`` will log a one-time warning and return + ``None`` for every sample. + """ + return None + + @cached_property + def action_spec(self) -> ActionSpec | None: + """Cached :class:`ActionSpec` from ``_build_action_spec``. + + Returns ``None`` when the subclass did not declare one; idle detection + is then skipped (with a one-time warning) until the subclass overrides + ``_build_action_spec``. + """ + return self._build_action_spec() + + @cached_property + def action_names(self) -> list[str] | None: + spec = self.action_spec + return spec.names if spec is not None else None + + # Idle-detection thresholds. Defined as **velocities** (per second) so the + # same numeric value means the same physical motion across datasets with + # different sampling rates; converted to per-frame at call time using + # ``self._fps`` via :meth:`_resolve_idle_thresholds`. + # + # Defaults: + # - ``idle_eps_t_per_sec`` = 5 mm/s (≈ 1 mm/frame at 5 Hz) + # - ``idle_eps_r_per_sec`` = 1.5°/s (geodesic, rotation-format aware) + # - ``idle_eps_g`` = 1e-2 unit gripper Δ (no fps) + # - ``idle_joint_threshold_per_sec`` = 5e-3 rad/s + # - ``idle_min_streak`` = 3 require ≥ 3 consecutive + # + # Subclasses can either override the ``*_per_sec`` attributes (preferred — + # keeps the velocity semantics) or set the corresponding ``idle_eps_*`` / + # ``idle_joint_threshold`` attribute to a non-``None`` value to bypass the + # per-fps conversion entirely (raw per-frame override). + idle_eps_t_per_sec: float = 5e-3 + idle_eps_r_per_sec: float = math.radians(1.5) + idle_eps_g: float = 1e-2 + idle_joint_threshold_per_sec: float = 5e-3 + idle_min_streak: int = 3 + + # Optional per-frame overrides. ``None`` (default) → use the ``*_per_sec`` + # attribute / fps conversion above. + idle_eps_t: float | None = None + idle_eps_r: float | None = None + idle_joint_threshold: float | None = None + + def _resolve_idle_thresholds(self) -> tuple[float, float, float, float]: + """Resolve per-frame idle thresholds for this dataset instance. + + Returns ``(eps_t, eps_r, eps_g, joint_threshold)`` in raw per-frame + units. Honours direct per-frame overrides if the subclass sets the + non-``_per_sec`` attribute; otherwise scales the ``_per_sec`` values + by ``self._fps``. + """ + fps = float(self._fps) if self._fps else 1.0 + eps_t = self.idle_eps_t if self.idle_eps_t is not None else self.idle_eps_t_per_sec / fps + eps_r = self.idle_eps_r if self.idle_eps_r is not None else self.idle_eps_r_per_sec / fps + joint_thr = ( + self.idle_joint_threshold + if self.idle_joint_threshold is not None + else self.idle_joint_threshold_per_sec / fps + ) + return float(eps_t), float(eps_r), float(self.idle_eps_g), float(joint_thr) + + def _compute_idle_frames(self, raw_action: torch.Tensor) -> torch.Tensor | None: + """Count idle frames in the *raw* (un-normalized) action chunk. + + Requires ``self.action_spec`` to be declared via ``_build_action_spec``. + Returns ``None`` when: + - ``pose_convention`` is not ``"backward_framewise"`` (TODO: extend), + - the subclass has not declared an ``ActionSpec`` (logs a one-time warning), + - the action layout does not match the declared spec. + + Detection thresholds come from the ``idle_eps_*`` class attributes + (overridable per dataset). Subclasses can also override this method + outright, or pass an explicit ``idle_frames`` integer via + ``**extras`` to :meth:`_build_result`. + """ + + # conventions (anchored / absolute) need different idle semantics. + if self._pose_convention != "backward_framewise": + if not getattr(self, "_warned_pose_convention", False): + log.warning( + f"Dataset {self.__class__.__name__}: pose_convention=" + f"{self._pose_convention!r} is not 'backward_framewise'; " + "skipping idle-frames detection. Centralize the dataset " + "to backward_framewise to enable IdleFrames captioning." + ) + self._warned_pose_convention = True + return None + + spec = self.action_spec + if spec is None: + if not getattr(self, "_warned_no_action_spec", False): + log.warning( + f"Dataset {self.__class__.__name__} has no action spec defined; " + "skipping idle-frames detection. Override _build_action_spec() to enable it." + ) + self._warned_no_action_spec = True + return None + + eps_t, eps_r, eps_g, joint_thr = self._resolve_idle_thresholds() + try: + n = compute_idle_frames( + raw_action, + spec, + eps_t=eps_t, + eps_r=eps_r, + eps_g=eps_g, + joint_threshold=joint_thr, + min_streak=self.idle_min_streak, + ) + except (ValueError, TypeError) as e: + if not getattr(self, "_warned_action_layout", False): + log.warning( + f"Dataset {self.__class__.__name__}: action layout does " + f"not match the declared ActionSpec " + f"(action_dim={int(raw_action.shape[-1])}, " + f"spec.dim={spec.dim}); skipping idle-frames detection. " + f"Underlying error: {e}" + ) + self._warned_action_layout = True + return None + return torch.tensor(n, dtype=torch.long) + + def _build_result( + self, + *, + mode: str, + video: torch.Tensor | None, + action: torch.Tensor, + ai_caption: str, + **extras: Any, + ) -> dict[str, Any]: + """Assemble the common return dict for ``__getitem__``. + + ``video`` is expected in raw LeRobot layout before final formatting. + Subclasses may pass extra keys (e.g. ``initial_pose``) via ``**extras``. + ``idle_frames`` is auto-computed from the raw (un-normalized) ``action`` + whenever the dataset's pose/rotation conventions allow it; subclasses + can override by passing ``idle_frames`` (int or scalar tensor) via + ``**extras``. + """ + # Compute idle_frames from the raw action before normalization, unless + # the subclass has provided one explicitly via ``**extras``. + if "idle_frames" not in extras: + idle_frames = self._compute_idle_frames(action) + if idle_frames is not None: + extras = {"idle_frames": idle_frames, **extras} + + normalized_action = self._normalize_action(action) # [T,D] + if self._skip_video_loading: + result: dict[str, Any] = {"action": normalized_action} + if "idle_frames" in extras: + result["idle_frames"] = extras["idle_frames"] + return result + formatted_video = self._convert_video(video) # [C,T,H,W] | None + return { + "ai_caption": ai_caption, + "video": formatted_video, + "action": normalized_action, + "conditioning_fps": torch.tensor(self._fps, dtype=torch.long), + "mode": mode, + "domain_id": torch.tensor(self._domain_id, dtype=torch.long), + "viewpoint": self._viewpoint, + **extras, + } + + def __len__(self) -> int: + return self._num_valid_indices diff --git a/cosmos_framework/data/vfm/action/dataloaders.py b/cosmos_framework/data/vfm/action/dataloaders.py new file mode 100644 index 0000000..f3a7404 --- /dev/null +++ b/cosmos_framework/data/vfm/action/dataloaders.py @@ -0,0 +1,311 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from __future__ import annotations + +import functools +import random +import time +from typing import Any, Callable, Iterator + +import numpy as np +import torch +import torch.utils.data + +from cosmos_framework.utils import distributed, log +from cosmos_framework.data.vfm.action.unified_dataset import ActionUnifiedIterableDataset +from cosmos_framework.data.vfm.joint_dataloader import custom_collate_fn + +# DataLoader kwargs that are safe to patch on a per-family basis from +# ``per_family_overrides``. Anything outside this set would either change +# the dataset identity, the seeding contract, or break PyTorch's +# expectations. +_ALLOWED_PER_FAMILY_OVERRIDE_KEYS: frozenset[str] = frozenset( + {"batch_size", "num_workers", "prefetch_factor", "persistent_workers", "pin_memory"} +) + + +def _action_worker_init_fn( + worker_id: int, seed: int = 42, use_deterministic_seed: bool = True, rank: int = 0, world_size: int = 1 +) -> None: + # DataLoader workers use spawn, so parent-process monkey patches do not + # carry over. Apply the LeRobot decoder-cache cap inside each worker. + from cosmos_framework.data.vfm.action.cosmos3_action_lerobot import _patch_decoder_cache + + _patch_decoder_cache() + + if use_deterministic_seed: + worker_seed = seed + rank * 9999 + worker_id + else: + worker_seed = int(time.time() * 1000) % (2**32) + rank * 9999 + worker_id + random.seed(worker_seed) + np.random.seed(worker_seed % (2**32)) + torch.manual_seed(worker_seed) + + info = torch.utils.data.get_worker_info() + assert info is not None + dataset = info.dataset + if isinstance(dataset, ActionUnifiedIterableDataset): + dataset.assign_worker(worker_id, info.num_workers, rank, world_size) + + +def create_action_worker_init_fn(seed: int = 42, use_deterministic_seed: bool = True) -> Callable[[int], None]: + """Create a worker_init_fn for Action training with ``ActionUnifiedIterableDataset``. + + Seeds RNGs first, then calls ``dataset.assign_worker()`` to set up + rank-level dataset assignment and worker-level shard distribution. + + Passed to ``DataLoader`` (or ``InfiniteDataLoader``) as the + ``worker_init_fn`` parameter. Only called when ``num_workers > 0``. + + Args: + seed: Base seed for deterministic worker seeding. Ignored when + ``use_deterministic_seed=False`` (time-based seed used instead). + use_deterministic_seed: If True, use the provided seed for reproducible + RNG initialization. If False, derive a time-based seed so that + each resume sees different data. This is preferred for large-scale + runs that resume frequently, and when ``in_order=False`` already + makes iteration order non-deterministic. + + Returns: + A ``worker_init_fn`` suitable for ``torch.utils.data.DataLoader``. + """ + try: + rank = distributed.get_rank() + world_size = distributed.get_world_size() + except RuntimeError: + rank = 0 + world_size = 1 + + return functools.partial( + _action_worker_init_fn, + seed=seed, + use_deterministic_seed=use_deterministic_seed, + rank=rank, + world_size=world_size, + ) + + +def _apply_per_family_overrides( + args: tuple, + kwargs: dict, + per_family_overrides: dict[str, dict[str, Any]], +) -> dict: + """Patch DataLoader kwargs for this rank based on its action family. + + Each global rank serves exactly one action family under Hare-Niemeyer + allocation (see :meth:`ActionUnifiedIterableDataset.assign_worker`). + This lets us tune ``batch_size`` / ``prefetch_factor`` / ``num_workers`` + / ``persistent_workers`` / ``pin_memory`` per family, even though they + share a single top-level ``action_data`` dataloader entry. + + Heavy families (``camera_720``, ``hand_pose``, ``agibotworld_beta``) + can shrink batch / prefetch to control host RSS, while small-token + high-IO families (``droid``, ``fractal``, ``bridge``) can expand the + in-flight buffer to absorb shard / page-cache stalls. + + No-ops (returns ``kwargs`` unchanged with a warning) when: + + * the dataset is not an :class:`ActionUnifiedIterableDataset` + (overrides are routed by family name, which only that class exposes); + * ``shard_across_workers=False`` — every worker iterates all families + via weighted random selection, so per-family override would be + meaningless; + * the resolved family has no entry in ``per_family_overrides``. + + Args: + args: Positional ``__init__`` args of the surrounding DataLoader. + ``args[0]`` is ``dataset`` when the caller used positional form. + kwargs: Keyword args of the surrounding DataLoader; mutated via + ``.update()`` with the resolved overrides. + per_family_overrides: Mapping from family name (matching + ``dataset_entry(name=...)``) to a dict of override kwargs. + + Returns: + The (possibly updated) ``kwargs`` dict. + + Raises: + ValueError: If an override dict contains keys outside + :data:`_ALLOWED_PER_FAMILY_OVERRIDE_KEYS`. + """ + dataset = kwargs.get("dataset", args[0] if args else None) + if not isinstance(dataset, ActionUnifiedIterableDataset): + log.warning( + "InfiniteDataLoader: per_family_overrides ignored - dataset is " + f"{type(dataset).__name__}, not ActionUnifiedIterableDataset." + ) + return kwargs + if not dataset._shard_across_workers: + log.warning( + "InfiniteDataLoader: per_family_overrides ignored - " + "shard_across_workers=False (every worker iterates all families)." + ) + return kwargs + + try: + rank = distributed.get_rank() + world_size = distributed.get_world_size() + except RuntimeError: + rank, world_size = 0, 1 + + # Validate override keys up-front so a typo fails loudly instead of + # silently doing nothing on most ranks. + for fam, ov in per_family_overrides.items(): + bad = set(ov) - _ALLOWED_PER_FAMILY_OVERRIDE_KEYS + if bad: + raise ValueError( + f"per_family_overrides[{fam!r}] has unsupported keys {sorted(bad)}; " + f"allowed: {sorted(_ALLOWED_PER_FAMILY_OVERRIDE_KEYS)}" + ) + + known_families = {entry.get("name", f"family_{i}") for i, entry in enumerate(dataset._datasets)} + unknown = set(per_family_overrides) - known_families + if unknown and rank == 0: + log.warning( + f"InfiniteDataLoader: per_family_overrides has unknown families " + f"{sorted(unknown)}; known: {sorted(known_families)}" + ) + + if world_size < len(dataset._datasets): + # Hare-Niemeyer requires world_size >= num_families. Skip + # per-family overrides in tiny test runs rather than crashing. + log.warning( + f"InfiniteDataLoader: per_family_overrides ignored - " + f"world_size ({world_size}) < num_families ({len(dataset._datasets)})." + ) + return kwargs + + _, family_name = ActionUnifiedIterableDataset.resolve_family_for_rank(dataset._datasets, rank, world_size) + + overrides = per_family_overrides.get(family_name) + if not overrides: + return kwargs + + base_snapshot = {k: kwargs.get(k) for k in overrides} + + # Guard against PyTorch invariants when an override sets num_workers=0. + new_num_workers = overrides.get("num_workers", kwargs.get("num_workers", 0)) + if new_num_workers == 0: + # PyTorch errors if prefetch_factor / persistent_workers are set + # while num_workers=0; clear them to keep the override safe. + kwargs.pop("prefetch_factor", None) + kwargs.pop("persistent_workers", None) + + kwargs.update(overrides) + + log.info( + f"InfiniteDataLoader: rank={rank}/{world_size} family={family_name!r} " + f"applying per_family_overrides {overrides} (was {base_snapshot})", + rank0_only=False, + ) + return kwargs + + +class InfiniteDataLoader(torch.utils.data.DataLoader): + """A dataloader that yields forever with proper seeding for reproducibility. + + All Action datasets are ``IterableDataset`` instances (map-style datasets + are automatically wrapped by :class:`~.transforms.MapToIterableAdapter`). + The loader catches ``StopIteration`` and restarts the iterator so that + iteration never ends. + """ + + def __init__( + self, + *args, + seed: int = 42, + use_deterministic_seed: bool = True, + per_family_overrides: dict[str, dict[str, Any]] | None = None, + **kwargs, + ) -> None: + """Initialize InfiniteDataLoader. + + Args: + *args: Positional arguments passed to parent DataLoader. + seed: Random seed for reproducible worker initialization. + Default is 42 for reproducibility. + use_deterministic_seed: If True, use the provided seed for reproducible + RNG initialization. If False, derive a time-based seed so that + each resume sees different data. This is preferred for large-scale + runs that resume frequently, and when ``in_order=False`` already + makes iteration order non-deterministic. + per_family_overrides: Optional ``{family_name: {kwarg: value, ...}}`` + mapping that patches DataLoader kwargs for whatever action + family this rank is assigned to under Hare-Niemeyer. Only + meaningful when the wrapped dataset is an + :class:`ActionUnifiedIterableDataset` with + ``shard_across_workers=True``. Allowed keys per family: + ``batch_size``, ``num_workers``, ``prefetch_factor``, + ``persistent_workers``, ``pin_memory``. + + Each rank serves exactly one family, so this lets heavy + families (``camera_720``, ``hand_pose``) shrink their + in-flight buffer to control host RSS, and small-token + high-IO families (``droid``, ``fractal``) expand it to + hide shard / page-cache stalls. Family names must match + ``dataset_entry(name=...)`` in + :func:`~.unified_dataset.wrap_dataset`'s ``list_of_datasets``. + + Example:: + + per_family_overrides={ + "camera_720_20260501": dict(batch_size=1, prefetch_factor=1), + "hand_pose_20260501": dict(batch_size=2, prefetch_factor=1), + "agibotworld_beta_20260501": dict(batch_size=2, prefetch_factor=1), + "fractal_20260501": dict(batch_size=8, prefetch_factor=3), + "droid_20260501": dict(batch_size=4, prefetch_factor=3), + "bridge_20260501": dict(batch_size=4, prefetch_factor=3), + } + **kwargs: Keyword arguments passed to parent DataLoader. + """ + kwargs.pop("shuffle", None) + kwargs["shuffle"] = False + + # Default to ``custom_collate_fn`` so that variable-length per-sample + # tensors (e.g. ``text_token_ids``) and multi-item keys (``video``, + # ``action``, ...) are returned as lists rather than stacked by + # PyTorch's ``default_collate``. + if kwargs.get("collate_fn") is None: + kwargs["collate_fn"] = custom_collate_fn + + # Apply per-family overrides BEFORE PyTorch sees the kwargs so the + # rank-specific batch_size / prefetch_factor / num_workers take + # effect at the DataLoader / worker_pool level, not just at runtime. + if per_family_overrides: + kwargs = _apply_per_family_overrides(args, kwargs, per_family_overrides) + + if "worker_init_fn" not in kwargs or kwargs["worker_init_fn"] is None: + kwargs["worker_init_fn"] = create_action_worker_init_fn(seed, use_deterministic_seed=use_deterministic_seed) + + num_workers = kwargs.get("num_workers", 0) + if num_workers == 0: + try: + rank = distributed.get_rank() + except RuntimeError: + rank = 0 + if use_deterministic_seed: + rank_seed = seed + rank * 9999 + else: + rank_seed = int(time.time() * 1000) % (2**32) + rank * 9999 + random.seed(rank_seed) + np.random.seed(rank_seed % (2**32)) + torch.manual_seed(rank_seed) + + super().__init__(*args, **kwargs) + self._stream_iterator: Iterator | None = None + + def __len__(self) -> int: + # Delegate to DataLoader which calls len(self.dataset). + # Raises TypeError if the underlying dataset has no __len__. + return super().__len__() + + def __iter__(self) -> Iterator: + """Yield batches forever.""" + while True: + if self._stream_iterator is None: + self._stream_iterator = super().__iter__() + try: + yield next(self._stream_iterator) # type: ignore[arg-type] + except StopIteration: + self._stream_iterator = super().__iter__() + yield next(self._stream_iterator) # type: ignore[arg-type] diff --git a/cosmos_framework/data/vfm/action/domain_utils.py b/cosmos_framework/data/vfm/action/domain_utils.py new file mode 100644 index 0000000..a38fb5c --- /dev/null +++ b/cosmos_framework/data/vfm/action/domain_utils.py @@ -0,0 +1,35 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Domain ID helpers for cross-embodiment action datasets.""" + +EMBODIMENT_TO_DOMAIN_ID: dict[str, int] = { + "no_action": 0, + "av": 1, + "camera_pose": 2, + "hand_pose": 3, + "pusht": 4, + "libero": 5, + "umi": 6, + "bridge_orig_lerobot": 7, + "droid_lerobot": 8, + "robomind-franka": 8, # Both Droid and RoboMIND-Franka are using robotiq and franka + "embodiment_b": 9, + "robomind-franka-dual": 12, + "robomind-ur": 13, + "agibotworld": 15, + "embodiment_c_gripper": 15, + "embodiment_c_gripper_ext": 15, + "fractal": 20, +} + + +def get_domain_id(embodiment_type: str) -> int: + """Get the domain ID for a given embodiment type.""" + key = embodiment_type.lower().strip() + if key not in EMBODIMENT_TO_DOMAIN_ID: + raise KeyError( + f"Unknown embodiment type: {embodiment_type!r}. " + f"Available embodiments: {sorted(EMBODIMENT_TO_DOMAIN_ID.keys())}" + ) + return EMBODIMENT_TO_DOMAIN_ID[key] diff --git a/cosmos_framework/data/vfm/action/droid_lerobot_dataset.py b/cosmos_framework/data/vfm/action/droid_lerobot_dataset.py new file mode 100644 index 0000000..2fe9200 --- /dev/null +++ b/cosmos_framework/data/vfm/action/droid_lerobot_dataset.py @@ -0,0 +1,486 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import json +import os +import random +from typing import Any, cast + +import numpy as np +import torch +import torch.nn.functional as F +import torchvision.transforms.v2 as T +from scipy.spatial.transform import Rotation as R + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.cosmos3_action_lerobot import ( + ActionNormalization, + ActionSpec, + BaseActionLeRobotDataset, + Gripper, + Joint, + Pos, + Rot, + build_action_spec, + build_episode_spans, + split_episode_ids, +) +from cosmos_framework.data.vfm.action.droid_lerobot_dataset_config import ( + _GRIPPER_STATE_FEATURE, + _JOINT_ACTION_FEATURE, + _JOINT_STATE_FEATURE, + ACTION_FEATURES, + HAS_MULTI_LANGUAGE_ANNOTATIONS, + IMAGE_FEATURES, + IS_FLAT_ACTION, + IS_GRIPPER_ACTION_FLIPPED, + LEROBOT_ROOTS, + STATE_FEATURES, +) +from cosmos_framework.data.vfm.action.pose_utils import ( + PoseConvention, + build_abs_pose_from_components, + convert_rotation, + pose_abs_to_rel, +) +from cosmos_framework.data.vfm.action.viewpoint_utils import Viewpoint + +_FILTER_DICT_PATH = "" + +# 90-degree clockwise rotation about the Z axis (in local frame), converting +# DROID Franka panda_link8 orientation to the OpenCV camera convention. +_DROID_TO_OPENCV: np.ndarray = np.array( + [ + [0.0, -1.0, 0.0], + [1.0, 0.0, 0.0], + [0.0, 0.0, 1.0], + ], + dtype=np.float32, +) + + +class DROIDLeRobotDataset(BaseActionLeRobotDataset): + """ """ + + def __init__( + self, + root: str = "", + fps: float = 15.0, + chunk_length: int = 16, + split_seed: int = 42, + split_val_ratio: float = 0.03, + split: str = "train", + mode: str = "policy", + pose_convention: PoseConvention = "backward_framewise", + action_normalization: ActionNormalization | None = None, + tolerance_s=2e-4, + viewpoint: Viewpoint = "concat_view", + use_success_only: bool = False, + video_mode: str | None = None, + action_space: str = "midtrain", + use_state: bool = False, + use_filter_dict: bool = False, + enable_fast_init: bool = False, + max_num_history_actions: int = 0, + use_image_augmentation: bool = False, + ) -> None: + """ """ + super().__init__( + fps=fps, + chunk_length=chunk_length, + split_seed=split_seed, + split_val_ratio=split_val_ratio, + split=split, + mode=mode, + embodiment_type="droid_lerobot", + viewpoint=viewpoint, + pose_convention=pose_convention, + rotation_format="rot6d", + action_normalization=action_normalization, + tolerance_s=tolerance_s, + enable_fast_init=enable_fast_init, + ) + self._use_success_only = use_success_only + self._video_mode = video_mode + self._action_space = action_space + self._use_state = use_state + self._use_filter_dict = use_filter_dict + self._max_num_history_actions = max_num_history_actions + self._use_image_augmentation = use_image_augmentation + if max_num_history_actions > 0 and action_space not in ("midtrain", "joint_pos"): + raise ValueError( + f"max_num_history_actions is only supported with action_space='midtrain' or 'joint_pos', got {action_space!r}" + ) + + self._is_val_temp_seg = split == "val_temp_seg" + self._to_opencv = _DROID_TO_OPENCV + + version = os.path.basename(root) + try: + lerobot_roots = LEROBOT_ROOTS[version] + self._image_features = IMAGE_FEATURES[version] + self._state_features = STATE_FEATURES[version] + self._action_features = ACTION_FEATURES[version] + self._is_flat_action = IS_FLAT_ACTION[version] + self._has_multi_language_annotations = HAS_MULTI_LANGUAGE_ANNOTATIONS[version] + self._is_gripper_action_flipped = IS_GRIPPER_ACTION_FLIPPED[version] + except KeyError as e: + raise ValueError(f"Unknown version: {version!r}. Supported: {list(LEROBOT_ROOTS.keys())}") from e + + if self._use_success_only and lerobot_roots: + lerobot_roots = [x for x in lerobot_roots if x.split("/", 1)[0] == "success"] + + self._all_shard_roots = [os.path.join(root, x) for x in lerobot_roots] if lerobot_roots else [root] + + observation_ts = [i * self._dt for i in range(0, self._chunk_length + 1)] + action_ts = [i * self._dt for i in range(0, self._chunk_length)] + if self._max_num_history_actions > 0 and self._action_space in ("midtrain", "joint_pos"): + observation_ts_ext = [i * self._dt for i in range(-self._max_num_history_actions, self._chunk_length + 1)] + action_ts_ext = [i * self._dt for i in range(-self._max_num_history_actions, self._chunk_length)] + else: + observation_ts_ext = observation_ts + action_ts_ext = action_ts + self._delta_timestamps: dict[str, list[float]] = { + self._state_features: observation_ts_ext, + self._action_features: action_ts_ext, + } + if self._viewpoint in ("wrist_view", "concat_view"): + self._delta_timestamps[self._image_features["wrist"]] = observation_ts + if self._viewpoint in ("third_person_view", "concat_view"): + self._delta_timestamps[self._image_features["left"]] = observation_ts + self._delta_timestamps[self._image_features["right"]] = observation_ts + if self._action_space == "joint_pos": + self._delta_timestamps[_JOINT_ACTION_FEATURE] = action_ts + if self._use_state or self._max_num_history_actions > 0: + self._delta_timestamps[_JOINT_STATE_FEATURE] = observation_ts_ext + self._delta_timestamps[_GRIPPER_STATE_FEATURE] = observation_ts_ext + if self._use_state and self._action_space != "joint_pos": + self._delta_timestamps[_GRIPPER_STATE_FEATURE] = observation_ts + + if self._use_filter_dict: + with open(_FILTER_DICT_PATH) as f: + self._filter_dict = json.load(f) + + self._image_augmentor: T.Compose | None = None + + def _append_index_records(self, *, meta, ds_idx: int, dataset_label: str | None = None) -> None: + """ """ + if not self._use_filter_dict: + super()._append_index_records(meta=meta, ds_idx=ds_idx, dataset_label=dataset_label) + return + + episode_ids = split_episode_ids( + total_episodes=meta.total_episodes, + seed=self._split_seed, + val_ratio=self._split_val_ratio, + split=self._split, + ) + episode_spans, _, sample_count = build_episode_spans( + meta.episodes, episode_ids, self._chunk_length, sample_stride=self._sample_stride + ) + + class_name = self.__class__.__name__ + label = f" [{dataset_label}]" + + log.info(f"{class_name}{label}: split={self._split}, num episodes={len(episode_ids)}") + + filtered_count = 0 + for episode_id, sample_start, valid_len in episode_spans: + ep_id_str = meta.episodes[episode_id]["episode_id"] + episode_key = f"gs://xembodiment_data/r2d2/r2d2-data-full/{ep_id_str}/recordings/MP4--gs://xembodiment_data/r2d2/r2d2-data-full/{ep_id_str}/trajectory.h5" + ranges = self._filter_dict.get(episode_key) + if ranges is None: + continue + for s, e in ranges: + sub_start = max(s, 0) + sub_end = min(e - self._chunk_length, valid_len) + sub_valid_len = max(0, sub_end - sub_start) + if sub_valid_len > 0: + self._episode_records.append((ds_idx, sample_start + sub_start, sub_valid_len, episode_id)) + self._num_valid_indices += sub_valid_len + self._episode_cum_ends.append(self._num_valid_indices) + filtered_count += sub_valid_len + + if sample_count > 0: + log.info( + f"{class_name}{label}: kept {filtered_count} / {sample_count} ({100.0 * filtered_count / sample_count:.2f} %) samples" + ) + + def _register_sources(self, indices: list[int] | None = None) -> None: + """ """ + super()._register_sources(indices) + if self._is_val_temp_seg: + self._apply_temp_seg_filter() + + def _apply_temp_seg_filter(self) -> None: + """Replace index records with one high-scoring segment per episode. + + A segment is interesting if either: + - The gripper action changes significantly (open/close transition), or + - The gripper is closed and the end-effector position is moving. + Among qualifying segments the one with the highest score is kept. + """ + ds = self._get_dataset(0) + chunk_size = self._chunk_length + 1 + gripper_change_threshold = 0.5 + ee_movement_threshold = 0.01 + + new_records: list[tuple[int, int, int, int]] = [] + num_episodes = len(self._episode_records) + + for ds_idx, sample_start, valid_len, episode_id in self._episode_records: + end = sample_start + valid_len + self._chunk_length + num_candidates = valid_len + if num_candidates <= 0: + continue + + episode_data = ds.hf_dataset[sample_start:end] + actions = torch.tensor(np.array(episode_data[self._action_features])) # [N,action_dim] + states = torch.tensor(np.array(episode_data[self._state_features])) # [N,state_dim] + + gripper_action = actions[:, 6] if self._is_flat_action else actions # [N] + ee_pos = states[:, :3] # [N,3] + ee_disp = (ee_pos[1:] - ee_pos[:-1]).norm(dim=-1) # [N-1] + + ee_disp_windows = ee_disp.unfold(0, self._chunk_length, 1) # [num_candidates,chunk_length] + gripper_windows = gripper_action.unfold(0, chunk_size, 1) # [num_candidates,chunk_size] + + gripper_range = gripper_windows.max(dim=1).values - gripper_windows.min(dim=1).values # [num_candidates] + total_ee_movement = ee_disp_windows.sum(dim=1) # [num_candidates] + gripper_closed_ratio = (gripper_windows < 0.5).float().mean(dim=1) # [num_candidates] + + has_gripper_change = gripper_range > gripper_change_threshold + gripper_closed = gripper_closed_ratio > 0.5 + has_ee_movement = total_ee_movement > ee_movement_threshold + + scores = torch.zeros(num_candidates) # [num_candidates] + scores[has_gripper_change] = 0.5 + gripper_range[has_gripper_change] + total_ee_movement[has_gripper_change] + + closed_and_moving = gripper_closed & ~has_gripper_change & has_ee_movement + scores[closed_and_moving] = 1.0 + total_ee_movement[closed_and_moving] + + if scores.max().item() > 0: + best_offset = int(scores.argmax().item()) + new_records.append((ds_idx, sample_start + best_offset, 1, episode_id)) + + self._episode_records = new_records + self._num_valid_indices = len(new_records) + self._episode_cum_ends = list(range(1, len(new_records) + 1)) + + log.info(f"DROIDLeRobotDataset: val_temp_seg kept {len(new_records)} segments from {num_episodes} episodes") + + def _compose_multi_view(self, sample: dict[str, Any]) -> torch.Tensor: + """Compose wrist, left, and right views into a single frame. + + Layout (per frame): + ┌──────────────┐ + │ wrist │ (H, W) + ├───────┬──────┤ + │ left │ right│ (H/2, W/2) each + └───────┴──────┘ + + Left and right exterior cameras are downscaled by 2x so that they + tile to the same width as the wrist view. The output height is 3H/2. + + Returns: + Composited raw video tensor in ``(T,C,H_out,W)`` float format. + """ + wrist = sample[self._image_features["wrist"]] # [T,C,H,W] + left = sample[self._image_features["left"]] # [T,C,H_l,W_l] + right = sample[self._image_features["right"]] # [T,C,H_r,W_r] + + if self._use_image_augmentation: + if self._image_augmentor is None: + _, _, h, w = wrist.shape + self._image_augmentor = T.Compose( + [ + T.RandomCrop((int(h * 0.95), int(w * 0.95))), + T.Resize((h, w), antialias=True), + T.ColorJitter(brightness=0.3, contrast=0.4, saturation=0.5, hue=0.08), + ] + ) + n, m = wrist.shape[0], wrist.shape[0] + left.shape[0] + combined = self._image_augmentor(torch.cat([wrist, left, right], dim=0)) + wrist, left, right = combined[:n], combined[n:m], combined[m:] + + _, _, h_w, w_w = wrist.shape + half_h, half_w = h_w // 2, w_w // 2 + + left = F.interpolate(left, size=(half_h, half_w), mode="bilinear", align_corners=False) # [T,C,H/2,W/2] + right = F.interpolate(right, size=(half_h, half_w), mode="bilinear", align_corners=False) # [T,C,H/2,W/2] + bottom = torch.cat([left, right], dim=-1) # [T,C,H/2,W] + + composite = torch.cat([wrist, bottom], dim=-2) # [T,C,3H/2,W] + return composite # [T,C,3H/2,W] + + def _build_action_spec(self) -> ActionSpec: + """DROID: 10D ``[Pos, Rot6d, Gripper]`` for ``ee_pose``, + 8D ``[Joint(7), Gripper]`` for ``joint_pos``. + """ + if self._action_space == "joint_pos": + return build_action_spec(Joint(n=7, label="joint"), Gripper()) + return build_action_spec(Pos(), Rot("rot6d"), Gripper()) + + def __getitem__(self, idx: int) -> dict[str, Any]: + """ """ + mode, _, _, sample = self._fetch_sample(idx) + + if self._has_multi_language_annotations: + tasks = sample["task"].split(" | ") + ai_caption = random.choice(tasks) + else: + ai_caption = sample["task"] + + if self._skip_video_loading: + video = None + elif self._video_mode is None: + if self._viewpoint == "concat_view": + video = self._compose_multi_view(sample) + else: + video = sample[self._image_features["wrist"]] # [T,C,H,W] + else: + if self._video_mode == "wrist": + video = sample[self._image_features["wrist"]] + if self._video_mode in ("rand_exterior", "wrist_rand_exterior"): + exterior_key = random.choice([self._image_features["left"], self._image_features["right"]]) + if self._video_mode == "rand_exterior": + video = sample[exterior_key] + else: + video = torch.cat([sample[self._image_features["wrist"]], sample[exterior_key]], dim=2) + if self._video_mode in ("wrist_left_exterior", "wrist_both_exterior"): + wrist = sample[self._image_features["wrist"]] + half_h, half_w = wrist.shape[2] // 2, wrist.shape[3] // 2 + left = F.interpolate( + sample[self._image_features["left"]], size=(half_h, half_w), mode="bilinear", align_corners=False + ) + if self._video_mode == "wrist_left_exterior": + right = torch.zeros_like(left) + if self._video_mode == "wrist_both_exterior": + right = F.interpolate( + sample[self._image_features["right"]], + size=(half_h, half_w), + mode="bilinear", + align_corners=False, + ) + video = torch.cat([wrist, torch.cat([left, right], dim=-1)], dim=-2) + + extras: dict[str, Any] = {} + + if self._action_space == "midtrain": + pose_convention = cast(PoseConvention, self._pose_convention) + state = sample[self._state_features] # [T+1, state_dim] or [H+T+1, state_dim] + poses_abs = build_abs_pose_from_components(state[:, 0:3], state[:, 3:6], "euler_xyz") + poses_abs[:, :3, :3] = poses_abs[:, :3, :3] @ self._to_opencv + initial_pose = torch.from_numpy(poses_abs[-self._chunk_length - 1].copy()).float() + poses_rel = pose_abs_to_rel(poses_abs, rotation_format="rot6d", pose_convention=pose_convention) + gripper = ( + sample[self._action_features][:, [6]] + if self._is_flat_action + else sample[self._action_features].unsqueeze(-1) + ) + if self._is_gripper_action_flipped: + gripper = 1.0 - gripper + action = torch.from_numpy( + np.concatenate([poses_rel[-self._chunk_length :], gripper[-self._chunk_length :]], axis=-1) + ).float() # [T,10] + extras["initial_pose"] = initial_pose + if self._max_num_history_actions > 0: + _, _, _, frame_offset = self._resolve_index(int(idx)) + num_available = min(self._max_num_history_actions, frame_offset * self._sample_stride) + actual_h = num_available + # with 0.5 probability, randomly sample the number of history frames + if random.random() < 0.5: + actual_h = random.randint(0, num_available) + if actual_h > 0: + hist_action_raw = torch.from_numpy( + np.concatenate( + [ + poses_rel[-self._chunk_length - actual_h : -self._chunk_length], + gripper[-self._chunk_length - actual_h : -self._chunk_length], + ], + axis=-1, + ) + ).float() + extras["history_action"] = self._normalize_action(hist_action_raw) + if self._use_state: + initial_gripper = sample[_GRIPPER_STATE_FEATURE][0].unsqueeze(-1) + if self._is_gripper_action_flipped: + initial_gripper = 1.0 - initial_gripper + initial_rot6d = convert_rotation(poses_abs[-self._chunk_length - 1, :3, :3], "matrix", "rot6d") + initial_state = torch.from_numpy( + np.concatenate((poses_abs[-self._chunk_length - 1, :3, 3], initial_rot6d, initial_gripper), axis=-1) + ).float() + action = torch.cat([initial_state.unsqueeze(0), action], dim=0) + if self._action_space == "ee_pose_delta": + state = sample[self._state_features] + pose = np.tile(np.eye(4), (state.shape[0], 1, 1)) + pose[:, :3, :3] = R.from_euler("xyz", state[:, 3:6]).as_matrix() + pose[:, :3, 3] = state[:, 0:3] + pose_delta = np.linalg.inv(pose[0]) @ pose[1:] + gripper = sample[self._action_features].unsqueeze(-1) + if self._is_gripper_action_flipped: + gripper = 1.0 - gripper + action = torch.from_numpy( + np.concatenate((pose_delta[:, :3, 3], pose_delta[:, :3, 0], pose_delta[:, :3, 1], gripper), axis=-1) + ).float() + if self._use_state: + initial_gripper = sample[_GRIPPER_STATE_FEATURE][0].unsqueeze(-1) + if self._is_gripper_action_flipped: + initial_gripper = 1.0 - initial_gripper + initial_state = torch.from_numpy( + np.concatenate((pose[0, :3, 3], pose[0, :3, 0], pose[0, :3, 1], initial_gripper), axis=-1) + ).float() + action = torch.cat([initial_state.unsqueeze(0), action], dim=0) + if self._action_space == "joint_pos": + gripper = sample[self._action_features][-self._chunk_length :].unsqueeze(-1) + if self._is_gripper_action_flipped: + gripper = 1.0 - gripper + action = torch.cat((sample[_JOINT_ACTION_FEATURE], gripper), dim=-1).float() + if self._max_num_history_actions > 0: + _, _, _, frame_offset = self._resolve_index(int(idx)) + num_available = min(self._max_num_history_actions, frame_offset * self._sample_stride) + actual_h = num_available + if random.random() < 0.5: + actual_h = random.randint(0, num_available) + if actual_h > 0: + hist_joint = sample[_JOINT_STATE_FEATURE][ + -self._chunk_length - 1 - actual_h : -self._chunk_length - 1 + ] + hist_gripper = sample[_GRIPPER_STATE_FEATURE][ + -self._chunk_length - 1 - actual_h : -self._chunk_length - 1 + ].unsqueeze(-1) + if self._is_gripper_action_flipped: + hist_gripper = 1.0 - hist_gripper + hist_action_raw = torch.cat((hist_joint, hist_gripper), dim=-1).float() + extras["history_action"] = self._normalize_action(hist_action_raw) + if self._use_state: + initial_gripper = sample[_GRIPPER_STATE_FEATURE][-self._chunk_length - 1].unsqueeze(-1) + if self._is_gripper_action_flipped: + initial_gripper = 1.0 - initial_gripper + initial_state = torch.cat( + (sample[_JOINT_STATE_FEATURE][-self._chunk_length - 1], initial_gripper), dim=-1 + ).float() + action = torch.cat([initial_state.unsqueeze(0), action], dim=0) + + if self._viewpoint == "concat_view" and self._video_mode in ( + None, + "wrist_left_exterior", + "wrist_both_exterior", + ): + extras["additional_view_description"] = ( + "The top row is from the wrist-mounted camera. " + "The bottom row contains two horizontally concatenated third-person perspective views of the scene from opposite sides, with the robot visible." + ) + + return self._build_result( + mode=mode, + video=video, + action=action, + ai_caption=ai_caption, + **extras, + ) + + @property + def action_dim(self) -> int: + """ """ + return 8 if self._action_space == "joint_pos" else 10 diff --git a/cosmos_framework/data/vfm/action/droid_lerobot_dataset_config.py b/cosmos_framework/data/vfm/action/droid_lerobot_dataset_config.py new file mode 100644 index 0000000..e97489c --- /dev/null +++ b/cosmos_framework/data/vfm/action/droid_lerobot_dataset_config.py @@ -0,0 +1,100 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +_INSTITUTIONS = [ + "AUTOLab", + "CLVR", + "GuptaLab", + "ILIAD", + "IPRL", + "IRIS", + "PennPAL", + "RAD", + "RAIL", + "REAL", + "RPL", + "TRI", + "WEIRD", +] + +LEROBOT_ROOTS = { + "droid_lerobot_20260115_no_noops": None, + "droid_plus_lerobot_320x180_20260406_sharded": [f"success/{x}" for x in _INSTITUTIONS] + + [f"failure/{x}" for x in _INSTITUTIONS], + "droid_plus_lerobot_320x180_20260406": ["success", "failure"], + "droid_plus_lerobot_640x360_20260412_sharded": [f"success/{x}" for x in _INSTITUTIONS] + + [f"failure/{x}" for x in _INSTITUTIONS], + "droid_plus_lerobot_640x360_20260412": ["success", "failure"], +} + +IMAGE_FEATURES = { + "droid_lerobot_20260115_no_noops": { + "wrist": "observation.images.wrist_image_left", + "left": "observation.images.exterior_image_1_left", + "right": "observation.images.exterior_image_2_left", + }, + "droid_plus_lerobot_320x180_20260406_sharded": { + "wrist": "observation.image.wrist_image_left", + "left": "observation.image.exterior_image_1_left", + "right": "observation.image.exterior_image_2_left", + }, + "droid_plus_lerobot_320x180_20260406": { + "wrist": "observation.image.wrist_image_left", + "left": "observation.image.exterior_image_1_left", + "right": "observation.image.exterior_image_2_left", + }, + "droid_plus_lerobot_640x360_20260412_sharded": { + "wrist": "observation.image.wrist_image_left", + "left": "observation.image.exterior_image_1_left", + "right": "observation.image.exterior_image_2_left", + }, + "droid_plus_lerobot_640x360_20260412": { + "wrist": "observation.image.wrist_image_left", + "left": "observation.image.exterior_image_1_left", + "right": "observation.image.exterior_image_2_left", + }, +} + +STATE_FEATURES = { + "droid_lerobot_20260115_no_noops": "observation.state", + "droid_plus_lerobot_320x180_20260406_sharded": "observation.state.cartesian_position", + "droid_plus_lerobot_320x180_20260406": "observation.state.cartesian_position", + "droid_plus_lerobot_640x360_20260412_sharded": "observation.state.cartesian_position", + "droid_plus_lerobot_640x360_20260412": "observation.state.cartesian_position", +} + +ACTION_FEATURES = { + "droid_lerobot_20260115_no_noops": "action", + "droid_plus_lerobot_320x180_20260406_sharded": "action.gripper_position", + "droid_plus_lerobot_320x180_20260406": "action.gripper_position", + "droid_plus_lerobot_640x360_20260412_sharded": "action.gripper_position", + "droid_plus_lerobot_640x360_20260412": "action.gripper_position", +} + +IS_FLAT_ACTION = { + "droid_lerobot_20260115_no_noops": True, + "droid_plus_lerobot_320x180_20260406_sharded": False, + "droid_plus_lerobot_320x180_20260406": False, + "droid_plus_lerobot_640x360_20260412_sharded": False, + "droid_plus_lerobot_640x360_20260412": False, +} + +HAS_MULTI_LANGUAGE_ANNOTATIONS = { + "droid_lerobot_20260115_no_noops": False, + "droid_plus_lerobot_320x180_20260406_sharded": True, + "droid_plus_lerobot_320x180_20260406": True, + "droid_plus_lerobot_640x360_20260412_sharded": True, + "droid_plus_lerobot_640x360_20260412": True, +} + +IS_GRIPPER_ACTION_FLIPPED = { + "droid_lerobot_20260115_no_noops": False, + "droid_plus_lerobot_320x180_20260406_sharded": True, + "droid_plus_lerobot_320x180_20260406": True, + "droid_plus_lerobot_640x360_20260412_sharded": True, + "droid_plus_lerobot_640x360_20260412": True, +} + +_JOINT_ACTION_FEATURE = "action.joint_position" +_JOINT_STATE_FEATURE = "observation.state.joint_positions" +_GRIPPER_STATE_FEATURE = "observation.state.gripper_position" diff --git a/cosmos_framework/data/vfm/action/dummy_dataset.py b/cosmos_framework/data/vfm/action/dummy_dataset.py new file mode 100644 index 0000000..94ef2a1 --- /dev/null +++ b/cosmos_framework/data/vfm/action/dummy_dataset.py @@ -0,0 +1,100 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import random +from typing import Any + +import torch +from torch.utils.data import Dataset + +from cosmos_framework.utils import log + + +class DummyDataset(Dataset): + """ + A dummy dataset that generates random images/videos, camera poses, and actions + mimicking the structure of UMI/iPhUMI datasets. + """ + + def __init__( + self, + length: int = 100, + image_size: int = 256, + chunk_length: int = 16, # must be divisible by 4 + camera_pose_dim: int = 9, # 3 pos + 6 rot + fps: int = 16, + mode: str = "joint", + randomize_resolution: bool = True, + ): + self.length = int(length) + self.image_size = image_size + self.chunk_length = chunk_length + assert self.chunk_length % 4 == 0, "chunk_length must be divisible by 4" + self.camera_pose_dim = camera_pose_dim + self.fps = fps + self.randomize_resolution = randomize_resolution + + assert mode in ["joint", "forward_dynamics", "inverse_dynamics", "policy", "image2video"], ( + "mode must be either joint, forward_dynamics, inverse_dynamics, policy, or image2video" + ) + self.mode = mode + + def __len__(self) -> int: + return self.length + + def __getitem__(self, idx: int) -> dict[str, Any]: + if self.mode == "joint": + mode = random.choice(["forward_dynamics", "inverse_dynamics", "policy", "image2video"]) + else: + mode = self.mode + + # Video: (C, T, H, W) uint8 + video_length = self.chunk_length + 1 + + if self.randomize_resolution: + # Randomize one dimension downward so the aspect ratio varies, while + # keeping both dimensions <= image_size. This ensures auto-detected + # resolution stays in the expected tier and exercises the padding path + # (the transforms pipeline will pad back up to the predefined target). + h = self.image_size - random.randint(32, 32) + w = self.image_size - random.randint(32, 32) + log.debug(f"DummyDataset[{idx}]: before padding resolution = ({h}, {w})") + else: + h = self.image_size + w = self.image_size + + video = torch.randint(0, 256, (3, video_length, h, w), dtype=torch.uint8) # [3,T+1,H,W] + + # Camera poses: (T, 9) float32 + # 3 position + 6 rotation (first two rows of rotation matrix flattened) + camera_poses = torch.randn(self.chunk_length, self.camera_pose_dim, dtype=torch.float32) # [T,camera_pose_dim] + + # EEF commands (actions): (T, 1) float32 + eef_commands = torch.randn(self.chunk_length, 1, dtype=torch.float32) # [T,1] + + # FPS: scalar (0-D tensor so batching produces [B] not [B, 1]) + fps = torch.tensor(self.fps, dtype=torch.long) # scalar + + # Index + key = torch.tensor([idx], dtype=torch.long) # [1] + + + # chunk_length is L, given L + 1 video frames and optionally L relative action + # video: predicting L video frames given 1 frame + # forward_dynamics: predicting L video frames given 1 frame and L action + # inverse_dynamics: predicting L action given L+1 frames (TODO: do we need a state too?) + # policy: predicting L action and L frames given 1 frame + + # Combine camera poses and eef commands into raw action tensor + action_tensor = torch.cat([camera_poses, eef_commands], dim=1) # [T,camera_pose_dim+1] + + return { + "video": video, + "action": action_tensor, + "conditioning_fps": fps, + "ai_caption": "A dummy video for testing.", + "mode": mode, + "__key__": key, + "domain_id": torch.tensor(0, dtype=torch.long), + "viewpoint": "ego_view", + } diff --git a/cosmos_framework/data/vfm/action/embodiment_b_dataset.py b/cosmos_framework/data/vfm/action/embodiment_b_dataset.py new file mode 100644 index 0000000..a1f1e3c --- /dev/null +++ b/cosmos_framework/data/vfm/action/embodiment_b_dataset.py @@ -0,0 +1,74 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Any + +import torch + +from cosmos_framework.data.vfm.action.cosmos3_action_lerobot import ( + ActionNormalization, + BaseActionLeRobotDataset, +) +from cosmos_framework.data.vfm.action.embodiment_b_dataset_config import LEROBOT_ROOTS +from cosmos_framework.data.vfm.action.viewpoint_utils import Viewpoint + + +class Embodiment_bDataset(BaseActionLeRobotDataset): + """ """ + + def __init__( + self, + fps: float = 10.0, + chunk_length: int = 16, + split_seed: int = 42, + split_val_ratio: float = 0.05, + split: str = "train", + video_key: str = "observation.images.camera_top_left", + mode: str = "policy", + action_normalization: ActionNormalization | None = None, + viewpoint: Viewpoint = "ego_view", + ) -> None: + """ """ + super().__init__( + fps=fps, + chunk_length=chunk_length, + split_seed=split_seed, + split_val_ratio=split_val_ratio, + split=split, + mode=mode, + embodiment_type="embodiment_b", + viewpoint=viewpoint, + action_normalization=action_normalization, + tolerance_s=1e-4, + ) + self._video_key = video_key + + self._all_shard_roots = [f"/{x}" for x in LEROBOT_ROOTS] + + self._delta_timestamps = { + "observation.images.camera_top_left": [i * self._dt for i in range(0, self._chunk_length + 1)], + "observation.images.camera_wrist_left": [i * self._dt for i in range(0, self._chunk_length + 1)], + "observation.images.camera_wrist_right": [i * self._dt for i in range(0, self._chunk_length + 1)], + "action.arm.position": [i * self._dt for i in range(0, self._chunk_length)], + "action.end.position": [i * self._dt for i in range(0, self._chunk_length)], + "action.effector.position": [i * self._dt for i in range(0, self._chunk_length)], + } + + def __getitem__(self, idx: int) -> dict[str, Any]: + """ """ + mode, _, _, sample = self._fetch_sample(idx) + + ai_caption = sample["task"] + + video = sample[self._video_key] # [T,C,H,W] + arm_position = sample["action.arm.position"] + end_position = sample["action.end.position"] + effector_position = sample["action.effector.position"] + action = torch.cat([arm_position, end_position, effector_position], dim=-1) # [T,30] + + return self._build_result(mode=mode, video=video, action=action, ai_caption=ai_caption) + + @property + def action_dim(self) -> int: + """ """ + return 30 diff --git a/cosmos_framework/data/vfm/action/embodiment_b_dataset_config.py b/cosmos_framework/data/vfm/action/embodiment_b_dataset_config.py new file mode 100644 index 0000000..1312079 --- /dev/null +++ b/cosmos_framework/data/vfm/action/embodiment_b_dataset_config.py @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +LEROBOT_ROOTS = [ + "data_sample_embodiment_b_20260123/Embodiment_b_data_for_Nvidia_20260123_1/1222_v30", + "data_sample_embodiment_b_20260123/Embodiment_b_data_for_Nvidia_20260123_2/1223_v30", + "data_sample_embodiment_b_20260123/Embodiment_b_data_for_Nvidia_20260123_2/1224_v30", + "data_sample_embodiment_b_20260123/Embodiment_b_data_for_Nvidia_20260123_3/1225_v30", + "data_sample_embodiment_b_20260123/Embodiment_b_data_for_Nvidia_20260123_3/1226_v30", + "data_sample_embodiment_b_20260123/Embodiment_b_data_for_Nvidia_20260123_4/1227_v30", + "data_sample_embodiment_b_20260123/Embodiment_b_data_for_Nvidia_20260123_5/1229_v30", + "data_sample_embodiment_b_20260123/Embodiment_b_data_for_Nvidia_20260123_5/1230_v30", + "data_sample_embodiment_b_20260123/Embodiment_b_data_for_Nvidia_20260123_6/1231_v30", + "data_sample_embodiment_b_20260123/Embodiment_b_data_for_Nvidia_20260123_6/1232_v30", + "data_sample_embodiment_b_20260123/Embodiment_b_data_for_Nvidia_20260123_7/1233_v30", + "data_sample_embodiment_b_20260123/Embodiment_b_data_for_Nvidia_20260123_8/1235_v30", +] diff --git a/cosmos_framework/data/vfm/action/embodiment_c_dataset.py b/cosmos_framework/data/vfm/action/embodiment_c_dataset.py new file mode 100644 index 0000000..40bfba9 --- /dev/null +++ b/cosmos_framework/data/vfm/action/embodiment_c_dataset.py @@ -0,0 +1,658 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Embodiment C datasets: multi-task LeRobot with instruction annotations from meta/info.json. + +Provides gripper variants. + +Action representation: + - **Relative FK-pose**: calibrated head camera pose + left/right + gripper-base wrist poses are computed via URDF forward kinematics from + ``observation.state``. Gripper wrist rotations are then converted through + the dataset ``_to_opencv`` mapping for action/viewer display. The viewer + can optionally request source-state FK link poses for direct mesh playback. + +View modes: + - **ego_view**: single ``observation.images.top_head`` camera. + - **concat_view**: top-head view on top, left/right wrist views resized and + concatenated horizontally on the bottom (like DROID). +""" + +from __future__ import annotations + +import json +import os +from collections.abc import Callable +from pathlib import Path +from typing import Any, Literal, cast + +import numpy as np +import torch +import torch.nn.functional as F +from lerobot.datasets.lerobot_dataset import LeRobotDatasetMetadata + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.embodiment_c_dataset_config import ( + CUSTOM_GRIPPER_SUBPATHS, + DEFAULT_CUSTOM_GRIPPER_EXT_ROOT, + DEFAULT_CUSTOM_GRIPPER_ROOT, + DEFAULT_OFFSHELF_GRIPPER_ROOT, + OFFSHELF_GRIPPER_SUBPATHS, +) +from cosmos_framework.data.vfm.action.embodiment_c_fk import ( + AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST, + apply_agibot_gripper_to_opencv, + compute_fk_transforms_batch, + compute_link_poses_batch, + convert_gripper_state_to_open_fraction, + extract_fk_transforms_from_link_poses, +) +from cosmos_framework.data.vfm.action.embodiment_c_spec import ( + AGIBOT_GEAR_EXT_STATE_LEFT_HAND_SLICE, + AGIBOT_GEAR_EXT_STATE_RIGHT_HAND_SLICE, + AGIBOT_GEAR_GRIPPER_NORMALIZER_EMBODIMENT_TYPE, + AGIBOT_GEAR_VIDEO_KEY, + get_embodiment_c_kind_spec, +) +from cosmos_framework.data.vfm.action.cosmos3_action_lerobot import ( + ActionNormalization, + ActionSpec, + BaseActionLeRobotDataset, + Gripper, + Pos, + Rot, + build_action_spec, + split_episode_ids, +) +from cosmos_framework.data.vfm.action.pose_utils import PoseConvention, pose_abs_to_rel +from cosmos_framework.data.vfm.action.viewpoint_utils import Viewpoint + +# Camera keys for concat_view. +_TOP_HEAD_KEY = AGIBOT_GEAR_VIDEO_KEY +_HAND_LEFT_KEY = "observation.images.hand_left" +_HAND_RIGHT_KEY = "observation.images.hand_right" + + +def _resolve_root_paths( + root: str | list[str] | tuple[str, ...], + expand_root_fn: Callable[[str], list[str]], +) -> list[str]: + """Normalize a root argument into the list of concrete shard roots.""" + + if isinstance(root, str): + return expand_root_fn(root) + resolved_roots: list[str] = [] + for item in root: + resolved_roots.extend(expand_root_fn(item)) + return resolved_roots + + +def _build_delta_timestamps(video_key: str, chunk_length: int, dt: float) -> dict[str, list[float]]: + """Build the LeRobot delta-timestamp layout for AgiBot samples.""" + + return { + video_key: [index * dt for index in range(0, chunk_length + 1)], + "observation.state": [index * dt for index in range(0, chunk_length + 1)], + } + + +def _load_instruction_segments(lerobot_root: str) -> dict[str, list[dict[str, Any]]]: + """Load instruction_segments from meta/info.json for a task path.""" + info_path = Path(lerobot_root) / "meta" / "info.json" + if not info_path.is_file(): + return {} + try: + with open(info_path) as f: + info = json.load(f) + segments = info.get("instruction_segments", {}) + return segments if isinstance(segments, dict) else {} + except Exception as e: + log.warning(f"EmbodimentCGripperDataset: failed to load {info_path}: {e}") + return {} + + +def _extract_high_level_instruction(entry: Any) -> str | None: + """Extract the episode-level instruction text from one metadata entry.""" + + if isinstance(entry, str): + text = entry.strip() + return text or None + if isinstance(entry, dict): + value = entry.get("high_level_instruction") + if isinstance(value, str): + text = value.strip() + return text or None + return None + + +def _load_high_level_instructions(lerobot_root: str) -> dict[str, str]: + """Load high_level_instruction from meta/info.json for a task path.""" + + info_path = Path(lerobot_root) / "meta" / "info.json" + if not info_path.is_file(): + return {} + try: + with open(info_path) as f: + info = json.load(f) + raw_instructions = info.get("high_level_instruction", {}) + if not isinstance(raw_instructions, dict): + return {} + high_level_instructions: dict[str, str] = {} + for episode_key, entry in raw_instructions.items(): + instruction = _extract_high_level_instruction(entry) + if instruction is not None: + high_level_instructions[str(episode_key)] = instruction + return high_level_instructions + except Exception as e: + log.warning(f"EmbodimentCGripperDataset: failed to load high_level_instruction from {info_path}: {e}") + return {} + + +def _coerce_frame_index(value: Any) -> int | None: + """Convert a frame index value to ``int`` when possible.""" + + try: + return int(value) + except (TypeError, ValueError): + return None + + +def _get_instruction_frame_bounds(segment: dict[str, Any]) -> tuple[int, int] | None: + """Extract inclusive ``[start,end]`` frame bounds from one instruction segment.""" + + start = _coerce_frame_index(segment.get("start_frame_index")) + end = _coerce_frame_index(segment.get("end_frame_index", segment.get("success_frame_index"))) + if start is None or end is None or end < start: + return None + return start, end + + +def _get_gripper_state_slices(embodiment_type: str, kind_spec: Any) -> tuple[slice, slice]: + """Return ``(right,left)`` state slices for scalar AgiBot gripper positions.""" + + if embodiment_type == "embodiment_c_gripper_ext": + return AGIBOT_GEAR_EXT_STATE_RIGHT_HAND_SLICE, AGIBOT_GEAR_EXT_STATE_LEFT_HAND_SLICE + state_hand_slice = kind_spec.state_hand_slice + return ( + slice(state_hand_slice.start + 1, state_hand_slice.start + 2), + slice(state_hand_slice.start, state_hand_slice.start + 1), + ) + + +def _build_instruction_episode_spans( + *, + episodes: Any, + episode_ids: list[int], + instruction_segments: dict[str, list[dict[str, Any]]], + high_level_instructions: dict[str, str], + chunk_length: int, + sample_stride: int = 1, +) -> tuple[list[tuple[int, int, int]], dict[int, int], int, int, int, int]: + """Build valid spans from instruction segments, with episode-level fallback.""" + + assert sample_stride >= 1, f"sample_stride must be >= 1, got {sample_stride}" + + dataset_from_index = list(episodes["dataset_from_index"]) + dataset_to_index = list(episodes["dataset_to_index"]) + length = list(episodes["length"]) + + spans: list[tuple[int, int, int]] = [] + episode_start_rows: dict[int, int] = {} + valid_count = 0 + sample_count = 0 + missing_episode_count = 0 + fallback_episode_count = 0 + + for episode_id in episode_ids: + episode_start = int(dataset_from_index[episode_id]) + episode_stop = int(dataset_to_index[episode_id]) + episode_length = int(length[episode_id]) + episode_start_rows[episode_id] = episode_start + sample_count += episode_length + + raw_segments = instruction_segments.get(str(episode_id), []) + if not raw_segments: + missing_episode_count += 1 + if str(episode_id) in high_level_instructions: + max_episode_frame = min(episode_length - 1, episode_stop - episode_start - 1) + valid_len = max_episode_frame - chunk_length + 1 + if valid_len > 0: + strided_valid_len = (valid_len + sample_stride - 1) // sample_stride + spans.append((episode_id, episode_start, strided_valid_len)) + valid_count += strided_valid_len + fallback_episode_count += 1 + continue + + max_episode_frame = min(episode_length - 1, episode_stop - episode_start - 1) + for segment in raw_segments: + bounds = _get_instruction_frame_bounds(segment) + if bounds is None: + continue + + segment_start, segment_end = bounds + segment_start = max(0, segment_start) + segment_end = min(max_episode_frame, segment_end) + valid_len = segment_end - segment_start - chunk_length + 1 + if valid_len <= 0: + continue + + strided_valid_len = (valid_len + sample_stride - 1) // sample_stride + spans.append((episode_id, episode_start + segment_start, strided_valid_len)) + valid_count += strided_valid_len + + return spans, episode_start_rows, valid_count, sample_count, missing_episode_count, fallback_episode_count + + +class EmbodimentCGripperDataset(BaseActionLeRobotDataset): + """Embodiment C Gripper dataset with deferred source registration. + + Sources are registered by ``_register_sources()`` which is called by + ``ActionUnifiedIterableDataset.assign_worker()`` during training, or + explicitly for standalone/eval use. Instruction segments and episode-level + high-level instructions are loaded alongside each source. + + Action layout: + ``[head_cam_delta(9), right_hand_delta(9), right_gripper(1), + left_hand_delta(9), left_gripper(1)]`` → 29 dims for gripper. + + Concat view layout: + ┌──────────────────┐ + │ top_head │ (H, W) + ├─────────┬────────┤ + │ hand_L │ hand_R │ (H/2, W/2) each + └─────────┴────────┘ + """ + + def __init__( + self, + root: str | list[str] | tuple[str, ...] = DEFAULT_OFFSHELF_GRIPPER_ROOT, + fps: float = 10.0, + chunk_length: int = 16, + split_seed: int = 42, + split_val_ratio: float = 0.005, + split: str = "train", + mode: str = "joint", + pose_convention: PoseConvention = "backward_framewise", + embodiment_type: str = "embodiment_c_gripper", + tolerance_s: float = 1e-3, + video_key: str = AGIBOT_GEAR_VIDEO_KEY, + action_normalization: ActionNormalization | None = None, + viewpoint: Viewpoint = "concat_view", + rotation_format: Literal["rot6d"] = "rot6d", + max_loaded_datasets: int = 32, + skip_video_loading: bool = False, + sample_stride: int = 1, + enable_fast_init: bool = False, + fast_init_max_workers: int = 64, + return_agibot_link_poses: bool = False, + ) -> None: + super().__init__( + fps=fps, + chunk_length=chunk_length, + split_seed=split_seed, + split_val_ratio=split_val_ratio, + split=split, + mode=mode, + embodiment_type=embodiment_type, + viewpoint=viewpoint, + pose_convention=pose_convention, + rotation_format=rotation_format, + action_normalization=action_normalization, + tolerance_s=tolerance_s, + max_loaded_datasets=max_loaded_datasets, + skip_video_loading=skip_video_loading, + sample_stride=sample_stride, + enable_fast_init=enable_fast_init, + fast_init_max_workers=fast_init_max_workers, + ) + self._video_key = video_key + self._kind_spec = get_embodiment_c_kind_spec(embodiment_type) + self._is_concat_view = viewpoint == "concat_view" + self._to_opencv: dict[str, np.ndarray] = AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST + self._return_agibot_link_poses: bool = return_agibot_link_poses + + self._all_shard_roots = _resolve_root_paths(root, self._expand_root) + if not self._all_shard_roots: + raise ValueError( + "EmbodimentCGripperDataset: no task directories found under root. " + "Point root to a dir with task_* subdirs, or to a single LeRobot path with meta/info.json." + ) + + frame_ts = [index * self._dt for index in range(0, self._chunk_length + 1)] + self._delta_timestamps = _build_delta_timestamps(self._video_key, self._chunk_length, self._dt) + if self._is_concat_view: + self._delta_timestamps[_HAND_LEFT_KEY] = frame_ts + self._delta_timestamps[_HAND_RIGHT_KEY] = frame_ts + + self._instruction_segments: list[dict[str, list[dict[str, Any]]]] = [] + self._high_level_instructions: list[dict[str, str]] = [] + self._task_root_paths: list[str] = [] + self._episode_start_rows: list[dict[int, int]] = [] + + def _normalizer_filename(self) -> str: + """Resolve Embodiment C gripper stats, sharing them with compatible ext data.""" + + if self._embodiment_type == "embodiment_c_gripper_ext": + return ( + f"{AGIBOT_GEAR_GRIPPER_NORMALIZER_EMBODIMENT_TYPE}_{self._pose_convention}_{self._rotation_format}.json" + ) + return super()._normalizer_filename() + + def _expand_root(self, root: str) -> list[str]: + """Expand a base root path to task-specific subpaths.""" + root_name = Path(root.rstrip("/")).name + if root_name == Path(DEFAULT_OFFSHELF_GRIPPER_ROOT).name: + return [os.path.join(root, sub) for sub in OFFSHELF_GRIPPER_SUBPATHS] + if root_name == Path(DEFAULT_CUSTOM_GRIPPER_ROOT).name: + return [os.path.join(root, sub) for sub in CUSTOM_GRIPPER_SUBPATHS] + return [root] + + def _register_sources(self, indices: list[int] | None = None) -> None: + if indices is None: + indices = list(range(len(self._all_shard_roots))) + for idx in indices: + path = self._all_shard_roots[idx] + instruction_segments = _load_instruction_segments(path) + high_level_instructions = _load_high_level_instructions(path) + self._task_root_paths.append(path) + self._instruction_segments.append(instruction_segments) + self._high_level_instructions.append(high_level_instructions) + try: + self._register_source( + root=path, + delta_timestamps=self._delta_timestamps, + tolerance_s=self._tolerance_s, + dataset_label=Path(path).name, + ) + except Exception as e: + self._task_root_paths.pop() + self._instruction_segments.pop() + self._high_level_instructions.pop() + log.warning(f"EmbodimentCGripperDataset: failed to load LeRobotDatasetMetadata at {path}: {e}") + continue + + def _append_index_records( + self, + *, + meta: LeRobotDatasetMetadata, + ds_idx: int, + dataset_label: str | None = None, + ) -> None: + """Populate index records using only frame starts that stay inside instruction segments.""" + + episode_ids = split_episode_ids( + total_episodes=meta.total_episodes, + seed=self._split_seed, + val_ratio=self._split_val_ratio, + split=self._split, + ) + instruction_segments = self._instruction_segments[ds_idx] + high_level_instructions = self._high_level_instructions[ds_idx] + episode_spans, episode_start_rows, valid_count, sample_count, missing_episode_count, fallback_episode_count = ( + _build_instruction_episode_spans( + episodes=meta.episodes, + episode_ids=episode_ids, + instruction_segments=instruction_segments, + high_level_instructions=high_level_instructions, + chunk_length=self._chunk_length, + sample_stride=self._sample_stride, + ) + ) + + while len(self._episode_start_rows) <= ds_idx: + self._episode_start_rows.append({}) + self._episode_start_rows[ds_idx] = episode_start_rows + + class_name = self.__class__.__name__ + label = f" [{dataset_label}]" if dataset_label else "" + log.info(f"{class_name}{label}: split={self._split}, num episodes={len(episode_ids)}") + if sample_count > 0: + log.info( + f"{class_name}{label}: kept {valid_count} / {sample_count} " + f"({100 * valid_count / sample_count:.2f} %) instruction-filtered samples" + ) + if missing_episode_count > 0: + if fallback_episode_count == missing_episode_count: + log.info( + f"{class_name}{label}: missing instruction_segments for {missing_episode_count} / " + f"{len(episode_ids)} episodes; using high_level_instruction fallback for all of them" + ) + else: + log.warning( + f"{class_name}{label}: missing instruction_segments for {missing_episode_count} / " + f"{len(episode_ids)} episodes; high_level_instruction fallback covered " + f"{fallback_episode_count} of them" + ) + + for episode_id, sample_start, valid_len in episode_spans: + self._episode_records.append((ds_idx, sample_start, valid_len, episode_id)) + self._num_valid_indices += valid_len + self._episode_cum_ends.append(self._num_valid_indices) + + def _get_ai_caption(self, ds_idx: int, episode_id: int, frame_index_in_episode: int) -> str: + segments = self._instruction_segments[ds_idx] + high_level_instructions = self._high_level_instructions[ds_idx] + ep_key = str(episode_id) + for seg in segments.get(ep_key, []): + bounds = _get_instruction_frame_bounds(seg) + if bounds is None: + continue + start, end = bounds + if start <= frame_index_in_episode <= end: + if "instruction" not in seg: + raise ValueError( + f"EmbodimentCGripperDataset: segment for episode_id={episode_id} " + f"frame_index_in_episode={frame_index_in_episode} missing 'instruction' key." + ) + return seg["instruction"] + if ep_key in high_level_instructions: + return high_level_instructions[ep_key] + raise ValueError( + f"EmbodimentCGripperDataset: no instruction annotation found for episode_id={episode_id} " + f"frame_index_in_episode={frame_index_in_episode} (ds_idx={ds_idx})." + ) + + # -- FK-based action construction ---------------------------------------- + + def _build_fk_action(self, sample: dict[str, Any]) -> tuple[torch.Tensor, dict[str, Any]]: + """Build relative FK-pose action plus absolute initial poses for reconstruction. + + Uses ``observation.state`` across the full ``T+1`` conditioning window to + compute absolute FK transforms, converts them to relative ``rot6d`` SE(3) + deltas, then concatenates hand/gripper values from the next observed + state frame for each delta. The source ``action`` column is intentionally + not requested or read: for Embodiment C, the training/viewer action is the + difference between consecutive observed states. + + Returns: + ``(action, extras)`` where ``action`` has shape ``(T, action_dim)`` + and ``extras`` carries the absolute initial poses for the head and + gripper-base wrist trajectories. When ``return_agibot_link_poses`` is + enabled, ``extras`` also carries native URDF link poses for direct + viewer mesh animation. + + Scalar hand actions are emitted in the shared viewer/action convention: + ``0.0`` means closed and ``1.0`` means open. + ``convert_gripper_state_to_open_fraction`` maps observed AgiBot gripper + state, including URDF ``[-pi/4,0]`` radians or actuator-close + ``[0,120]`` degrees, into that same open-fraction range. + """ + obs_state = sample["observation.state"] # [T+1,S] + states_np = obs_state.detach().cpu().numpy().astype(np.float32, copy=False) # [T+1,S] + action_steps = int(states_np.shape[0] - 1) + if action_steps <= 0: + raise ValueError(f"{self.__class__.__name__}: observation.state must contain at least 2 frames.") + link_poses = None + if self._return_agibot_link_poses: + link_poses = compute_link_poses_batch(states_np, self._embodiment_type) # {name:[T+1,4,4]} + native_fk = extract_fk_transforms_from_link_poses(link_poses) # {name:[T+1,4,4]} + else: + native_fk = compute_fk_transforms_batch(states_np, self._embodiment_type) # {name:[T+1,4,4]} + fk = apply_agibot_gripper_to_opencv(native_fk, self._to_opencv) # {name:[T+1,4,4]} + pose_convention = cast(PoseConvention, self._pose_convention) + head_rel = pose_abs_to_rel(fk["head_camera"], rotation_format="rot6d", pose_convention=pose_convention) # [T,9] + right_rel = pose_abs_to_rel( + fk["right_wrist"], rotation_format="rot6d", pose_convention=pose_convention + ) # [T,9] + left_rel = pose_abs_to_rel(fk["left_wrist"], rotation_format="rot6d", pose_convention=pose_convention) # [T,9] + # Normalize observed AgiBot gripper state to the viewer/action + # convention: 0.0=closed, 1.0=open. Actuator-close state values use + # 0=open and 120=closed, so small open-state jitter stays near 1.0 open. + right_state_slice, left_state_slice = _get_gripper_state_slices(self._embodiment_type, self._kind_spec) + right_hand = convert_gripper_state_to_open_fraction(states_np[1:, right_state_slice]) # [T,1] + left_hand = convert_gripper_state_to_open_fraction(states_np[1:, left_state_slice]) # [T,1] + action_np = np.concatenate( + [ + head_rel, # [T,9] + right_rel, # [T,9] + right_hand, # [T,1|6] + left_rel, # [T,9] + left_hand, # [T,1|6] + ], + axis=-1, + ).astype(np.float32, copy=False) # [T,A] + extras = { + "initial_pose": torch.from_numpy(fk["head_camera"][0].copy()).float(), # [4,4] + "initial_pose_right": torch.from_numpy(fk["right_wrist"][0].copy()).float(), # [4,4] + "initial_pose_left": torch.from_numpy(fk["left_wrist"][0].copy()).float(), # [4,4] + } + if link_poses is not None: + agibot_link_poses = { + link_name: torch.from_numpy(poses.copy()).float() for link_name, poses in link_poses.items() + } # {name:[T+1,4,4]} + extras["agibot_link_poses"] = agibot_link_poses + return torch.from_numpy(action_np).float(), extras # [T,A] + + # -- Multi-view composition ---------------------------------------------- + + def _compose_multi_view(self, sample: dict[str, Any]) -> torch.Tensor: + """Compose top-head, left-hand, and right-hand views into a single frame. + + Layout (per frame): + ┌──────────────────┐ + │ top_head │ (H, W) + ├─────────┬────────┤ + │ hand_L │ hand_R │ (H/2, W/2) each + └─────────┴────────┘ + + Left and right hand cameras are downscaled by 2× so they tile to the + same width as the top-head view. Output height is 3H/2. + + Returns: + Composited video tensor in raw LeRobot ``(T, C, H_out, W)`` float format. + """ + top = sample[_TOP_HEAD_KEY] # [T,C,H,W] + left = sample[_HAND_LEFT_KEY] # [T,C,H_l,W_l] + right = sample[_HAND_RIGHT_KEY] # [T,C,H_r,W_r] + + _, _, h_top, w_top = top.shape + half_h, half_w = h_top // 2, w_top // 2 + + left = F.interpolate(left, size=(half_h, half_w), mode="bilinear", align_corners=False) # [T,C,H/2,W/2] + right = F.interpolate(right, size=(half_h, half_w), mode="bilinear", align_corners=False) # [T,C,H/2,W/2] + bottom = torch.cat([left, right], dim=-1) # [T,C,H/2,W] + + composite = torch.cat([top, bottom], dim=-2) # [T,C,3H/2,W] + return composite # [T,C,3H/2,W] + + # -- __getitem__ --------------------------------------------------------- + + def _resolve_sample(self, idx: int) -> tuple[str, int, int, int, dict[str, Any]]: + """Resolve a flat index to one dataset row and its associated metadata.""" + + mode = self._choose_mode() + dataset_idx, row_idx, episode_id, _ = self._resolve_index(int(idx)) + frame_index_in_episode = row_idx - self._episode_start_rows[dataset_idx][episode_id] + sample = self._get_dataset(dataset_idx)[row_idx] + return mode, dataset_idx, episode_id, frame_index_in_episode, sample + + def __getitem__(self, idx: int) -> dict[str, Any]: + mode, dataset_idx, episode_id, frame_index_in_episode, sample = self._resolve_sample(idx) + ai_caption = self._get_ai_caption(dataset_idx, episode_id, frame_index_in_episode) + + # Respect skip_video_loading (used by action-stats / eval scripts that + # only need ``action``). Base class strips ``observation.image*`` from + # delta_timestamps in this mode, so ``sample`` does not contain + # ``self._video_key`` and direct indexing would KeyError. Mirrors the + # droid / robomind pattern. + if self._skip_video_loading: + video = None + # Video: concat_view or single ego_view. + elif self._is_concat_view: + video = self._compose_multi_view(sample) + else: + video = sample[self._video_key] # [T,C,H,W] + + # Action: relative FK-pose based. + action, action_extras = self._build_fk_action(sample) + + extras: dict[str, Any] = {**action_extras} + if self._is_concat_view: + extras["additional_view_description"] = ( + "The top row shows the head-mounted camera view looking down at the workspace. " + "The bottom row contains two horizontally concatenated wrist-mounted camera views: " + "the left hand camera on the left and the right hand camera on the right." + ) + + result = self._build_result(mode=mode, video=video, action=action, ai_caption=ai_caption, **extras) + result["__episode_id__"] = episode_id + result["__task_root__"] = self._task_root_paths[dataset_idx] + return result + + def _build_action_spec(self) -> ActionSpec: + """Embodiment C gripper bimanual layout (29D). + + ``[head_pos+rot6d (9) | right_pos+rot6d (9) | right_gripper (1) + | left_pos+rot6d (9) | left_gripper (1)]`` + + All three SE(3) blocks (head camera + both wrists) participate in + idle-frame detection: a chunk only counts as idle if the head is + steady AND both arms are at rest AND both grippers are unchanged. + Override this method (or use ``Reserved`` for head dims) if you want + head motion to be ignored by idle detection. + """ + return build_action_spec( + Pos(prefix="head"), + Rot("rot6d", prefix="head"), + Pos(prefix="right"), + Rot("rot6d", prefix="right"), + Gripper(prefix="right"), + Pos(prefix="left"), + Rot("rot6d", prefix="left"), + Gripper(prefix="left"), + ) + + @property + def action_dim(self) -> int: + return 29 + + +class EmbodimentCGripperExtDataset(EmbodimentCGripperDataset): + """Embodiment C Gripper Extended dataset for custom tasks. + + The source episodes use a different state layout (``state=[94]``) compared + to the standard gripper (``state=[32]``). Source action columns are not used; + emitted FK-pose actions are derived from consecutive observed states. + + Key layout differences handled here: + + **State (94-dim):** + - Arm joint positions at ``[54:68]`` (vs ``[0:14]`` in standard). + - Head yaw/pitch at ``[82:84]`` (vs ``[16]``/``[17]``). + - Waist pitch/lift at ``[84:86]`` (vs ``[18]``/``[19]``). + - Mobile-base position/quaternion at ``[86:93]``. + + The FK engine (``_extract_joint_values_from_state``) handles the state + remapping for the ``embodiment_c_gripper_ext`` embodiment type and folds + mobile-base motion into the head/gripper-base poses. The emitted FK-pose + action is the same 29-dim layout as standard gripper: + ``[head(9), right(9), right_grip(1), left(9), left_grip(1)]``. + """ + + def __init__( + self, + root: str | list[str] | tuple[str, ...] = DEFAULT_CUSTOM_GRIPPER_EXT_ROOT, + embodiment_type: str = "embodiment_c_gripper_ext", + **kwargs: Any, + ) -> None: + super().__init__(root=root, embodiment_type=embodiment_type, **kwargs) diff --git a/cosmos_framework/data/vfm/action/embodiment_c_dataset_config.py b/cosmos_framework/data/vfm/action/embodiment_c_dataset_config.py new file mode 100644 index 0000000..f366ec2 --- /dev/null +++ b/cosmos_framework/data/vfm/action/embodiment_c_dataset_config.py @@ -0,0 +1,153 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Default data roots for Embodiment C datasets.""" + +from __future__ import annotations + +DEFAULT_OFFSHELF_GRIPPER_ROOT = "" +DEFAULT_CUSTOM_GRIPPER_ROOT = "" + +OFFSHELF_GRIPPER_SUBPATHS = ( + # 20251016_500h/gripper + "20251016_500h/gripper/task_1018", + "20251016_500h/gripper/task_1173", + "20251016_500h/gripper/task_1174", + "20251016_500h/gripper/task_1183", + "20251016_500h/gripper/task_1185", + "20251016_500h/gripper/task_1187", + "20251016_500h/gripper/task_1189", + "20251016_500h/gripper/task_1237", + "20251016_500h/gripper/task_1245", + "20251016_500h/gripper/task_1251", + "20251016_500h/gripper/task_1252", + "20251016_500h/gripper/task_1259", + "20251016_500h/gripper/task_1286", + "20251016_500h/gripper/task_797", + "20251016_500h/gripper/task_798", + "20251016_500h/gripper/task_799", + "20251016_500h/gripper/task_800", + "20251016_500h/gripper/task_809", + "20251016_500h/gripper/task_812", + "20251016_500h/gripper/task_813", + "20251016_500h/gripper/task_815", + "20251016_500h/gripper/task_827", + "20251016_500h/gripper/task_828", + "20251016_500h/gripper/task_841", + "20251016_500h/gripper/task_857", + "20251016_500h/gripper/task_869", + "20251016_500h/gripper/task_876", + "20251016_500h/gripper/task_893", + "20251016_500h/gripper/task_901", + "20251016_500h/gripper/task_903", + "20251016_500h/gripper/task_954", + "20251016_500h/gripper/task_957", + "20251016_500h/gripper/task_964", + "20251016_500h/gripper/task_968", + # 20251031_500h/gripper + "20251031_500h/gripper/task_1018", + "20251031_500h/gripper/task_1174", + "20251031_500h/gripper/task_1183", + "20251031_500h/gripper/task_1185", + "20251031_500h/gripper/task_1237", + "20251031_500h/gripper/task_1251", + "20251031_500h/gripper/task_1286", + "20251031_500h/gripper/task_1288", + "20251031_500h/gripper/task_1292", + "20251031_500h/gripper/task_1293", + "20251031_500h/gripper/task_1296", + "20251031_500h/gripper/task_1298", + "20251031_500h/gripper/task_1299", + "20251031_500h/gripper/task_1307", + "20251031_500h/gripper/task_1310", + "20251031_500h/gripper/task_1313", + "20251031_500h/gripper/task_1314", + "20251031_500h/gripper/task_1326", + "20251031_500h/gripper/task_1331", + "20251031_500h/gripper/task_1366", + "20251031_500h/gripper/task_1368", + "20251031_500h/gripper/task_1370", + "20251031_500h/gripper/task_1371", + "20251031_500h/gripper/task_1372", + "20251031_500h/gripper/task_1389", + "20251031_500h/gripper/task_1390", + "20251031_500h/gripper/task_1405", + "20251031_500h/gripper/task_1425", + "20251031_500h/gripper/task_1430", + "20251031_500h/gripper/task_1432", + "20251031_500h/gripper/task_1439", + "20251031_500h/gripper/task_1448", + "20251031_500h/gripper/task_1450", + "20251031_500h/gripper/task_1452", + "20251031_500h/gripper/task_1453", + "20251031_500h/gripper/task_1479", + "20251031_500h/gripper/task_1487", + "20251031_500h/gripper/task_1491", +) + +# Custom gripper tasks with extended 94-dim state. These require +# EmbodimentCGripperExtDataset because their arm, head, waist, and gripper state +# offsets differ from the standard gripper layout. +DEFAULT_CUSTOM_GRIPPER_EXT_ROOT = ( + # 20251113_330h + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251113_330h/2025110304/gripper/task_3578", + # 20251218_561h + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025120301/gripper/task_3529", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025120301/gripper/task_3545", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025120301/gripper/task_3547", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025120401/gripper/task_3529", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025120401/gripper/task_3545", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025120501/gripper/task_3529", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025120501/gripper/task_3545", +) + +_CUSTOM_GRIPPER_TASK_ROOTS = ( + # 20251113_330h + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251113_330h/2025111101/gripper/task_2156", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251113_330h/2025111101/gripper/task_3578", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251113_330h/2025111102/gripper/task_2156", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251113_330h/2025111102/gripper/task_3578", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251113_330h/2025111104/gripper/task_3800", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251113_330h/2025111201/gripper/task_4102", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251113_330h/2025111202/gripper/task_4102", + # 20251218_561h + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025120301/gripper/task_3719", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025120301/gripper/task_3800", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025120301/gripper/task_4392", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025120401/gripper/task_3800", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025120401/gripper/task_4392", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025120505/gripper/task_3800", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025120505/gripper/task_4392", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025120801/gripper/task_3798", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025120901/gripper/task_3798", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025121001/gripper/task_3798", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025121101/gripper/task_3798", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20251218_561h/2025121201/gripper/task_3798", + # 20260205_260h + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_1183", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_1185", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_1187", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_797", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_798", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_799", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_809", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_812", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_813", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_815", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_827", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_828", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_841", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_857", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_869", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_876", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_893", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_901", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_954", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_957", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_964", + f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/20260205_260h/task_968", +) + +CUSTOM_GRIPPER_SUBPATHS = tuple( + root.removeprefix(f"{DEFAULT_CUSTOM_GRIPPER_ROOT}/") for root in _CUSTOM_GRIPPER_TASK_ROOTS +) diff --git a/cosmos_framework/data/vfm/action/embodiment_c_fk.py b/cosmos_framework/data/vfm/action/embodiment_c_fk.py new file mode 100644 index 0000000..6afff3e --- /dev/null +++ b/cosmos_framework/data/vfm/action/embodiment_c_fk.py @@ -0,0 +1,499 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Lightweight Embodiment C forward kinematics for datasets and viewers.""" + +from __future__ import annotations + +import xml.etree.ElementTree as ET +from dataclasses import dataclass +from functools import lru_cache + +import numpy as np + +from cosmos_framework.data.vfm.action.embodiment_c_spec import ( + AGIBOT_GEAR_ARM_JOINT_NAMES_LEFT, + AGIBOT_GEAR_ARM_JOINT_NAMES_RIGHT, + AGIBOT_GEAR_ARM_STATE_SLICE, + AGIBOT_GEAR_EXT_ARM_STATE_SLICE, + AGIBOT_GEAR_EXT_STATE_HEAD_PITCH_IDX, + AGIBOT_GEAR_EXT_STATE_HEAD_YAW_IDX, + AGIBOT_GEAR_EXT_STATE_LEFT_HAND_SLICE, + AGIBOT_GEAR_EXT_STATE_RIGHT_HAND_SLICE, + AGIBOT_GEAR_EXT_STATE_ROBOT_ORIENTATION_SLICE, + AGIBOT_GEAR_EXT_STATE_ROBOT_POSITION_SLICE, + AGIBOT_GEAR_EXT_STATE_WAIST_LIFT_IDX, + AGIBOT_GEAR_EXT_STATE_WAIST_PITCH_IDX, + AGIBOT_GEAR_GRIPPER_OPEN_ACTUATOR_DEG, + AGIBOT_GEAR_GRIPPER_OPEN_ANGLE_RAD, + AGIBOT_GEAR_HEAD_CAMERA_LINK_NAME, + AGIBOT_GEAR_HEAD_PITCH_JOINT_NAME, + AGIBOT_GEAR_HEAD_YAW_JOINT_NAME, + AGIBOT_GEAR_LEFT_EE_LINK_NAME, + AGIBOT_GEAR_LEFT_GRIPPER_JOINT_MIMICS, + AGIBOT_GEAR_RIGHT_EE_LINK_NAME, + AGIBOT_GEAR_RIGHT_GRIPPER_JOINT_MIMICS, + AGIBOT_GEAR_STATE_HEAD_PITCH_IDX, + AGIBOT_GEAR_STATE_HEAD_YAW_IDX, + AGIBOT_GEAR_STATE_WAIST_LIFT_IDX, + AGIBOT_GEAR_STATE_WAIST_PITCH_IDX, + AGIBOT_GEAR_WAIST_LIFT_JOINT_NAME, + AGIBOT_GEAR_WAIST_PITCH_JOINT_NAME, + get_embodiment_c_embodiment_spec, + get_embodiment_c_kind_spec, + get_embodiment_c_urdf_path, +) +from cosmos_framework.data.vfm.action.pose_utils import convert_rotation + +# Offset from gripper base/flange to the center point between the finger pads. +DEFAULT_GRIPPER_LENGTH_M = 0.14308 +_GRIPPER_VALUE_EPS = 1e-4 +_QUATERNION_NORM_EPS = 1e-8 +_GRIPPER_ACTUATOR_OVERSHOOT_DEG = 5.0 +# Main-branch wrist rotations composed with one extra local-Z 180 degree rotation. +AGIBOT_GEAR_LEFT_GRIPPER_TO_OPENCV: np.ndarray = np.asarray( + [ + [0.0, 1.0, 0.0], + [-1.0, 0.0, 0.0], + [0.0, 0.0, 1.0], + ], + dtype=np.float32, +) +AGIBOT_GEAR_RIGHT_GRIPPER_TO_OPENCV: np.ndarray = np.asarray( + [ + [0.0, -1.0, 0.0], + [1.0, 0.0, 0.0], + [0.0, 0.0, 1.0], + ], + dtype=np.float32, +) +AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST: dict[str, np.ndarray] = { + "right_wrist": AGIBOT_GEAR_RIGHT_GRIPPER_TO_OPENCV, + "left_wrist": AGIBOT_GEAR_LEFT_GRIPPER_TO_OPENCV, +} + + +@dataclass(frozen=True) +class AgibotGearViewerBatch: + """Absolute poses and auxiliary gripper values aligned with action steps.""" + + head_camera_poses: np.ndarray + right_wrist_poses: np.ndarray + left_wrist_poses: np.ndarray + right_gripper: np.ndarray | None + left_gripper: np.ndarray | None + + +def _scale_to_unit_interval(values: np.ndarray, scale: float) -> np.ndarray: + """Scale non-negative gripper actuator values to ``[0,1]``.""" + + return np.clip(values / scale, 0.0, 1.0).astype(np.float32, copy=False) + + +def _scale_negative_to_unit_interval(values: np.ndarray, scale: float) -> np.ndarray: + """Scale URDF-style negative gripper angles to ``[0,1]`` open fractions.""" + + return np.clip(-values / scale, 0.0, 1.0).astype(np.float32, copy=False) + + +def _to_numpy_float32(array: object) -> np.ndarray: + """Convert a tensor-like object to a float32 NumPy array.""" + + try: + import torch + + if isinstance(array, torch.Tensor): + return array.detach().cpu().numpy().astype(np.float32, copy=False) + except ImportError: + pass + return np.asarray(array, dtype=np.float32) + + +def _normalize_quaternions_xyzw(quaternions: np.ndarray) -> np.ndarray: + """Normalize ``xyzw`` quaternions, treating all-zero rows as identity.""" + + normalized = np.asarray(quaternions, dtype=np.float32).copy() # [T,4] + norms = np.linalg.norm(normalized, axis=-1, keepdims=True) # [T,1] + valid = norms[:, 0] >= _QUATERNION_NORM_EPS # [T] + normalized[valid] = normalized[valid] / norms[valid] # [T_valid,4] + normalized[~valid] = np.asarray([0.0, 0.0, 0.0, 1.0], dtype=np.float32) # [T_invalid,4] + return normalized + + +def _quat_xyzw_to_rotation_matrix(quaternions: np.ndarray) -> np.ndarray: + """Convert ``xyzw`` quaternions to rotation matrices.""" + + normalized = _normalize_quaternions_xyzw(quaternions) # [T,4] + rotations = convert_rotation( + normalized, + input_format="quat_xyzw", + output_format="matrix", + normalize_matrix=True, + ) + return np.asarray(rotations, dtype=np.float32) + + +def build_robot_base_transforms(positions: np.ndarray, quaternions: np.ndarray) -> np.ndarray: + """Build robot-base poses from position and ``xyzw`` quaternion arrays.""" + + positions = np.asarray(positions, dtype=np.float32) # [T,3] + quaternions = np.asarray(quaternions, dtype=np.float32) # [T,4] + if positions.ndim != 2 or positions.shape[1] != 3: + raise ValueError(f"robot base positions must have shape [T,3], got {positions.shape}.") + if quaternions.ndim != 2 or quaternions.shape[1] != 4: + raise ValueError(f"robot base quaternions must have shape [T,4], got {quaternions.shape}.") + if positions.shape[0] != quaternions.shape[0]: + raise ValueError( + f"robot base positions/quaternions must share T, got {positions.shape[0]} and {quaternions.shape[0]}." + ) + + transforms = np.tile(np.eye(4, dtype=np.float32), (positions.shape[0], 1, 1)) # [T,4,4] + transforms[:, :3, :3] = _quat_xyzw_to_rotation_matrix(quaternions) # [T,3,3] + transforms[:, :3, 3] = positions # [T,3] + return transforms + + +def _invert_rigid_transform(transform: np.ndarray) -> np.ndarray: + """Invert one homogeneous rigid transform.""" + + inverse = np.eye(4, dtype=np.float32) # [4,4] + rotation_t = transform[:3, :3].T.astype(np.float32, copy=False) # [3,3] + inverse[:3, :3] = rotation_t + inverse[:3, 3] = -(rotation_t @ transform[:3, 3]) # [3] + return inverse + + +def apply_robot_base_motion_to_poses( + poses_by_name: dict[str, np.ndarray], + positions: np.ndarray, + quaternions: np.ndarray, +) -> dict[str, np.ndarray]: + """Apply mobile-base motion to FK poses, normalized to the first frame.""" + + base_poses = build_robot_base_transforms(positions, quaternions) # [T,4,4] + initial_base_inv = _invert_rigid_transform(base_poses[0]) # [4,4] + base_motion = np.einsum("ij,tjk->tik", initial_base_inv, base_poses).astype(np.float32, copy=False) # [T,4,4] + return { + name: np.einsum("tij,tjk->tik", base_motion, poses).astype(np.float32, copy=False) # [T,4,4] + for name, poses in poses_by_name.items() + } + + +def _apply_ext_base_motion_to_poses( + poses_by_name: dict[str, np.ndarray], + states: np.ndarray, + embodiment_type: str, +) -> dict[str, np.ndarray]: + """Apply ext mobile-base motion to FK poses, normalized to the first frame.""" + + if embodiment_type != "embodiment_c_gripper_ext": + return poses_by_name + if states.shape[1] < AGIBOT_GEAR_EXT_STATE_ROBOT_ORIENTATION_SLICE.stop: + raise ValueError( + f"embodiment_c_gripper_ext state must include robot pose through index " + f"{AGIBOT_GEAR_EXT_STATE_ROBOT_ORIENTATION_SLICE.stop - 1}, got shape {states.shape}." + ) + + positions = states[:, AGIBOT_GEAR_EXT_STATE_ROBOT_POSITION_SLICE].astype(np.float32, copy=False) # [T,3] + quaternions = states[:, AGIBOT_GEAR_EXT_STATE_ROBOT_ORIENTATION_SLICE].astype(np.float32, copy=False) # [T,4] + return apply_robot_base_motion_to_poses(poses_by_name, positions, quaternions) + + +def apply_agibot_gripper_to_opencv( + poses_by_name: dict[str, np.ndarray], + to_opencv_by_wrist: dict[str, np.ndarray], +) -> dict[str, np.ndarray]: + """Post-rotate AgiBot gripper wrist poses into OpenCV convention.""" + + aligned = {name: poses.astype(np.float32, copy=True) for name, poses in poses_by_name.items()} # {name:[...,4,4]} + for wrist_name, wrist_to_opencv in to_opencv_by_wrist.items(): + poses = aligned.get(wrist_name) + if poses is None: + continue + aligned[wrist_name][..., :3, :3] = poses[..., :3, :3] @ wrist_to_opencv.astype(poses.dtype) # [...,3,3] + return aligned + + +def _get_embodiment_c_mujoco_kinematics_xml() -> str: + """Build a MuJoCo-loadable kinematics-only XML string from the committed URDF.""" + + root = ET.parse(get_embodiment_c_urdf_path()).getroot() + mujoco_element = root.find("mujoco") + if mujoco_element is None: + mujoco_element = ET.Element("mujoco") + root.insert(0, mujoco_element) + compiler_element = mujoco_element.find("compiler") + if compiler_element is None: + compiler_element = ET.SubElement(mujoco_element, "compiler") + compiler_element.attrib["fusestatic"] = "false" + + for link_element in root.findall("link"): + for child_element in list(link_element): + if child_element.tag in {"visual", "collision"}: + link_element.remove(child_element) + + return ET.tostring(root, encoding="unicode") + + +class _MujocoFk: + """MuJoCo-backed FK engine for the committed AgiBot G1 omnipicker URDF.""" + + def __init__(self) -> None: + import mujoco + + self._mujoco = mujoco + self.model = mujoco.MjModel.from_xml_string(_get_embodiment_c_mujoco_kinematics_xml()) + self.data = mujoco.MjData(self.model) + self._joint_qpos_addresses: dict[str, int] = {} + for joint_id in range(self.model.njnt): + joint_name = mujoco.mj_id2name(self.model, mujoco.mjtObj.mjOBJ_JOINT, joint_id) + if joint_name is not None: + self._joint_qpos_addresses[joint_name] = int(self.model.jnt_qposadr[joint_id]) + + def link_poses(self, joint_values: dict[str, float]) -> dict[str, np.ndarray]: + """Return world transforms for every named body in the MuJoCo model.""" + + self.data.qpos[:] = 0.0 + for joint_name, joint_value in joint_values.items(): + qpos_address = self._joint_qpos_addresses.get(joint_name) + if qpos_address is not None: + self.data.qpos[qpos_address] = float(joint_value) + self._mujoco.mj_forward(self.model, self.data) + + poses: dict[str, np.ndarray] = {} + for body_id in range(1, self.model.nbody): + body_name = self._mujoco.mj_id2name(self.model, self._mujoco.mjtObj.mjOBJ_BODY, body_id) + if body_name is None: + continue + transform = np.eye(4, dtype=np.float32) + transform[:3, :3] = self.data.xmat[body_id].reshape(3, 3).astype(np.float32, copy=False) + transform[:3, 3] = self.data.xpos[body_id].astype(np.float32, copy=False) + poses[body_name] = transform + return poses + + +@lru_cache(maxsize=1) +def _get_fk_engine() -> _MujocoFk: + """Return a cached MuJoCo FK engine for the committed AgiBot URDF.""" + + return _MujocoFk() + + +def _extract_joint_values_from_state(state: np.ndarray, embodiment_type: str) -> dict[str, float]: + """Map one observation.state vector to the URDF joint names used for FK.""" + + if embodiment_type == "embodiment_c_gripper_ext": + # Ext layout: 94-dim state with joints at different offsets. + arm_state = state[AGIBOT_GEAR_EXT_ARM_STATE_SLICE] + head_yaw = float(state[AGIBOT_GEAR_EXT_STATE_HEAD_YAW_IDX]) + head_pitch = float(state[AGIBOT_GEAR_EXT_STATE_HEAD_PITCH_IDX]) + waist_lift = float(state[AGIBOT_GEAR_EXT_STATE_WAIST_LIFT_IDX]) + waist_pitch = float(state[AGIBOT_GEAR_EXT_STATE_WAIST_PITCH_IDX]) + else: + arm_state = state[AGIBOT_GEAR_ARM_STATE_SLICE] + head_yaw = float(state[AGIBOT_GEAR_STATE_HEAD_YAW_IDX]) + head_pitch = float(state[AGIBOT_GEAR_STATE_HEAD_PITCH_IDX]) + waist_pitch = float(state[AGIBOT_GEAR_STATE_WAIST_PITCH_IDX]) + waist_lift = float(state[AGIBOT_GEAR_STATE_WAIST_LIFT_IDX]) + + joint_values = { + AGIBOT_GEAR_WAIST_LIFT_JOINT_NAME: float(waist_lift), + AGIBOT_GEAR_WAIST_PITCH_JOINT_NAME: float(waist_pitch), + AGIBOT_GEAR_HEAD_YAW_JOINT_NAME: float(head_yaw), + AGIBOT_GEAR_HEAD_PITCH_JOINT_NAME: float(head_pitch), + } + joint_values.update({name: float(arm_state[idx]) for idx, name in enumerate(AGIBOT_GEAR_ARM_JOINT_NAMES_LEFT)}) + joint_values.update({name: float(arm_state[7 + idx]) for idx, name in enumerate(AGIBOT_GEAR_ARM_JOINT_NAMES_RIGHT)}) + _set_gripper_joint_values_from_state(joint_values, state, embodiment_type) + return joint_values + + +def _set_gripper_joint_values_from_state( + joint_values: dict[str, float], + state: np.ndarray, + embodiment_type: str, +) -> None: + """Map observed scalar gripper state into all omnipicker finger joints.""" + + embodiment_spec = get_embodiment_c_embodiment_spec(embodiment_type) + if embodiment_spec.kind != "gripper": + return + + if embodiment_type == "embodiment_c_gripper_ext": + left_raw = float(state[AGIBOT_GEAR_EXT_STATE_LEFT_HAND_SLICE][0]) + right_raw = float(state[AGIBOT_GEAR_EXT_STATE_RIGHT_HAND_SLICE][0]) + else: + kind_spec = get_embodiment_c_kind_spec(embodiment_type) + state_hand_slice = kind_spec.state_hand_slice + left_raw = float(state[state_hand_slice.start]) + right_raw = float(state[state_hand_slice.start + 1]) + + left_open = float(convert_gripper_state_to_open_fraction(np.asarray([left_raw], dtype=np.float32))[0]) # [1] + right_open = float(convert_gripper_state_to_open_fraction(np.asarray([right_raw], dtype=np.float32))[0]) # [1] + for opening, joint_mimics in ( + (left_open, AGIBOT_GEAR_LEFT_GRIPPER_JOINT_MIMICS), + (right_open, AGIBOT_GEAR_RIGHT_GRIPPER_JOINT_MIMICS), + ): + primary_angle = -float(np.clip(opening, 0.0, 1.0)) * AGIBOT_GEAR_GRIPPER_OPEN_ANGLE_RAD + for joint_name, multiplier, offset in joint_mimics: + joint_values[joint_name] = multiplier * primary_angle + offset + + +def compute_fk_transforms( + state: np.ndarray, + embodiment_type: str, +) -> dict[str, np.ndarray]: + """Compute native-frame calibrated head-camera and gripper-base transforms for one state.""" + + fk_engine = _get_fk_engine() + link_poses = fk_engine.link_poses(_extract_joint_values_from_state(state, embodiment_type)) + + return { + "head_camera": link_poses[AGIBOT_GEAR_HEAD_CAMERA_LINK_NAME].astype(np.float32, copy=False), + "right_wrist": link_poses[AGIBOT_GEAR_RIGHT_EE_LINK_NAME].astype(np.float32, copy=False), + "left_wrist": link_poses[AGIBOT_GEAR_LEFT_EE_LINK_NAME].astype(np.float32, copy=False), + } + + +def extract_fk_transforms_from_link_poses(link_poses: dict[str, np.ndarray]) -> dict[str, np.ndarray]: + """Extract calibrated head-camera and gripper-base transforms from URDF link poses.""" + + return { + "head_camera": link_poses[AGIBOT_GEAR_HEAD_CAMERA_LINK_NAME].astype(np.float32, copy=False), + "right_wrist": link_poses[AGIBOT_GEAR_RIGHT_EE_LINK_NAME].astype(np.float32, copy=False), + "left_wrist": link_poses[AGIBOT_GEAR_LEFT_EE_LINK_NAME].astype(np.float32, copy=False), + } + + +def compute_fk_transforms_batch( + states: np.ndarray, + embodiment_type: str, +) -> dict[str, np.ndarray]: + """Compute absolute transforms for a batch of AgiBot observation states.""" + + num_steps = int(states.shape[0]) + head_camera = np.empty((num_steps, 4, 4), dtype=np.float32) + right_wrist = np.empty((num_steps, 4, 4), dtype=np.float32) + left_wrist = np.empty((num_steps, 4, 4), dtype=np.float32) + + for step in range(num_steps): + transforms = compute_fk_transforms(states[step], embodiment_type) + head_camera[step] = transforms["head_camera"] + right_wrist[step] = transforms["right_wrist"] + left_wrist[step] = transforms["left_wrist"] + + transforms_by_name = { + "head_camera": head_camera, + "right_wrist": right_wrist, + "left_wrist": left_wrist, + } + return _apply_ext_base_motion_to_poses(transforms_by_name, states, embodiment_type) + + +def compute_link_poses_batch(states: np.ndarray, embodiment_type: str) -> dict[str, np.ndarray]: + """Compute absolute URDF link poses for a batch of AgiBot observation states.""" + + num_steps = int(states.shape[0]) + if num_steps == 0: + return {} + + fk_engine = _get_fk_engine() + first_link_poses = fk_engine.link_poses(_extract_joint_values_from_state(states[0], embodiment_type)) + batched_link_poses = {link_name: np.empty((num_steps, 4, 4), dtype=np.float32) for link_name in first_link_poses} + for link_name, pose in first_link_poses.items(): + batched_link_poses[link_name][0] = pose.astype(np.float32, copy=False) + + for step in range(1, num_steps): + link_poses = fk_engine.link_poses(_extract_joint_values_from_state(states[step], embodiment_type)) + for link_name, pose in link_poses.items(): + batched_link_poses[link_name][step] = pose.astype(np.float32, copy=False) + + return _apply_ext_base_motion_to_poses(batched_link_poses, states, embodiment_type) + + +def convert_gripper_state_to_open_fraction(values: np.ndarray) -> np.ndarray: + """Convert observed AgiBot gripper state to viewer/dataset open fractions. + + The shared viewer/action convention is ``0=closed`` and ``1=open``. + Observed AgiBot gripper state uses actuator-close angle units: ``0`` is + open and ``120`` is closed. Some episodes contain small closed-state + overshoot above ``120``; those values are accepted and clipped to fully + closed. Small open-state sensor jitter such as ``0.217`` must therefore + remain nearly fully open, not be interpreted as a normalized close fraction. + """ + + values = np.asarray(values, dtype=np.float32) + if values.size == 0: + return values + if not np.isfinite(values).all(): + raise ValueError("AgiBot gripper values contain NaN or Inf values.") + + min_value = float(np.min(values)) + max_value = float(np.max(values)) + if ( + min_value < -_GRIPPER_VALUE_EPS + and min_value >= -AGIBOT_GEAR_GRIPPER_OPEN_ANGLE_RAD - _GRIPPER_VALUE_EPS + and max_value <= _GRIPPER_VALUE_EPS + ): + return _scale_negative_to_unit_interval(values, AGIBOT_GEAR_GRIPPER_OPEN_ANGLE_RAD) + if ( + min_value < -_GRIPPER_VALUE_EPS + and min_value >= -np.degrees(AGIBOT_GEAR_GRIPPER_OPEN_ANGLE_RAD) - _GRIPPER_VALUE_EPS + and max_value <= _GRIPPER_VALUE_EPS + ): + return _scale_negative_to_unit_interval(values, np.degrees(AGIBOT_GEAR_GRIPPER_OPEN_ANGLE_RAD)) + max_actuator_value = AGIBOT_GEAR_GRIPPER_OPEN_ACTUATOR_DEG + _GRIPPER_ACTUATOR_OVERSHOOT_DEG + if min_value >= -_GRIPPER_VALUE_EPS and max_value <= max_actuator_value + _GRIPPER_VALUE_EPS: + close_fraction = _scale_to_unit_interval(values, AGIBOT_GEAR_GRIPPER_OPEN_ACTUATOR_DEG) # [*] + return (1.0 - close_fraction).astype(np.float32, copy=False) # [*] + + raise ValueError( + f"Unsupported AgiBot gripper value range; min={min_value:.4f}, max={max_value:.4f}. " + f"Expected URDF angle [-pi/4,0] or actuator-close degrees [0,{max_actuator_value:.1f}] " + f"(values above {AGIBOT_GEAR_GRIPPER_OPEN_ACTUATOR_DEG:.1f} are clipped closed)." + ) + + +def extract_gripper_states(states: np.ndarray, embodiment_type: str) -> tuple[np.ndarray | None, np.ndarray | None]: + """Extract left/right scalar gripper positions from observed AgiBot states.""" + + embodiment_spec = get_embodiment_c_embodiment_spec(embodiment_type) + if embodiment_spec.kind != "gripper": + return None, None + + if embodiment_type == "embodiment_c_gripper_ext": + right = states[:, AGIBOT_GEAR_EXT_STATE_RIGHT_HAND_SLICE].reshape(states.shape[0], -1) # [T,1] + left = states[:, AGIBOT_GEAR_EXT_STATE_LEFT_HAND_SLICE].reshape(states.shape[0], -1) # [T,1] + else: + kind_spec = get_embodiment_c_kind_spec(embodiment_type) + state_hand_slice = kind_spec.state_hand_slice + right = states[:, state_hand_slice.start + 1 : state_hand_slice.start + 2].reshape(states.shape[0], -1) # [T,1] + left = states[:, state_hand_slice.start : state_hand_slice.start + 1].reshape(states.shape[0], -1) # [T,1] + + right_gripper = convert_gripper_state_to_open_fraction(right[:, 0]) # [T] + left_gripper = convert_gripper_state_to_open_fraction(left[:, 0]) # [T] + return right_gripper, left_gripper + + +def build_viewer_batch(sample: dict[str, object], embodiment_type: str) -> AgibotGearViewerBatch: + """Build viewer-ready absolute poses from one AgiBot dataset sample.""" + + states = _to_numpy_float32(sample["observation.state"]) # [T+1,S] + num_steps = int(states.shape[0] - 1) + if num_steps <= 0: + raise ValueError("AgiBot viewer batch requires observation.state to contain at least 2 frames.") + state_steps = states[1:] # [T,S] + + native_transforms = compute_fk_transforms_batch(state_steps, embodiment_type) # {name:[T,4,4]} + transforms = apply_agibot_gripper_to_opencv( + native_transforms, + AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST, + ) # {name:[T,4,4]} + right_gripper, left_gripper = extract_gripper_states(state_steps, embodiment_type) + return AgibotGearViewerBatch( + head_camera_poses=transforms["head_camera"], + right_wrist_poses=transforms["right_wrist"], + left_wrist_poses=transforms["left_wrist"], + right_gripper=right_gripper, + left_gripper=left_gripper, + ) diff --git a/cosmos_framework/data/vfm/action/embodiment_c_fk_test.py b/cosmos_framework/data/vfm/action/embodiment_c_fk_test.py new file mode 100644 index 0000000..b2dbdd3 --- /dev/null +++ b/cosmos_framework/data/vfm/action/embodiment_c_fk_test.py @@ -0,0 +1,467 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from __future__ import annotations + +import importlib.util +from pathlib import Path + +import pytest +import torch + +from cosmos_framework.data.vfm.action.embodiment_c_dataset import EmbodimentCGripperDataset +from cosmos_framework.data.vfm.action.embodiment_c_fk import ( + AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST, + build_viewer_batch, +) +from cosmos_framework.data.vfm.action.embodiment_c_spec import ( + AGIBOT_GEAR_EXT_STATE_ROBOT_ORIENTATION_SLICE, + AGIBOT_GEAR_EXT_STATE_ROBOT_POSITION_SLICE, + AGIBOT_GEAR_GRIPPER_OPEN_ANGLE_RAD, + AGIBOT_GEAR_HEAD_CAMERA_LINK_NAME, + AGIBOT_GEAR_LEFT_EE_LINK_NAME, + AGIBOT_GEAR_LEFT_GRIPPER_CENTER_LINK_NAME, + AGIBOT_GEAR_RIGHT_EE_LINK_NAME, + get_embodiment_c_urdf_path, +) +from cosmos_framework.data.vfm.action.agibotworld_beta_dataset import AgiBotWorldBetaDataset +from cosmos_framework.data.vfm.action.domain_utils import get_domain_id +from cosmos_framework.data.vfm.action.urdf_visualizer.viewer import DATASETS + +_REPO_ROOT = Path(__file__).resolve().parents[5] +_LOCAL_GRIPPER_SAMPLE_ROOT = _REPO_ROOT / "dev" / "embodiment_c_samples" / "gripper" / "task_sample_gripper" +_REQUIRES_MUJOCO = pytest.mark.skipif( + importlib.util.find_spec("mujoco") is None, + reason="requires mujoco until CI docker images include it", +) + + +def _make_sample(state_dim: int) -> dict[str, torch.Tensor]: + state = torch.zeros((3, state_dim), dtype=torch.float32) # [T+1,S] + return { + "observation.state": state, + } + + +def _make_beta_robot_pose(num_steps: int) -> dict[str, torch.Tensor]: + robot_quat = torch.zeros((num_steps, 4), dtype=torch.float32) # [T,4] + robot_quat[:, 3] = 1.0 + return { + "observation.states.robot.position": torch.zeros((num_steps, 3), dtype=torch.float32), # [T,3] + "observation.states.robot.orientation": robot_quat, # [T,4] + } + + +def test_get_embodiment_c_urdf_path_exists() -> None: + assert get_embodiment_c_urdf_path().name == "G1_omnipicker_calibrated.urdf" + assert get_embodiment_c_urdf_path().parent.name == "urdf_visualizer" + assert get_embodiment_c_urdf_path().is_file() + + +@_REQUIRES_MUJOCO +def test_fk_uses_calibrated_head_camera_link() -> None: + import numpy as np + + from cosmos_framework.data.vfm.action.embodiment_c_fk import compute_fk_transforms, compute_link_poses_batch + + state = np.zeros(32, dtype=np.float32) + fk = compute_fk_transforms(state, "embodiment_c_gripper") + link_poses = compute_link_poses_batch(state[None, :], "embodiment_c_gripper") + + np.testing.assert_allclose(fk["head_camera"], link_poses[AGIBOT_GEAR_HEAD_CAMERA_LINK_NAME][0], atol=1e-6) + + +@_REQUIRES_MUJOCO +def test_fk_uses_gripper_base_not_center_link() -> None: + import numpy as np + + from cosmos_framework.data.vfm.action.embodiment_c_fk import compute_fk_transforms, compute_link_poses_batch + + state = np.zeros(32, dtype=np.float32) + fk = compute_fk_transforms(state, "embodiment_c_gripper") + link_poses = compute_link_poses_batch(state[None, :], "embodiment_c_gripper") + + np.testing.assert_allclose(fk["left_wrist"], link_poses[AGIBOT_GEAR_LEFT_EE_LINK_NAME][0], atol=1e-6) + np.testing.assert_allclose(fk["right_wrist"], link_poses[AGIBOT_GEAR_RIGHT_EE_LINK_NAME][0], atol=1e-6) + assert not np.allclose( + fk["left_wrist"][:3, 3], + link_poses[AGIBOT_GEAR_LEFT_GRIPPER_CENTER_LINK_NAME][0, :3, 3], + atol=1e-6, + ) + + +@_REQUIRES_MUJOCO +@pytest.mark.L0 +def test_dataset_gripper_poses_are_aligned_to_opencv_convention() -> None: + import numpy as np + + from cosmos_framework.data.vfm.action.embodiment_c_fk import compute_fk_transforms, compute_link_poses_batch + + states = np.zeros((3, 32), dtype=np.float32) # [T+1,S] + dataset = EmbodimentCGripperDataset( + root=["unused"], + chunk_length=2, + skip_video_loading=True, + return_agibot_link_poses=True, + ) + action, extras = dataset._build_fk_action({"observation.state": torch.from_numpy(states)}) + fk = compute_fk_transforms(states[0], "embodiment_c_gripper") + link_poses = compute_link_poses_batch(states[:1], "embodiment_c_gripper") + to_opencv = AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST + + assert action.shape == (2, 29) + assert "agibot_link_poses" in extras + assert extras["agibot_link_poses"][AGIBOT_GEAR_LEFT_EE_LINK_NAME].shape == (3, 4, 4) + np.testing.assert_allclose( + extras["initial_pose_left"].numpy()[:3, :3], + fk["left_wrist"][:3, :3] @ to_opencv["left_wrist"], + atol=1e-6, + ) + np.testing.assert_allclose( + extras["initial_pose_right"].numpy()[:3, :3], + fk["right_wrist"][:3, :3] @ to_opencv["right_wrist"], + atol=1e-6, + ) + np.testing.assert_allclose(fk["left_wrist"], link_poses[AGIBOT_GEAR_LEFT_EE_LINK_NAME][0], atol=1e-6) + np.testing.assert_allclose(fk["right_wrist"], link_poses[AGIBOT_GEAR_RIGHT_EE_LINK_NAME][0], atol=1e-6) + np.testing.assert_allclose( + extras["initial_pose_left"].numpy()[:3, 3], + link_poses[AGIBOT_GEAR_LEFT_EE_LINK_NAME][0, :3, 3], + atol=1e-6, + ) + np.testing.assert_allclose( + extras["initial_pose_right"].numpy()[:3, 3], + link_poses[AGIBOT_GEAR_RIGHT_EE_LINK_NAME][0, :3, 3], + atol=1e-6, + ) + + +@_REQUIRES_MUJOCO +@pytest.mark.L0 +def test_dataset_does_not_return_agibot_link_poses_by_default() -> None: + import numpy as np + + states = np.zeros((3, 32), dtype=np.float32) # [T+1,S] + dataset = EmbodimentCGripperDataset(root=["unused"], chunk_length=2, skip_video_loading=True) + + _, extras = dataset._build_fk_action({"observation.state": torch.from_numpy(states)}) + + assert "agibot_link_poses" not in extras + + +@pytest.mark.L0 +def test_agibot_gripper_to_opencv_composes_extra_180deg_z_rotation() -> None: + import numpy as np + + expected_left = np.asarray( + [ + [0.0, 1.0, 0.0], + [-1.0, 0.0, 0.0], + [0.0, 0.0, 1.0], + ], + dtype=np.float32, + ) + expected_right = np.asarray( + [ + [0.0, -1.0, 0.0], + [1.0, 0.0, 0.0], + [0.0, 0.0, 1.0], + ], + dtype=np.float32, + ) + + np.testing.assert_allclose(AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST["left_wrist"], expected_left, atol=1e-6) + np.testing.assert_allclose(AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST["right_wrist"], expected_right, atol=1e-6) + + +def test_renderer_undoes_agibot_to_opencv_for_ik() -> None: + import numpy as np + + from cosmos_framework.data.vfm.action.urdf_visualizer.unified_renderer import _undo_wrist_to_opencv_for_ik + + to_opencv = AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST + native_poses = np.tile(np.eye(4, dtype=np.float32), (2, 1, 1)) # [T,4,4] + native_poses[:, :3, 3] = np.asarray([[0.1, 0.0, 0.2], [0.2, 0.0, 0.2]], dtype=np.float32) # [T,3] + viewer_poses = native_poses.copy() # [T,4,4] + viewer_poses[:, :3, :3] = viewer_poses[:, :3, :3] @ to_opencv["right_wrist"] # [T,3,3] + + ik_poses = _undo_wrist_to_opencv_for_ik(viewer_poses, to_opencv, "right_wrist") + + assert ik_poses is not None + np.testing.assert_allclose(ik_poses, native_poses, atol=1e-6) + + +def test_gripper_open_fraction_conversions() -> None: + import numpy as np + + from cosmos_framework.data.vfm.action.embodiment_c_fk import convert_gripper_state_to_open_fraction + + open_jitter_degrees = np.array([0.0, 0.21733333], dtype=np.float32) + np.testing.assert_allclose( + convert_gripper_state_to_open_fraction(open_jitter_degrees), + [1.0, 0.9981889], + atol=1e-6, + ) + np.testing.assert_allclose( + convert_gripper_state_to_open_fraction(np.array([0.0], dtype=np.float32)), + [1.0], + atol=1e-6, + ) + + radians = np.array([-AGIBOT_GEAR_GRIPPER_OPEN_ANGLE_RAD, -AGIBOT_GEAR_GRIPPER_OPEN_ANGLE_RAD / 2.0, 0.0]) + np.testing.assert_allclose(convert_gripper_state_to_open_fraction(radians), [1.0, 0.5, 0.0], atol=1e-6) + + actuator_degrees = np.array([0.0, 60.0, 120.0], dtype=np.float32) + np.testing.assert_allclose(convert_gripper_state_to_open_fraction(actuator_degrees), [1.0, 0.5, 0.0], atol=1e-6) + + +@pytest.mark.L0 +def test_gripper_open_fraction_clips_small_actuator_overshoot() -> None: + import numpy as np + + from cosmos_framework.data.vfm.action.embodiment_c_fk import convert_gripper_state_to_open_fraction + + actuator_degrees = np.array([122.1857], dtype=np.float32) # [1] + np.testing.assert_allclose(convert_gripper_state_to_open_fraction(actuator_degrees), [0.0], atol=1e-6) + + +@_REQUIRES_MUJOCO +def test_link_poses_apply_observed_gripper_state() -> None: + import numpy as np + + from cosmos_framework.data.vfm.action.embodiment_c_fk import compute_link_poses_batch + + states = np.zeros((2, 32), dtype=np.float32) # [T,S] + states[1, 14] = 120.0 + link_poses = compute_link_poses_batch(states, "embodiment_c_gripper") + + open_pose = link_poses["gripper_l_inner_link1"][0] # [4,4] + closed_pose = link_poses["gripper_l_inner_link1"][1] # [4,4] + assert not np.allclose(open_pose, closed_pose) + + +@_REQUIRES_MUJOCO +def test_standard_body_head_layout_uses_state19_as_waist_lift() -> None: + import numpy as np + + from cosmos_framework.data.vfm.action.embodiment_c_fk import compute_fk_transforms_batch + + states = np.zeros((2, 32), dtype=np.float32) # [T,S] + states[:, 17] = 0.25 # head pitch + states[:, 18] = 0.5 # waist pitch + states[:, 19] = 0.1 # waist lift + states[1, 19] = 0.2 + + fk = compute_fk_transforms_batch(states, "embodiment_c_gripper") + + for key in ("head_camera", "right_wrist", "left_wrist"): + np.testing.assert_allclose(fk[key][1, 2, 3] - fk[key][0, 2, 3], 0.1, atol=1e-6) + + +@_REQUIRES_MUJOCO +def test_build_viewer_batch_gripper_shapes() -> None: + sample = _make_sample(state_dim=32) + + batch = build_viewer_batch(sample, "embodiment_c_gripper") + + assert batch.head_camera_poses.shape == (2, 4, 4) + assert batch.right_wrist_poses.shape == (2, 4, 4) + assert batch.left_wrist_poses.shape == (2, 4, 4) + assert batch.right_gripper is not None + assert batch.left_gripper is not None + assert batch.right_gripper.shape == (2,) + assert batch.left_gripper.shape == (2,) + + +@_REQUIRES_MUJOCO +@pytest.mark.skipif(not _LOCAL_GRIPPER_SAMPLE_ROOT.is_dir(), reason="local Embodiment C sample data is unavailable") +def test_embodiment_c_dataset_loads_local_gripper_sample_without_video() -> None: + dataset = EmbodimentCGripperDataset( + root=[str(_LOCAL_GRIPPER_SAMPLE_ROOT)], + fps=30.0, + split="full", + mode="policy", + skip_video_loading=True, + ) + dataset._register_sources() + + sample = dataset[0] + + assert len(dataset) == 184 + assert sample["action"].shape == (16, 29) + assert sample["__episode_id__"] == 0 + assert sample["__task_root__"] == str(_LOCAL_GRIPPER_SAMPLE_ROOT) + + +@_REQUIRES_MUJOCO +def test_agibotworld_beta_builds_fk_action_for_viewer() -> None: + dataset = AgiBotWorldBetaDataset( + root=[], + chunk_length=2, + split="full", + mode="policy", + ) + sample = { + "observation.states.effector.position": torch.zeros((3, 2), dtype=torch.float32), + "observation.states.joint.position": torch.zeros((3, 14), dtype=torch.float32), + "observation.states.head.position": torch.zeros((3, 2), dtype=torch.float32), + "observation.states.waist.position": torch.zeros((3, 2), dtype=torch.float32), + **_make_beta_robot_pose(3), + } + + action, extras = dataset._build_fk_action(sample) + + assert action.shape == (2, 29) + assert extras["initial_pose"].shape == (4, 4) + assert extras["initial_pose_right"].shape == (4, 4) + assert extras["initial_pose_left"].shape == (4, 4) + + +@_REQUIRES_MUJOCO +def test_agibotworld_beta_link_poses_include_observed_base_motion() -> None: + import numpy as np + + dataset = AgiBotWorldBetaDataset( + root=[], + chunk_length=2, + split="full", + mode="policy", + return_agibot_link_poses=True, + ) + robot_pose = _make_beta_robot_pose(3) + robot_pose["observation.states.robot.position"] = torch.tensor( + [[0.0, 0.0, 0.0], [0.4, -0.2, 0.0], [0.7, -0.1, 0.0]], + dtype=torch.float32, + ) # [T+1,3] + sample = { + "observation.states.effector.position": torch.zeros((3, 2), dtype=torch.float32), + "observation.states.joint.position": torch.zeros((3, 14), dtype=torch.float32), + "observation.states.head.position": torch.zeros((3, 2), dtype=torch.float32), + "observation.states.waist.position": torch.zeros((3, 2), dtype=torch.float32), + **robot_pose, + } + + _, extras = dataset._build_fk_action(sample) + + assert "agibot_link_poses" in extras + head_poses = extras["agibot_link_poses"][AGIBOT_GEAR_HEAD_CAMERA_LINK_NAME].numpy() # [T+1,4,4] + np.testing.assert_allclose(head_poses[1, :3, 3] - head_poses[0, :3, 3], [0.4, -0.2, 0.0], atol=1e-6) + np.testing.assert_allclose(head_poses[2, :3, 3] - head_poses[0, :3, 3], [0.7, -0.1, 0.0], atol=1e-6) + + +@_REQUIRES_MUJOCO +@pytest.mark.L0 +def test_agibotworld_beta_keeps_debug_detail_out_of_ai_caption(monkeypatch: pytest.MonkeyPatch) -> None: + dataset = AgiBotWorldBetaDataset( + root=[], + chunk_length=2, + split="full", + mode="policy", + ) + video = torch.zeros((3, 3, 4, 4), dtype=torch.float32) # [T+1,C,H,W] + wrist_video = torch.zeros((3, 3, 2, 2), dtype=torch.float32) # [T+1,C,H,W] + sample = { + "task": "Open the wardrobe and hang the clothes. | Debug detail for viewer only.", + "observation.images.head": video, + "observation.images.hand_left": wrist_video, + "observation.images.hand_right": wrist_video, + "observation.states.effector.position": torch.zeros((3, 2), dtype=torch.float32), # [T+1,2] + "observation.states.joint.position": torch.zeros((3, 14), dtype=torch.float32), # [T+1,14] + "observation.states.head.position": torch.zeros((3, 2), dtype=torch.float32), # [T+1,2] + "observation.states.waist.position": torch.zeros((3, 2), dtype=torch.float32), # [T+1,2] + **_make_beta_robot_pose(3), + } + + def _fake_fetch_sample(_idx: int) -> tuple[str, int, int, dict[str, object]]: + return "policy", 0, 0, sample + + monkeypatch.setattr(dataset, "_fetch_sample", _fake_fetch_sample) + + result = dataset[0] + + assert result["ai_caption"] == "Open the wardrobe and hang the clothes." + assert result["debug_caption"] == "Debug detail for viewer only." + + +def test_agibotworld_beta_viewer_entry_declares_agibot_robot_embodiment() -> None: + assert DATASETS["agibotworld_beta"].robot_embodiment_type == "embodiment_c_gripper" + assert DATASETS["agibotworld_beta"].dataset_kwargs["return_agibot_link_poses"] is True + + +def test_embodiment_c_gripper_ext_registered_for_viewer_load() -> None: + assert DATASETS["embodiment_c_gripper_ext"].robot_embodiment_type == "embodiment_c_gripper_ext" + assert get_domain_id("embodiment_c_gripper_ext") == get_domain_id("embodiment_c_gripper") + + +def test_agibot_viewer_entries_do_not_force_debug_roots() -> None: + for name in ("embodiment_c_gripper", "embodiment_c_gripper_ext", "agibotworld_beta"): + kwargs = DATASETS[name].dataset_kwargs + assert "root" not in kwargs + assert "max_loaded_datasets" not in kwargs + + +@_REQUIRES_MUJOCO +def test_build_viewer_batch_gripper_ext_shapes() -> None: + """Ext: 94-dim state → same viewer batch shape as standard gripper.""" + sample = _make_sample(state_dim=94) + + batch = build_viewer_batch(sample, "embodiment_c_gripper_ext") + + assert batch.head_camera_poses.shape == (2, 4, 4) + assert batch.right_wrist_poses.shape == (2, 4, 4) + assert batch.left_wrist_poses.shape == (2, 4, 4) + assert batch.right_gripper is not None + assert batch.left_gripper is not None + assert batch.right_gripper.shape == (2,) + assert batch.left_gripper.shape == (2,) + + +@_REQUIRES_MUJOCO +def test_ext_fk_uses_correct_state_indices() -> None: + """Verify ext FK reads arm joints from state[54:68], not state[0:14].""" + import numpy as np + + from cosmos_framework.data.vfm.action.embodiment_c_fk import compute_fk_transforms + + # Build a 94-dim state where arm joints at [54:68] differ from [0:14]. + state_ext = np.zeros(94, dtype=np.float32) + # Put distinctive values in the correct ext arm joint positions. + state_ext[54:61] = [0.1, -0.2, 0.3, -0.1, 0.2, 0.5, -0.3] # left arm + state_ext[61:68] = [-0.1, 0.2, -0.3, 0.1, -0.2, -0.5, 0.3] # right arm + state_ext[82] = 0.0 # head yaw + state_ext[83] = 0.3 # head pitch + state_ext[84] = 0.5 # waist pitch + state_ext[85] = 0.35 # waist lift + + # Build an equivalent 32-dim standard state with the same joint values. + state_std = np.zeros(32, dtype=np.float32) + state_std[0:7] = state_ext[54:61] # left arm + state_std[7:14] = state_ext[61:68] # right arm + state_std[16] = 0.0 # head yaw + state_std[17] = 0.3 # head pitch + state_std[18] = 0.5 # waist pitch + state_std[19] = 0.35 # waist lift + + fk_ext = compute_fk_transforms(state_ext, "embodiment_c_gripper_ext") + fk_std = compute_fk_transforms(state_std, "embodiment_c_gripper") + + # Ext and standard FK should produce identical transforms. + for key in ("head_camera", "right_wrist", "left_wrist"): + np.testing.assert_allclose(fk_ext[key], fk_std[key], atol=1e-6, err_msg=f"FK mismatch for {key}") + + +@_REQUIRES_MUJOCO +def test_ext_fk_applies_robot_base_motion_to_batch_poses() -> None: + """Ext FK folds state/robot pose into all head and wrist trajectories.""" + import numpy as np + + from cosmos_framework.data.vfm.action.embodiment_c_fk import compute_fk_transforms_batch + + states = np.zeros((2, 94), dtype=np.float32) # [T,S] + states[:, AGIBOT_GEAR_EXT_STATE_ROBOT_ORIENTATION_SLICE] = np.array([0.0, 0.0, 0.0, 1.0], dtype=np.float32) + states[1, AGIBOT_GEAR_EXT_STATE_ROBOT_POSITION_SLICE] = np.array([0.4, -0.2, 0.0], dtype=np.float32) + + fk = compute_fk_transforms_batch(states, "embodiment_c_gripper_ext") + + for key in ("head_camera", "right_wrist", "left_wrist"): + np.testing.assert_allclose(fk[key][1, :3, 3] - fk[key][0, :3, 3], [0.4, -0.2, 0.0], atol=1e-6) diff --git a/cosmos_framework/data/vfm/action/embodiment_c_spec.py b/cosmos_framework/data/vfm/action/embodiment_c_spec.py new file mode 100644 index 0000000..bfc01a3 --- /dev/null +++ b/cosmos_framework/data/vfm/action/embodiment_c_spec.py @@ -0,0 +1,145 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Shared Embodiment C metadata used by datasets and visualizers.""" + +from __future__ import annotations + +import math +from dataclasses import dataclass +from pathlib import Path +from typing import Literal + +AgibotGearKind = Literal["gripper"] + +AGIBOT_GEAR_VIDEO_KEY = "observation.images.top_head" +AGIBOT_GEAR_URDF_FILENAME = "G1_omnipicker_calibrated.urdf" +AGIBOT_GEAR_ROBOT_NAME = "embodiment_c" +AGIBOT_GEAR_GRIPPER_NORMALIZER_EMBODIMENT_TYPE = "embodiment_c_gripper" +AGIBOT_GEAR_ARM_STATE_SLICE = slice(0, 14) +AGIBOT_GEAR_STATE_HEAD_YAW_IDX = 16 +AGIBOT_GEAR_STATE_HEAD_PITCH_IDX = 17 +AGIBOT_GEAR_STATE_WAIST_PITCH_IDX = 18 +AGIBOT_GEAR_STATE_WAIST_LIFT_IDX = 19 +AGIBOT_GEAR_HEAD_PITCH_JOINT_NAME = "idx04_head_pitch_joint" + +# -- Ext layout constants (94-dim state) ------------------------------------- +# The ext split stores joints at different offsets from the standard layout. +AGIBOT_GEAR_EXT_ARM_STATE_SLICE = slice(54, 68) +AGIBOT_GEAR_EXT_STATE_HEAD_YAW_IDX = 82 +AGIBOT_GEAR_EXT_STATE_HEAD_PITCH_IDX = 83 +AGIBOT_GEAR_EXT_STATE_WAIST_PITCH_IDX = 84 +AGIBOT_GEAR_EXT_STATE_WAIST_LIFT_IDX = 85 +AGIBOT_GEAR_EXT_STATE_ROBOT_POSITION_SLICE = slice(86, 89) +AGIBOT_GEAR_EXT_STATE_ROBOT_ORIENTATION_SLICE = slice(89, 93) +AGIBOT_GEAR_EXT_STATE_LEFT_HAND_SLICE = slice(0, 1) +AGIBOT_GEAR_EXT_STATE_RIGHT_HAND_SLICE = slice(1, 2) +AGIBOT_GEAR_ARM_BASE_LINK_NAME = "arm_base_link" +AGIBOT_GEAR_HEAD_LINK_NAME = "head_pitch_link" +AGIBOT_GEAR_HEAD_CAMERA_LINK_NAME = "head_camera_link" +AGIBOT_GEAR_LEFT_EE_LINK_NAME = "gripper_l_base_link" +AGIBOT_GEAR_RIGHT_EE_LINK_NAME = "gripper_r_base_link" +AGIBOT_GEAR_LEFT_GRIPPER_CENTER_LINK_NAME = "gripper_l_center_link" +AGIBOT_GEAR_RIGHT_GRIPPER_CENTER_LINK_NAME = "gripper_r_center_link" +AGIBOT_GEAR_ARM_JOINT_NAMES_LEFT = tuple(f"idx{4 + i:02d}_left_arm_joint{i}" for i in range(1, 8)) +AGIBOT_GEAR_ARM_JOINT_NAMES_RIGHT = tuple(f"idx{11 + i:02d}_right_arm_joint{i}" for i in range(1, 8)) +AGIBOT_GEAR_WAIST_LIFT_JOINT_NAME = "idx01_waist_lift_joint" +AGIBOT_GEAR_WAIST_PITCH_JOINT_NAME = "idx02_waist_pitch_joint" +AGIBOT_GEAR_HEAD_YAW_JOINT_NAME = "idx03_head_yaw_joint" +AGIBOT_GEAR_GRIPPER_OPEN_ANGLE_RAD = math.pi / 4.0 +AGIBOT_GEAR_GRIPPER_OPEN_ACTUATOR_DEG = 120.0 +AGIBOT_GEAR_LEFT_GRIPPER_JOINT_MIMICS = ( + ("idx31_gripper_l_inner_joint1", 1.0, 0.0), + ("idx32_gripper_l_inner_joint3", 0.1, 0.0), + ("idx33_gripper_l_inner_joint4", 0.25, 0.0), + ("idx39_gripper_l_inner_joint0", -0.7, 0.0), + ("idx41_gripper_l_outer_joint1", -1.0, 0.0), + ("idx42_gripper_l_outer_joint3", 0.1, 0.0), + ("idx43_gripper_l_outer_joint4", -0.25, 0.0), + ("idx49_gripper_l_outer_joint0", 0.7, 0.0), +) +AGIBOT_GEAR_RIGHT_GRIPPER_JOINT_MIMICS = ( + ("idx71_gripper_r_inner_joint1", 1.0, 0.0), + ("idx72_gripper_r_inner_joint3", 0.1, 0.0), + ("idx73_gripper_r_inner_joint4", 0.25, 0.0), + ("idx79_gripper_r_inner_joint0", -0.7, 0.0), + ("idx81_gripper_r_outer_joint1", -1.0, 0.0), + ("idx82_gripper_r_outer_joint3", 0.1, 0.0), + ("idx83_gripper_r_outer_joint4", -0.25, 0.0), + ("idx89_gripper_r_outer_joint0", 0.7, 0.0), +) + + +@dataclass(frozen=True) +class AgibotGearKindSpec: + """Layout metadata shared across all embodiments of one hand kind.""" + + kind: AgibotGearKind + state_hand_slice: slice + + +@dataclass(frozen=True) +class AgibotGearEmbodimentSpec: + """Per-embodiment metadata shared by training and visualization code.""" + + embodiment_type: str + kind: AgibotGearKind + action_dim: int + + +AGIBOT_GEAR_KIND_SPECS: dict[AgibotGearKind, AgibotGearKindSpec] = { + "gripper": AgibotGearKindSpec( + kind="gripper", + state_hand_slice=slice(14, 16), + ), +} + +AGIBOT_GEAR_EMBODIMENT_SPECS: dict[str, AgibotGearEmbodimentSpec] = { + "embodiment_c_gripper": AgibotGearEmbodimentSpec( + embodiment_type="embodiment_c_gripper", + kind="gripper", + action_dim=29, # FK output: head(9)+right(9)+gripper(1)+left(9)+gripper(1) + ), + "embodiment_c_gripper_ext": AgibotGearEmbodimentSpec( + embodiment_type="embodiment_c_gripper_ext", + kind="gripper", + action_dim=29, # FK output: head(9)+right(9)+gripper(1)+left(9)+gripper(1) + ), +} + + +def get_embodiment_c_embodiment_spec(embodiment_type: str) -> AgibotGearEmbodimentSpec: + """Return the registered spec for one AgiBot embodiment.""" + + try: + return AGIBOT_GEAR_EMBODIMENT_SPECS[embodiment_type] + except KeyError as exc: + raise ValueError( + f"Unknown Embodiment C embodiment_type={embodiment_type!r}. " + f"Expected one of {sorted(AGIBOT_GEAR_EMBODIMENT_SPECS)}." + ) from exc + + +def get_embodiment_c_kind_spec(embodiment_type: str | AgibotGearKind) -> AgibotGearKindSpec: + """Resolve an embodiment type or kind to its shared layout metadata.""" + + kind = embodiment_type if embodiment_type in AGIBOT_GEAR_KIND_SPECS else get_embodiment_c_kind(embodiment_type) + return AGIBOT_GEAR_KIND_SPECS[kind] + + +def get_embodiment_c_kind(embodiment_type: str) -> AgibotGearKind: + """Return the hand kind used by one AgiBot embodiment.""" + + return get_embodiment_c_embodiment_spec(embodiment_type).kind + + +def get_embodiment_c_action_dim(embodiment_type: str) -> int: + """Return the action dimension for one AgiBot embodiment.""" + + return get_embodiment_c_embodiment_spec(embodiment_type).action_dim + + +def get_embodiment_c_urdf_path() -> Path: + """Return the committed Embodiment C G1 omnipicker URDF path.""" + + return Path(__file__).resolve().parent / "urdf_visualizer" / AGIBOT_GEAR_URDF_FILENAME diff --git a/cosmos_framework/data/vfm/action/filter_bridge_dataset.py b/cosmos_framework/data/vfm/action/filter_bridge_dataset.py new file mode 100644 index 0000000..6b8cb2b --- /dev/null +++ b/cosmos_framework/data/vfm/action/filter_bridge_dataset.py @@ -0,0 +1,682 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Filter and split bridge_orig_lerobot dataset. + +Step 1 (filter): + - Load task instructions from tasks.parquet, normalize text, classify each + task as clean or flagged (gibberish / single-word / question / non-English / + pattern-matched), merge duplicate texts, then write: + bridge_tasks_clean.json — clean tasks + bridge_tasks_flagged.json — flagged tasks + +Step 2 (split, unless --filter_only): + - Create {name}_clean dataset dir (symlinked data/videos, filtered meta). + - Optionally create {name}_dirty dir with --save_dirty. + - Optionally trim static head/tail frames with --static_threshold. + +Usage: + CMD="python filter_bridge_dataset.py" + + # Filter only (produce JSONs, no dataset split) + $CMD --root /path/to/dataset --filter_only + + # Filter + split (clean only, default) + $CMD --root /path/to/dataset + + # Filter + split (clean + dirty) + $CMD --root /path/to/dataset --save_dirty + + # Trim static frames (action[:6] near zero) from clean episodes + $CMD --root /path/to/dataset --static_threshold 5e-4 + + # Custom output locations + $CMD --root /path/to/dataset --output_dir /tmp/results --output_name my_bridge + +Arguments: + --root Path to the original LeRobot dataset + --output_dir Dir for JSON outputs (default: script dir) + --output_name Base name for split dirs (default: derived from --root) + --static_threshold Trim static head/tail frames; 0 = off (default), try 5e-4 + --save_dirty Also create the dirty (flagged) split + --filter_only Only run filter step, skip split +""" + +import argparse +import glob +import json +import os +import re +import shutil + +import nltk +import numpy as np +import pandas as pd + +nltk.download("words", quiet=True) +from nltk.corpus import words as nltk_words # noqa: E402 + +ENGLISH_WORDS = set(w.lower() for w in nltk_words.words()) +TASK_WORD_ALLOWLIST = { + # Common robotics task words that are valid English but missing from nltk.words. + "fridge", +} + +FLAGGED_PATTERNS = [ + r".*image.*", + r".*https?.*", + r".*\bnot\b.*", + r".*\bnothing\b.*", + r".*\banything\b.*", + r".*\bdidn.?t\b.*", + r".*\bdoesn.?t\b.*", + r".*\baren.?t\b.*", + r".*\bupload\b.*", + r".*\bpicture\b.*", + r"\(.*\)", +] + + +# --------------------------------------------------------------------------- +# Text normalization +# --------------------------------------------------------------------------- + + +def normalize_task(text: str) -> str: + t = text.strip() + t = t.strip("\"'''") + t = t.replace("\u201a\u00c4\u00f4", "'") + t = re.sub(r"\s{2,}", " ", t) + if "." in t and " " not in t: + t = t.replace(".", " ") + if "_" in t and " " not in t: + t = t.replace("_", " ") + return t.strip() + + +# --------------------------------------------------------------------------- +# Classification helpers +# --------------------------------------------------------------------------- + + +def _clean_word(word: str) -> str: + return word.lower().strip(".,!?;:'\"()-/\\") + + +def _is_english(word: str) -> bool: + w = _clean_word(word) + if not w: + return True + if w in TASK_WORD_ALLOWLIST: + return True + if not w.isalpha(): + alpha_part = "".join(c for c in w if c.isalpha()) + if len(alpha_part) >= 3 and alpha_part not in ENGLISH_WORDS: + return False + return True + if len(w) <= 2: + return True + if w in ENGLISH_WORDS: + return True + for suffix in ("s", "ed", "ing", "ly", "er", "est", "tion", "ness"): + stem = w.removesuffix(suffix) + if len(stem) >= 2 and stem in ENGLISH_WORDS: + return True + return False + + +def is_gibberish(text: str) -> tuple[bool, str]: + t = text.strip() + if not t: + return False, "" + words = t.split() + alpha_words = [w for w in words if any(c.isalpha() for c in w)] + if not alpha_words: + return True, "no_alphabetic_words" + non_english = [w for w in alpha_words if not _is_english(w)] + ratio = len(non_english) / len(alpha_words) + if len(alpha_words) <= 2 and len(non_english) >= 1: + return ( + True, + f"short_non_english({len(non_english)}/{len(alpha_words)}): {non_english[:8]}", + ) + if len(alpha_words) >= 3 and ratio >= 0.6: + return ( + True, + f"non_english_words({len(non_english)}/{len(alpha_words)}): {non_english[:8]}", + ) + if len(alpha_words) >= 6 and ratio >= 0.5: + return ( + True, + f"many_non_english({len(non_english)}/{len(alpha_words)}): {non_english[:8]}", + ) + if re.search(r"(.)\1{4,}", t.lower()): + return True, "excessive_char_repetition" + short_non_eng = [w for w in alpha_words if len(_clean_word(w)) <= 3 and not _is_english(w)] + if len(alpha_words) >= 5 and len(short_non_eng) / len(alpha_words) >= 0.6: + return ( + True, + f"many_short_nonsense({len(short_non_eng)}/{len(alpha_words)}): {short_non_eng[:8]}", + ) + return False, "" + + +def is_pattern_flagged(text: str) -> bool: + t = text.strip().lower() + if not t: + return True + return any(re.match(pat, t) for pat in FLAGGED_PATTERNS) + + +def is_single_word(text: str) -> bool: + words = text.strip().split() + return sum(1 for w in words if any(c.isalpha() for c in w)) == 1 + + +def is_question(text: str) -> bool: + t = text.strip() + if t.endswith("?"): + return True + t_lower = t.lower() + q_starts = ( + "what ", + "how ", + "why ", + "where ", + "when ", + "who ", + "which ", + "did ", + "does ", + "do ", + "can ", + "could ", + "would ", + "should ", + "will ", + ) + return any(t_lower.startswith(q) for q in q_starts) + + +def is_non_english(text: str) -> bool: + return sum(1 for c in text.strip() if not c.isascii() and c.isalpha()) >= 2 + + +def classify_task(text: str) -> list[str]: + reasons = [] + if is_pattern_flagged(text): + reasons.append("pattern") + gib, _ = is_gibberish(text) + if gib: + reasons.append("gibberish") + if is_question(text): + reasons.append("question") + if is_non_english(text): + reasons.append("non_english") + if is_single_word(text) and not reasons: + reasons.append("single_word") + return reasons + + +# --------------------------------------------------------------------------- +# I/O helpers +# --------------------------------------------------------------------------- + + +def save_compact_json(path, description, num_tasks, num_episodes, tasks): + with open(path, "w") as f: + f.write("{\n") + f.write(f' "description": {json.dumps(description)},\n') + f.write(f' "num_tasks": {num_tasks},\n') + f.write(f' "num_episodes": {num_episodes},\n') + f.write(' "tasks": [\n') + for i, t in enumerate(tasks): + line = json.dumps(t, ensure_ascii=False) + comma = "," if i < len(tasks) - 1 else "" + f.write(f" {line}{comma}\n") + f.write(" ]\n") + f.write("}\n") + + +def load_task_text_col(root): + tasks_df = pd.read_parquet(os.path.join(root, "meta", "tasks.parquet")).reset_index() + text_candidates = [c for c in tasks_df.columns if c != "task_index"] + if text_candidates: + return tasks_df, text_candidates[0] + if tasks_df["task_index"].dtype == object: + tasks_df["task_text"] = tasks_df["task_index"] + tasks_df["task_index"] = range(len(tasks_df)) + return tasks_df, "task_text" + raise RuntimeError(f"Cannot find task text column: {tasks_df.columns.tolist()}") + + +# --------------------------------------------------------------------------- +# Step 1: Filter tasks +# --------------------------------------------------------------------------- + + +def filter_tasks(root, output_dir): + print(f"nltk dictionary: {len(ENGLISH_WORDS)} words") + tasks_df, text_col = load_task_text_col(root) + print(f"Total unique tasks: {len(tasks_df)}") + + data_files = sorted(glob.glob(os.path.join(root, "data", "**", "*.parquet"), recursive=True)) + print(f"Reading {len(data_files)} data parquet files ...") + dfs = [pd.read_parquet(f, columns=["episode_index", "task_index"]) for f in data_files] + data_df = pd.concat(dfs, ignore_index=True) + + ep_tasks = data_df.drop_duplicates("episode_index")[["episode_index", "task_index"]] + task_counts = ep_tasks.groupby("task_index").size().reset_index(name="num_episodes") + total_episodes = ep_tasks["episode_index"].nunique() + print(f"Total episodes: {total_episodes}") + + merged = task_counts.merge(tasks_df, on="task_index", how="left") + merged[text_col] = merged[text_col].apply(normalize_task) + merged["flag_reasons"] = merged[text_col].apply(classify_task) + merged["flagged"] = merged["flag_reasons"].apply(lambda r: len(r) > 0) + + # Merge duplicate task texts + merged["task_normalized"] = merged[text_col].str.strip().str.lower() + + def _merge_group(group): + return { + "task": group[text_col].iloc[0].strip(), + "task_indices": sorted(group["task_index"].tolist()), + "num_episodes": int(group["num_episodes"].sum()), + "flagged": bool(group["flagged"].any()), + "flag_reasons": sorted(set(r for reasons in group["flag_reasons"] for r in reasons)), + } + + grouped = merged.groupby("task_normalized").apply(_merge_group, include_groups=False).tolist() + + clean_tasks = sorted([t for t in grouped if not t["flagged"]], key=lambda x: -x["num_episodes"]) + flagged_tasks = sorted([t for t in grouped if t["flagged"]], key=lambda x: -x["num_episodes"]) + + total_clean_eps = sum(t["num_episodes"] for t in clean_tasks) + total_flagged_eps = sum(t["num_episodes"] for t in flagged_tasks) + + os.makedirs(output_dir, exist_ok=True) + clean_path = os.path.join(output_dir, "bridge_tasks_clean.json") + save_compact_json( + clean_path, + "Clean task instructions", + len(clean_tasks), + total_clean_eps, + clean_tasks, + ) + + flagged_path = os.path.join(output_dir, "bridge_tasks_flagged.json") + save_compact_json( + flagged_path, + "Flagged task instructions", + len(flagged_tasks), + total_flagged_eps, + flagged_tasks, + ) + + print( + f"\nClean: {len(clean_tasks)} tasks / {total_clean_eps} episodes ({total_clean_eps / total_episodes * 100:.1f}%)" + ) + print( + f"Flagged: {len(flagged_tasks)} tasks / {total_flagged_eps} episodes ({total_flagged_eps / total_episodes * 100:.1f}%)" + ) + for label in ("pattern", "gibberish", "single_word", "question", "non_english"): + n = sum(1 for t in flagged_tasks if label in t["flag_reasons"]) + if n: + print(f" - {label}: {n}") + print(f"Saved: {clean_path}") + print(f"Saved: {flagged_path}") + + return clean_path, flagged_path + + +# --------------------------------------------------------------------------- +# Step 2: Split dataset (symlinked data/videos, filtered meta) +# --------------------------------------------------------------------------- + + +def build_episode_task_map(root): + data_files = sorted(glob.glob(os.path.join(root, "data", "**", "*.parquet"), recursive=True)) + dfs = [pd.read_parquet(f, columns=["episode_index", "task_index"]) for f in data_files] + data_df = pd.concat(dfs, ignore_index=True) + return data_df.drop_duplicates("episode_index").set_index("episode_index")["task_index"].to_dict() + + +def create_split(orig_root, output_root, episode_set, orig_info, orig_episodes_df, label, trims=None): + """Create a dataset split by zeroing out frame ranges for excluded episodes. + + LeRobot uses episode_index as a positional index into the episodes table, + so we must keep ALL rows. Instead of removing rows, we zero out + dataset_from_index and dataset_to_index for excluded episodes, which makes + build_episode_spans compute valid_len <= 0 and skip them. + + If *trims* is provided, head/tail static frames are trimmed by adjusting + dataset_from_index / dataset_to_index for included episodes. + """ + os.makedirs(output_root, exist_ok=True) + for subdir in ["data", "videos"]: + src = os.path.join(orig_root, subdir) + dst = os.path.join(output_root, subdir) + if os.path.islink(dst): + os.remove(dst) + elif os.path.isdir(dst): + shutil.rmtree(dst) + elif os.path.exists(dst): + os.remove(dst) + os.symlink(src, dst) + + modified_eps = orig_episodes_df.copy() + excluded_mask = ~modified_eps["episode_index"].isin(episode_set) + has_range = "dataset_from_index" in modified_eps.columns and "dataset_to_index" in modified_eps.columns + + if has_range: + modified_eps.loc[excluded_mask, "dataset_from_index"] = 0 + modified_eps.loc[excluded_mask, "dataset_to_index"] = 0 + + if trims: + for idx in modified_eps.index: + ep_id = modified_eps.at[idx, "episode_index"] + if ep_id in episode_set and ep_id in trims: + head_trim, tail_trim = trims[ep_id] + modified_eps.at[idx, "dataset_from_index"] += head_trim + modified_eps.at[idx, "dataset_to_index"] -= tail_trim + if modified_eps.at[idx, "dataset_from_index"] >= modified_eps.at[idx, "dataset_to_index"]: + modified_eps.at[idx, "dataset_from_index"] = 0 + modified_eps.at[idx, "dataset_to_index"] = 0 + + total_frames = 0 + included = modified_eps[~excluded_mask] + if has_range: + total_frames = int((included["dataset_to_index"] - included["dataset_from_index"]).sum()) + + meta_dir = os.path.join(output_root, "meta") + os.makedirs(meta_dir, exist_ok=True) + + ep_meta_dir = os.path.join(meta_dir, "episodes", "chunk-000") + os.makedirs(ep_meta_dir, exist_ok=True) + modified_eps.to_parquet(os.path.join(ep_meta_dir, "file-000.parquet"), index=False) + + shutil.copy2( + os.path.join(orig_root, "meta", "tasks.parquet"), + os.path.join(meta_dir, "tasks.parquet"), + ) + stats_src = os.path.join(orig_root, "meta", "stats.json") + if os.path.exists(stats_src): + shutil.copy2(stats_src, os.path.join(meta_dir, "stats.json")) + + new_info = orig_info.copy() + new_info["total_episodes"] = len(orig_episodes_df) + if total_frames > 0: + new_info["total_frames"] = total_frames + new_info["splits"] = {"train": f"0:{len(orig_episodes_df)}"} + with open(os.path.join(meta_dir, "info.json"), "w") as f: + json.dump(new_info, f, indent=4) + + print( + f" {label}: {len(episode_set)} valid episodes (of {len(orig_episodes_df)} total), " + f"{total_frames} frames -> {output_root}" + ) + + +def compute_static_trims(root, episode_set, threshold): + """Compute per-episode head/tail static frame counts for trimming. + + A frame is "static" when max(abs(action[:6])) < threshold, where the + first 6 action dims are translation + rotation. + + Returns (trims_dict, details_list): + - trims_dict: {episode_index: (head_trim, tail_trim)} + - details_list: list of dicts for the JSON report + """ + data_files = sorted(glob.glob(os.path.join(root, "data", "**", "*.parquet"), recursive=True)) + print(f" Reading action data from {len(data_files)} files for static frame detection (threshold={threshold}) ...") + dfs = [pd.read_parquet(f, columns=["episode_index", "action"]) for f in data_files] + data_df = pd.concat(dfs, ignore_index=True) + + trims = {} + details = [] + total_head, total_tail = 0, 0 + fully_static = 0 + total_checked = 0 + + for ep_id, ep_df in data_df.groupby("episode_index"): + if ep_id not in episode_set: + continue + total_checked += 1 + + actions = np.stack(ep_df["action"].values) + trans_rot = actions[:, :6] + static_mask = np.abs(trans_rot).max(axis=1) < threshold + n_static = int(static_mask.sum()) + ep_len = len(ep_df) + + if n_static == 0: + continue + + head_trim = 0 + for i in range(ep_len): + if static_mask[i]: + head_trim += 1 + else: + break + + tail_trim = 0 + for i in range(ep_len - 1, -1, -1): + if static_mask[i]: + tail_trim += 1 + else: + break + + if head_trim + tail_trim >= ep_len: + fully_static += 1 + trims[int(ep_id)] = (0, ep_len) + details.append( + { + "episode_index": int(ep_id), + "ep_len": ep_len, + "head": 0, + "tail": ep_len, + "fully_static": True, + } + ) + total_tail += ep_len + continue + + if head_trim > 0 or tail_trim > 0: + trims[int(ep_id)] = (head_trim, tail_trim) + details.append( + { + "episode_index": int(ep_id), + "ep_len": ep_len, + "head": head_trim, + "tail": tail_trim, + "fully_static": False, + } + ) + total_head += head_trim + total_tail += tail_trim + + print( + f" {total_checked} eps checked, {len(trims)} trimmed (fully_static={fully_static}), frames: head={total_head} tail={total_tail} sum={total_head + total_tail}" + ) + return trims, details + + +def save_static_report(path, threshold, details): + """Save a JSON report of static frame trimming.""" + details_sorted = sorted(details, key=lambda x: -(x["head"] + x["tail"])) + total_head = sum(d["head"] for d in details) + total_tail = sum(d["tail"] for d in details) + fully_static = sum(1 for d in details if d["fully_static"]) + + with open(path, "w") as f: + f.write("{\n") + f.write(' "description": "Static frame trimming report",\n') + f.write(f' "threshold": {threshold},\n') + f.write(f' "episodes_trimmed": {len(details)},\n') + f.write(f' "episodes_fully_static": {fully_static},\n') + f.write( + f' "total_frames_trimmed": {{"head": {total_head}, "tail": {total_tail}, "sum": {total_head + total_tail}}},\n' + ) + f.write(' "episodes": [\n') + for i, d in enumerate(details_sorted): + line = json.dumps(d, ensure_ascii=False) + comma = "," if i < len(details_sorted) - 1 else "" + f.write(f" {line}{comma}\n") + f.write(" ]\n") + f.write("}\n") + print(f"Saved: {path}") + + +def split_dataset( + orig_root, + flagged_json, + output_base, + output_dir, + output_name=None, + static_threshold=0, + save_dirty=False, +): + with open(os.path.join(orig_root, "meta", "info.json")) as f: + orig_info = json.load(f) + ep_meta_files = sorted( + glob.glob( + os.path.join(orig_root, "meta", "episodes", "**", "*.parquet"), + recursive=True, + ) + ) + orig_episodes_df = pd.concat([pd.read_parquet(f) for f in ep_meta_files], ignore_index=True) + + with open(flagged_json) as f: + flagged_indices = set() + for t in json.load(f)["tasks"]: + flagged_indices.update(t["task_indices"]) + + ep_task_map = build_episode_task_map(orig_root) + + if "task_index" not in orig_episodes_df.columns: + task_series = pd.Series(ep_task_map, name="task_index") + task_series.index.name = "episode_index" + orig_episodes_df = orig_episodes_df.merge(task_series.reset_index(), on="episode_index", how="left") + + all_episodes = set(orig_episodes_df["episode_index"].tolist()) + clean_eps, dirty_eps = set(), set() + for ep in all_episodes: + ti = ep_task_map.get(ep, -1) + if ti in flagged_indices: + dirty_eps.add(ep) + else: + clean_eps.add(ep) + + trims = None + if static_threshold > 0: + print("\nStep 2b: Detecting static head/tail frames ...") + trims, details = compute_static_trims(orig_root, clean_eps, static_threshold) + report_path = os.path.join(output_dir, "bridge_static_trims.json") + save_static_report(report_path, static_threshold, details) + + base_name = output_name or os.path.basename(orig_root.rstrip("/")) + print(f"\nSplitting: {len(clean_eps)} clean, {len(dirty_eps)} dirty (save_dirty={save_dirty})") + create_split( + orig_root, + os.path.join(output_base, f"{base_name}_clean"), + clean_eps, + orig_info, + orig_episodes_df, + "CLEAN", + trims, + ) + if save_dirty: + create_split( + orig_root, + os.path.join(output_base, f"{base_name}_dirty"), + dirty_eps, + orig_info, + orig_episodes_df, + "DIRTY", + ) + + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + + +def main(): + parser = argparse.ArgumentParser(description="Filter and split bridge dataset") + parser.add_argument( + "--root", + default="", + ) + parser.add_argument( + "--output_json_dir", + default=None, + help="Dir for JSON outputs (default: same as this script)", + ) + parser.add_argument( + "--output_name", + default=None, + help="Base name for split dirs (default: derived from --root)", + ) + parser.add_argument( + "--output_data_dir", + default=None, + help="Dir for output split dataset (default: parent of root)", + ) + parser.add_argument( + "--static_threshold", + type=float, + default=0, + help="Trim static head/tail frames where max(abs(action[:6])) < threshold (0=off, try 5e-4)", + ) + parser.add_argument( + "--save_dirty", + action="store_true", + help="Also create the dirty (flagged) split (default: off)", + ) + parser.add_argument("--filter_only", action="store_true", help="Only run filter step, skip split") + args = parser.parse_args() + + output_dir = args.output_json_dir or os.path.dirname(os.path.abspath(__file__)) + + print("=" * 80) + print("Step 1: Filter tasks") + print("=" * 80) + clean_json, flagged_json = filter_tasks(args.root, output_dir) + + if args.static_threshold > 0 and args.filter_only: + print() + print("=" * 80) + print("Step 2: Static frame detection (dry-run)") + print("=" * 80) + with open(flagged_json) as f: + flagged_indices = set() + for t in json.load(f)["tasks"]: + flagged_indices.update(t["task_indices"]) + ep_task_map = build_episode_task_map(args.root) + clean_eps = {ep for ep, ti in ep_task_map.items() if ti not in flagged_indices} + _, details = compute_static_trims(args.root, clean_eps, args.static_threshold) + report_path = os.path.join(output_dir, "bridge_static_trims.json") + save_static_report(report_path, args.static_threshold, details) + + if not args.filter_only: + print() + print("=" * 80) + print("Step 2: Split dataset") + print("=" * 80) + output_base = args.output_data_dir if args.output_data_dir else os.path.dirname(args.root.rstrip("/")) + split_dataset( + args.root, + flagged_json, + output_base, + output_dir, + args.output_name, + args.static_threshold, + args.save_dirty, + ) + + print("\nDone!") + + +if __name__ == "__main__": + main() diff --git a/cosmos_framework/data/vfm/action/fractal.py b/cosmos_framework/data/vfm/action/fractal.py new file mode 100644 index 0000000..4547398 --- /dev/null +++ b/cosmos_framework/data/vfm/action/fractal.py @@ -0,0 +1,195 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# Fractal (fractal20220817_data) — Google Robot RT-1 dataset +# LeRobot v2.0 format from IPEC-COMMUNITY/fractal20220817_data_lerobot +# +# Robot: google_robot +# 87,212 episodes, 3,786,400 frames, 599 tasks, fps=3 +# state: [x, y, z, rx, ry, rz, rw, gripper] (8D, quaternion) +# action: [x, y, z, roll, pitch, yaw, gripper] (7D, delta) +# video: observation.images.image (256×320) + +from typing import Any, cast + +import numpy as np +import torch +from lerobot.datasets.lerobot_dataset import LeRobotDatasetMetadata + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.cosmos3_action_lerobot import ( + ActionNormalization, + ActionSpec, + BaseActionLeRobotDataset, + Gripper, + Pos, + Rot, + build_action_spec, +) +from cosmos_framework.data.vfm.action.pose_utils import ( + PoseConvention, + build_abs_pose_from_components, + pose_abs_to_rel, +) +from cosmos_framework.data.vfm.action.viewpoint_utils import Viewpoint + +_VALID_POSE_CONVENTIONS = ("backward_anchored", "backward_framewise") +# These episodes contain base motion, which breaks the fixed-base Google Robot +# action assumption used by training and the viewer. +_SKIPPED_EPISODE_IDS: frozenset[int] = frozenset({29, 189, 382}) + +# Google Robot raw EE frame has x/y axes rotated ~90° around z compared to +# OpenCV convention. Rz(-90°) as a right-multiply corrects this: +# new_x = -old_y (rightward), new_y = old_x (downward), z unchanged (approach). +_GOOGLE_ROBOT_TO_OPENCV = np.array([[0, 1, 0], [-1, 0, 0], [0, 0, 1]], dtype=np.float32) + +# --------------------------------------------------------------------------- +# TCP → flange (gripper body) offset +# --------------------------------------------------------------------------- +# The fractal dataset records EE poses at ``link_gripper_tcp`` — a calibrated +# tool-center-point 164 mm past the gripper body (``link_gripper``), roughly +# at the fingertip. For action learning we re-reference poses to the +# *gripper body* (``link_gripper``) because: +# 1. It is the last actuated link — its pose is fully determined by joint +# angles, whereas the TCP has a tiny calibration-dependent tilt (~0.25°). +# 2. Gripper body is a more natural frame for grasping tasks: the position +# is at the wrist, not at the fragile fingertip. +# 3. The ~10 cm offset reduces the lever-arm effect of small rotation +# errors on position accuracy. +# +# The constant below is the SE(3) transform from ``link_gripper_tcp`` to +# ``link_gripper``, computed from the SimplerEnv URDF via pinocchio FK at the +# neutral configuration: +# T = oMf[link_gripper_tcp]⁻¹ · oMf[link_gripper] +# +# Source URDF: https://github.com/simpler-env/ManiSkill2_real2sim +# → mani_skill2_real2sim/assets/descriptions/google_robot_description/ +# fmt: off +_TCP_TO_FLANGE = np.array([ + [+0.9999897671, -0.0008686425, +0.0044397163, -0.0050618476], + [+0.0008745501, +0.9999987346, -0.0013288658, -0.0016717725], + [-0.0044385564, +0.0013327349, +0.9999892615, -0.1635144743], + [+0.0000000000, +0.0000000000, +0.0000000000, +1.0000000000], +], dtype=np.float32) +# fmt: on + + +class FractalLeRobotDataset(BaseActionLeRobotDataset): + """Action wrapper for the Fractal (Google RT-1) dataset in LeRobot format.""" + + def __init__( + self, + root: str = "", + fps: float = 3.0, + chunk_length: int = 16, + split_seed: int = 42, + split_val_ratio: float = 0.05, + split: str = "train", + mode: str = "policy", + pose_convention: PoseConvention = "backward_framewise", + action_normalization: ActionNormalization | None = None, + viewpoint: Viewpoint = "ego_view", + enable_fast_init: bool = False, + ) -> None: + """Initialize FractalLeRobotDataset. + + Args: + root: Path to the local LeRobot dataset root. + fps: Frames per second of the dataset. + chunk_length: Number of action frames per sample. + split_seed: Seed for deterministic train/val splitting. + split_val_ratio: Fraction of episodes held out for validation. + split: One of "train", "val", or "full". + mode: Training mode — "policy", "forward_dynamics", + "inverse_dynamics", "image2video", or "joint". + pose_convention: Relative-pose convention used to encode SE(3) + actions. Supports ``"backward_framewise"`` and + ``"backward_anchored"``. Set to ``None`` to disable action + construction outside image-to-video mode. + action_normalization: Optional bundled-stats normalization + (``"quantile"`` / ``"quantile_rot"`` / ``"meanstd"`` / ``"minmax"``); + ``None`` returns raw actions. + viewpoint: Camera viewpoint type for this dataset. + """ + super().__init__( + fps=fps, + chunk_length=chunk_length, + split_seed=split_seed, + split_val_ratio=split_val_ratio, + split=split, + mode=mode, + embodiment_type="fractal", + viewpoint=viewpoint, + pose_convention=pose_convention, + rotation_format="rot6d", + action_normalization=action_normalization, + tolerance_s=1e-4, + enable_fast_init=enable_fast_init, + ) + + self._to_opencv = _GOOGLE_ROBOT_TO_OPENCV + self._all_shard_roots = [root] + + self._delta_timestamps = { + "observation.images.image": [i * self._dt for i in range(0, self._chunk_length + 1)], + "observation.state": [i * self._dt for i in range(0, self._chunk_length + 1)], + "action": [i * self._dt for i in range(0, self._chunk_length)], + } + + def _filter_valid_episodes(self, meta: LeRobotDatasetMetadata, episode_ids: list[int]) -> list[int]: + """Drop known-bad raw Fractal episode IDs before index spans are built.""" + kept = [ep_id for ep_id in episode_ids if ep_id not in _SKIPPED_EPISODE_IDS] + dropped = len(episode_ids) - len(kept) + if dropped: + log.info( + f"FractalLeRobotDataset: dropped {dropped} / {len(episode_ids)} " + f"episodes from skip list {sorted(_SKIPPED_EPISODE_IDS)} " + f"(total_episodes={meta.total_episodes})" + ) + return kept + + def _build_action_spec(self) -> ActionSpec: + """Fractal: 10D = ``[Pos(3), Rot6d(6), Gripper(1)]``.""" + return build_action_spec(Pos(dim=3), Rot("rot6d"), Gripper()) + + def __getitem__(self, idx: int) -> dict[str, Any]: + """Return a single training sample.""" + mode, _, _, sample = self._fetch_sample(idx) + + ai_caption = sample["task"] + + video = sample["observation.images.image"] # [T,C,H,W] + + # State layout: [x, y, z, rx, ry, rz, rw, gripper] (T+1 frames) + # Quaternion order from dataset: (rx, ry, rz, rw) matches scipy's (x, y, z, w). + state = sample["observation.state"] # [T+1, 8] + poses_abs = build_abs_pose_from_components( + state[:, 0:3], + state[:, 3:7], + "quat_xyzw", + ) + # 1. TCP → flange: shift from link_gripper_tcp to link_gripper + poses_abs = poses_abs @ _TCP_TO_FLANGE + # 2. Kinematics → OpenCV convention (rotation only) + poses_abs[:, :3, :3] = poses_abs[:, :3, :3] @ self._to_opencv + initial_pose = torch.from_numpy(poses_abs[0].copy()).float() + poses_rel = pose_abs_to_rel( + poses_abs, + rotation_format="rot6d", + pose_convention=cast(PoseConvention, self._pose_convention), + ) + action = torch.cat( + [ + torch.from_numpy(poses_rel).float(), # SE3 relative pose (rot6d) + sample["action"][:, [6]], # gripper (1D) + ], + dim=-1, + ) # [T, 10] + return self._build_result( + mode=mode, video=video, action=action, ai_caption=ai_caption, initial_pose=initial_pose + ) + + @property + def action_dim(self) -> int: + """Action dimensionality: position(3) + 6D rotation(6) + gripper(1) = 10.""" + return 10 diff --git a/cosmos_framework/data/vfm/action/hand_pose_dataset.py b/cosmos_framework/data/vfm/action/hand_pose_dataset.py new file mode 100644 index 0000000..8394ed5 --- /dev/null +++ b/cosmos_framework/data/vfm/action/hand_pose_dataset.py @@ -0,0 +1,1347 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Hand-pose manipulation dataset. + +A bimanual human hand dataset in LeRobot v3 format with 21-keypoint hand pose +annotations (positions + per-joint quaternion rotations) in camera space, +along with per-frame camera pose (position + quaternion rotation), task/subtask +labels and RGB video. + +Action layout: ``[camera, R_wrist, R_fingers, L_wrist, L_fingers]`` — camera +pose followed by right-hand then left-hand components. + +Action space (``pose_convention``) +------------------------------- +Both action spaces share the same three-stage computation: + + 1. Compute **absolute** SE(3) poses for camera and both wrists. + 2. Compute **finger positions** in the per-frame wrist coordinate frame. + 3. Convert absolute camera/wrist poses to **relative** representations + (anchored or frame-wise). + +Layout (both modes): ``[camera, R_wrist, R_fingers, L_wrist, L_fingers]`` + + ``backward_anchored`` + Camera and wrist poses anchored to frame 0: + ``P_{0}^{-1} @ P_{t}`` for camera and each wrist. + Fingers are positions in the current wrist frame. + + ``backward_framewise`` + Frame-wise SE(3) deltas: + ``P_{t-1}^{-1} @ P_{t}`` for camera and each wrist. + Fingers are positions in the current wrist frame. + +Action dimensions +~~~~~~~~~~~~~~~~~ +Both modes have the same dimensionality: + - Camera: ``3 + rot_dim`` + - Per hand (×2): wrist ``(3 + rot_dim)`` + fingers ``(N_finger × 3)`` + - Total: ``(3 + rot_dim) + 2 × ((3 + rot_dim) + N_finger × 3)`` + +Example with ``rot6d`` rotation, ``wrist_plus_finger_tips`` (5 fingertips): + - Camera: 3 + 6 = 9D + - Per hand: wrist (3 + 6 = 9D) + fingers (5 × 3 = 15D) = 24D + - Total: 9 + 24 + 24 = **57D** + +Rotation format (``rotation_format``) +-------------------------------------- +Applied uniformly to both hand joint rotations and camera ego-motion rotation: + - ``rot9d``: flattened 3x3 rotation matrix (default, converted from quaternions) + - ``rot6d``: first 2 columns of the rotation matrix (continuous, Zhou et al. CVPR 2019) + - ``euler_xyz``: Euler ``xyz`` angles in radians +""" + +import os +import random +from bisect import bisect_right +from pathlib import Path +from typing import Any, Literal + +import numpy as np +import pandas as pd +import torch + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.cosmos3_action_lerobot import ( + ActionNormalization, + BaseActionLeRobotDataset, + _parallel_map, + build_episode_spans, + split_episode_ids, +) +from cosmos_framework.data.vfm.action.hand_pose_dataset_config import ( + CAM_POSITION_KEY, + CAM_ROTATION_KEY, + FINGERTIP_JOINT_IDXS, + HAND_LEFT_POSITION_KEY, + HAND_LEFT_ROTATION_KEY, + HAND_POSE_DATASETS, + HAND_RIGHT_POSITION_KEY, + HAND_RIGHT_ROTATION_KEY, + NO_ACTION_SKIP_LABEL_PREFIXES, + NO_ACTION_SKIP_LABEL_SUBSTRINGS, + NO_ACTION_SKIP_LABELS, + NUM_JOINTS, + QUAT_DIM_PER_JOINT, + ROTATION_FORMAT_DIM, + WRIST_FRAME_ALIGN_EMBODIMENT_A, + WRIST_JOINT_IDX, +) +from cosmos_framework.data.vfm.action.pose_utils import ( + RotationConvention, + build_abs_pose_from_components, + pose_abs_to_rel, +) +from cosmos_framework.data.vfm.action.viewpoint_utils import Viewpoint + + +class HandPoseDataset(BaseActionLeRobotDataset): + """Hand-pose manipulation dataset backed by LeRobot v3. + + Each sample returns a video chunk and the corresponding hand-pose action + representation. Uses deferred source registration via + ``BaseActionLeRobotDataset``. + """ + + def __init__( + self, + root: str | list[str] = HAND_POSE_DATASETS["embodiment_a_feb08_500hr"], + fps: float = 15.0, + chunk_length: int = 16, + split_seed: int = 42, + split_val_ratio: float = 0.005, + split: str = "train", + mode: str = "policy", + embodiment_type: str = "hand_pose", + video_key: str = "observation.images.main", + keypoint_option: Literal[ + "wrist_only", "wrist_plus_fingers", "wrist_plus_finger_tips" + ] = "wrist_plus_finger_tips", + rotation_format: RotationConvention = "rot6d", + pose_convention: Literal[ + "backward_anchored", + "backward_framewise", + ] = "backward_framewise", + action_normalization: ActionNormalization | None = None, + intra_episode_val_ratio: float = 0.0, + tolerance_s: float = 2e-4, + drop_unannotated_edge_frames: bool = True, + unannotated_pos_l1_threshold: float = 1e-6, + max_item_retries: int = 16, + return_overlay_data: bool = False, + max_episodes: int | None = None, + episode_ids: list[int] | None = None, + load_subtasks: bool = False, + snap_to_subtask: bool = False, + skip_no_action: bool = False, + max_subtasks_per_episode: int | None = None, + viewpoint: Viewpoint = "ego_view", + enable_fast_init: bool = False, + ) -> None: + super().__init__( + fps=fps, + chunk_length=chunk_length, + split_seed=split_seed, + split_val_ratio=split_val_ratio, + split=split, + mode=mode, + embodiment_type=embodiment_type, + viewpoint=viewpoint, + pose_convention=pose_convention, + rotation_format=rotation_format, + action_normalization=action_normalization, + tolerance_s=tolerance_s, + enable_fast_init=enable_fast_init, + ) + if max_episodes is not None and episode_ids is not None: + raise ValueError("Cannot specify both max_episodes and episode_ids.") + self._max_episodes = max_episodes + self._episode_ids = episode_ids + + self._video_key = video_key + self._load_subtasks = load_subtasks or skip_no_action or snap_to_subtask + self._snap_to_subtask = snap_to_subtask + self._skip_no_action = skip_no_action + self._max_subtasks_per_episode = max_subtasks_per_episode + self._subtask_transitions: dict[tuple[int, int], list[tuple[int, int]]] = {} + self._per_shard_orig_subtask_names: dict[int, dict[int, str]] = {} + self._ds_idx_to_shard_idx: list[int] | None = None + self._orig_subtask_index: dict[int, np.ndarray] = {} + self._cached_subtask_col: dict[int, np.ndarray] = {} + self._orig_subtask_transitions: dict[tuple[int, int], list[tuple[int, int]]] = {} + self._subtask_start_indices: dict[tuple[int, int], list[tuple[int, int]]] = {} + self._keypoint_option = keypoint_option + self._rotation_format = rotation_format + self._pose_convention = pose_convention + self._intra_episode_val_ratio = intra_episode_val_ratio + self._drop_unannotated_edge_frames = drop_unannotated_edge_frames + self._unannotated_pos_l1_threshold = unannotated_pos_l1_threshold + self._max_item_retries = max_item_retries + self._warned_tolerance_once = False + self._warned_unannotated_once = False + self._warned_nan_action_once = False + self._warned_build_action_error_once = False + self._logged_decode_failures: set[tuple[int, int, int]] = set() + self._return_overlay_data = return_overlay_data + + # Normalize root: config may pass OmegaConf ListConfig for list of shards; ensure list of str. + if isinstance(root, str): + _shard_roots = [root] + else: + _shard_roots = [str(p) for p in root] + + self._root = _shard_roots[0] + self._all_shard_roots = _shard_roots + + # Hand pose keys (camera-space positions + rotations). + self._position_keys = [HAND_LEFT_POSITION_KEY, HAND_RIGHT_POSITION_KEY] + self._rotation_keys = [HAND_LEFT_ROTATION_KEY, HAND_RIGHT_ROTATION_KEY] + + if keypoint_option not in {"wrist_only", "wrist_plus_fingers", "wrist_plus_finger_tips"}: + raise ValueError(f"Unsupported keypoint_option: {keypoint_option!r}") + if not (0.0 <= intra_episode_val_ratio < 1.0): + raise ValueError(f"intra_episode_val_ratio must be in [0, 1). Got: {intra_episode_val_ratio}") + + if keypoint_option == "wrist_plus_finger_tips": + self._position_joint_indices: tuple[int, ...] = (WRIST_JOINT_IDX, *FINGERTIP_JOINT_IDXS) + elif keypoint_option == "wrist_plus_fingers": + self._position_joint_indices = tuple(range(NUM_JOINTS)) + else: + self._position_joint_indices = (WRIST_JOINT_IDX,) + + # Compute raw action dimension. + # Layout: [camera, R_wrist, R_fingers, L_wrist, L_fingers] + # Camera: 3 (pos) + rot_dim. Per hand: 3 (wrist pos) + rot_dim + (N_finger × 3). + rot_dim_per_joint = ROTATION_FORMAT_DIM[rotation_format] + num_pos_joints = len(self._position_joint_indices) + num_finger_joints = num_pos_joints - 1 # exclude wrist + per_hand_dim = 3 + rot_dim_per_joint + num_finger_joints * 3 + self._raw_action_dim = (3 + rot_dim_per_joint) + 2 * per_hand_dim + + # Build delta_timestamps: T+1 video frames, T+1 hand states. + ts = [i * self._dt for i in range(self._chunk_length + 1)] + self._delta_timestamps = { + self._video_key: list(ts), + } + for key in ( + HAND_LEFT_POSITION_KEY, + HAND_RIGHT_POSITION_KEY, + HAND_LEFT_ROTATION_KEY, + HAND_RIGHT_ROTATION_KEY, + ): + self._delta_timestamps[key] = list(ts) + + # Always load camera pose (both action spaces include camera). + self._delta_timestamps[CAM_POSITION_KEY] = list(ts) + self._delta_timestamps[CAM_ROTATION_KEY] = list(ts) + + self._episode_intrinsics: dict[int, np.ndarray] = {} + if return_overlay_data: + self._load_episode_intrinsics(self._all_shard_roots[0]) + + # Load task/subtask label mappings for captions. + self._task_names: dict[int, str] = {} + self._per_shard_subtask_names: dict[int, dict[int, str]] = {} + self._load_task_labels() + + _POSE_CONVENTION_DESC = { + "backward_anchored": ( + "Camera + wrist poses anchored to frame 0, fingers in wrist frame. " + "Layout: [camera, R_wrist, R_fingers, L_wrist, L_fingers]" + ), + "backward_framewise": ( + "Frame-wise camera + wrist deltas, fingers in wrist frame. " + "Layout: [camera, R_wrist, R_fingers, L_wrist, L_fingers]" + ), + } + log.info( + f"HandPoseDataset configuration:\n" + f" root = {root}\n" + f" split = {split}\n" + f" mode = {mode}\n" + f" pose_convention = {pose_convention}\n" + f" -> {_POSE_CONVENTION_DESC.get(pose_convention, 'unknown')}\n" + f" intra_episode_val_ratio = {intra_episode_val_ratio}\n" + f" position_keys = {self._position_keys}\n" + f" rotation_keys = {self._rotation_keys}\n" + f" keypoint_option = {keypoint_option}\n" + f" selected_joint_count = {num_pos_joints} ({num_finger_joints} finger joints)\n" + f" rotation_format = {rotation_format} ({rot_dim_per_joint}D per joint)\n" + f" raw_action_dim = {self._raw_action_dim} " + f"(cam {3 + rot_dim_per_joint}D + 2 × hand {per_hand_dim}D)\n" + f" chunk_length = {chunk_length} (video frames: {chunk_length + 1})\n" + f" fps = {fps}\n" + f" tolerance_s = {tolerance_s}\n" + f" drop_unannotated_edge_frames = {drop_unannotated_edge_frames}\n" + f" unannotated_pos_l1_threshold = {unannotated_pos_l1_threshold}\n" + f" max_item_retries = {max_item_retries}\n" + f" max_episodes = {max_episodes}\n" + f" episode_ids = {episode_ids}\n" + f" snap_to_subtask = {snap_to_subtask}\n" + f" skip_no_action = {skip_no_action}\n" + f" max_subtasks_per_episode = {self._max_subtasks_per_episode}\n" + f" domain_id = {self._domain_id} ({embodiment_type})" + ) + + # ------------------------------------------------------------------------- + # Per-episode subtask transition cache + # ------------------------------------------------------------------------- + + def _get_subtask_transitions(self, ds_idx: int, ep_idx: int) -> list[tuple[int, int]]: + """Return sorted ``(row_start, subtask_index)`` transitions for an episode. + + Built lazily on first access by slicing the ``subtask_index`` column + from the LeRobot HF dataset (cheap Arrow column access). + """ + key = (ds_idx, ep_idx) + cached = self._subtask_transitions.get(key) + if cached is not None: + return cached + + ds = self._get_dataset(ds_idx) + ep = ds.meta.episodes[ep_idx] + ep_start: int = ep["dataset_from_index"] + ep_end: int = ep["dataset_to_index"] + + if ds._absolute_to_relative_idx is not None: + rel_indices = [ds._absolute_to_relative_idx[i] for i in range(ep_start, ep_end)] + subtask_col = ds.hf_dataset["subtask_index"][rel_indices] + else: + subtask_col = ds.hf_dataset["subtask_index"][ep_start:ep_end] + + transitions: list[tuple[int, int]] = [] + prev_si = None + for offset, si_val in enumerate(subtask_col): + si = int(si_val) if not isinstance(si_val, int) else si_val + if si != prev_si: + transitions.append((ep_start + offset, si)) + prev_si = si + + self._subtask_transitions[key] = transitions + return transitions + + # ------------------------------------------------------------------------- + # Original (non-augcap) subtask data for "No action" filtering + # ------------------------------------------------------------------------- + + def _load_orig_subtask_index(self, ds_idx: int) -> np.ndarray: + """Lazily load the original subtask_index column for a dataset shard. + + Reads parquet files from the shard's ``data/`` directory and extracts + the ``subtask_index`` column as a flat numpy array. + """ + if ds_idx in self._orig_subtask_index: + return self._orig_subtask_index[ds_idx] + + build_args = self._dataset_build_args[ds_idx] + if build_args is None: + self._orig_subtask_index[ds_idx] = np.array([], dtype=np.int64) + return self._orig_subtask_index[ds_idx] + + shard_root: str | None = build_args["root"] + if shard_root is None: + self._orig_subtask_index[ds_idx] = np.array([], dtype=np.int64) + return self._orig_subtask_index[ds_idx] + data_dir = Path(shard_root) / "data" + parquet_files = sorted(data_dir.rglob("*.parquet")) + dfs = [pd.read_parquet(str(f), columns=["subtask_index"]) for f in parquet_files] + if dfs: + arr = pd.concat(dfs, ignore_index=True)["subtask_index"].to_numpy(dtype=np.int64) + else: + arr = np.array([], dtype=np.int64) + + self._orig_subtask_index[ds_idx] = arr + log.info(f"HandPoseDataset: loaded {len(arr)} original subtask indices for ds_idx={ds_idx}") + return arr + + def _get_orig_subtask_transitions(self, ds_idx: int, ep_idx: int) -> list[tuple[int, int]]: + """Return sorted ``(row_start, subtask_index)`` transitions from the original data. + + Reads subtask indices from the shard root's ``data/`` parquets using + the same episode boundaries as the loaded dataset. + """ + key = (ds_idx, ep_idx) + cached = self._orig_subtask_transitions.get(key) + if cached is not None: + return cached + + orig_indices = self._load_orig_subtask_index(ds_idx) + if len(orig_indices) == 0: + self._orig_subtask_transitions[key] = [] + return [] + + ds = self._get_dataset(ds_idx) + ep = ds.meta.episodes[ep_idx] + ep_start: int = ep["dataset_from_index"] + ep_end: int = ep["dataset_to_index"] + + subtask_col = orig_indices[ep_start:ep_end] + + transitions: list[tuple[int, int]] = [] + prev_si = None + for offset, si_val in enumerate(subtask_col): + si = int(si_val) + if si != prev_si: + transitions.append((ep_start + offset, si)) + prev_si = si + + self._orig_subtask_transitions[key] = transitions + return transitions + + def _is_no_action(self, subtask_index: int, ds_idx: int = 0) -> bool: + """Return ``True`` if the original subtask name indicates an idle segment. + + Matches the name (normalized: ``_`` → space, stripped, lowercased) + against three rule sets from ``hand_pose_dataset_config``: + exact ``NO_ACTION_SKIP_LABELS``, ``NO_ACTION_SKIP_LABEL_PREFIXES``, or + ``NO_ACTION_SKIP_LABEL_SUBSTRINGS``. + """ + name = self._get_orig_subtask_names_for_ds(ds_idx).get(subtask_index, "") + normalized = name.replace("_", " ").strip().lower() + if normalized in NO_ACTION_SKIP_LABELS: + return True + if NO_ACTION_SKIP_LABEL_PREFIXES and normalized.startswith(NO_ACTION_SKIP_LABEL_PREFIXES): + return True + if any(sub in normalized for sub in NO_ACTION_SKIP_LABEL_SUBSTRINGS): + return True + return False + + def _skip_no_action_subtask(self, ds_idx: int, row_idx: int, ep_idx: int) -> int | None: + """Advance ``row_idx`` past any original "No action" subtask. + + Returns: + The (possibly advanced) ``row_idx``, or ``None`` if every + remaining subtask in the episode is "No action". + """ + orig_transitions = self._get_orig_subtask_transitions(ds_idx, ep_idx) + if not orig_transitions: + return row_idx + + row_starts = [t[0] for t in orig_transitions] + ti = bisect_right(row_starts, row_idx) - 1 + if ti < 0: + return row_idx + + _, orig_si = orig_transitions[ti] + if not self._is_no_action(orig_si, ds_idx): + return row_idx + + for next_ti in range(ti + 1, len(orig_transitions)): + _, next_si = orig_transitions[next_ti] + if not self._is_no_action(next_si, ds_idx): + return orig_transitions[next_ti][0] + + return None + + # ------------------------------------------------------------------------- + # Build-time subtask-level reindexing for uniform subtask sampling + # ------------------------------------------------------------------------- + + def _transitions(self, col: np.ndarray, ep_start: int) -> list[tuple[int, int]]: + """``(row_start, value)`` pairs at each change point (first row always emitted). + + Vectorized when ``enable_fast_init=True``, else the original Python + ``enumerate`` loop. + """ + if self._enable_fast_init: + if len(col) == 0: + return [] + mask = np.concatenate(([True], col[1:] != col[:-1])) + idx = np.flatnonzero(mask) + return list(zip((ep_start + idx).tolist(), col[idx].astype(np.int64).tolist())) + transitions: list[tuple[int, int]] = [] + prev: int | None = None + for offset, val in enumerate(col): + v = int(val) + if v != prev: + transitions.append((ep_start + offset, v)) + prev = v + return transitions + + def _rebuild_snap_indices(self, ds_idx: int, meta: Any, records_before: int) -> None: + """Replace frame-level indices with subtask-level indices. + + When ``snap_to_subtask`` is active, the default frame-level indexing + biases sampling toward longer subtasks. This method replaces the + episode records with subtask-level records so that each subtask gets + exactly one index, yielding uniform sampling over subtasks. + + When ``skip_no_action`` is also active, subtasks whose original label + starts with "No action" are excluded at build time. + """ + build_args = self._dataset_build_args[ds_idx] + if build_args is None: + return + shard_root: str | None = build_args["root"] + if shard_root is None: + return + + if self._enable_fast_init: + subtask_col = self._cached_subtask_col[ds_idx] + else: + data_dir = Path(shard_root) / "data" + parquet_files = sorted(data_dir.rglob("*.parquet")) + dfs = [pd.read_parquet(str(f), columns=["subtask_index"]) for f in parquet_files] + if not dfs: + return + subtask_col = pd.concat(dfs, ignore_index=True)["subtask_index"].to_numpy(dtype=np.int64) + if len(subtask_col) == 0: + return + + orig_subtask_col: np.ndarray | None = None + if self._skip_no_action: + orig_subtask_col = self._load_orig_subtask_index(ds_idx) + + fps_ratio = round(meta.fps / self._fps) + + ep_from = list(meta.episodes["dataset_from_index"]) + ep_to = list(meta.episodes["dataset_to_index"]) + + new_records: list[tuple[int, int, int, int]] = [] + total_subtasks = 0 + total_skipped_na = 0 + total_skipped_short = 0 + + for rec in self._episode_records[records_before:]: + rec_ds_idx, sample_start, _old_valid_len, episode_id = rec + assert rec_ds_idx == ds_idx + + ep_start: int = ep_from[episode_id] + ep_end: int = ep_to[episode_id] + + transitions = self._transitions(subtask_col[ep_start:ep_end], ep_start) + + orig_trans: list[tuple[int, int]] | None = None + orig_row_starts: list[int] | None = None + if orig_subtask_col is not None and len(orig_subtask_col) > 0: + orig_trans = self._transitions(orig_subtask_col[ep_start:ep_end], ep_start) + orig_row_starts = [t[0] for t in orig_trans] + + subtask_starts: list[tuple[int, int]] = [] + for i, (row_start, _si) in enumerate(transitions): + if i + 1 < len(transitions): + native_len = transitions[i + 1][0] - row_start + else: + native_len = ep_end - row_start + + if self._get_snapped_video_frame_count(native_len, fps_ratio) == 0: + total_skipped_short += 1 + continue + + if self._skip_no_action and orig_trans is not None and orig_row_starts is not None: + ti = bisect_right(orig_row_starts, row_start) - 1 + if ti >= 0: + _, orig_si = orig_trans[ti] + if self._is_no_action(orig_si, ds_idx): + total_skipped_na += 1 + continue + + subtask_starts.append((row_start, native_len)) + + if self._max_subtasks_per_episode is not None: + subtask_starts = subtask_starts[: self._max_subtasks_per_episode] + + self._subtask_start_indices[(ds_idx, episode_id)] = subtask_starts + + # Pre-populate transition caches so DataLoader workers inherit + # them read-only via COW instead of rebuilding per-worker. + self._subtask_transitions[(ds_idx, episode_id)] = transitions + if orig_trans is not None: + self._orig_subtask_transitions[(ds_idx, episode_id)] = orig_trans + + num_subtasks = len(subtask_starts) + total_subtasks += num_subtasks + if num_subtasks > 0: + new_records.append((ds_idx, sample_start, num_subtasks, episode_id)) + + self._episode_records = self._episode_records[:records_before] + new_records + self._episode_cum_ends = self._episode_cum_ends[:records_before] + self._num_valid_indices = self._episode_cum_ends[-1] if records_before > 0 else 0 + for rec in new_records: + self._num_valid_indices += rec[2] + self._episode_cum_ends.append(self._num_valid_indices) + + cap_info = "" + if self._max_subtasks_per_episode is not None: + cap_info = f", capped at first {self._max_subtasks_per_episode}/ep" + log.info( + f"HandPoseDataset: snap_to_subtask reindex — " + f"{total_subtasks} subtask starts across {len(new_records)} episodes{cap_info} " + f"(skipped {total_skipped_na} no-action, {total_skipped_short} too-short)" + ) + + # ------------------------------------------------------------------------- + # Episode filtering (override base class without modifying it) + # ------------------------------------------------------------------------- + + def _append_index_records(self, *, meta: Any, ds_idx: int, dataset_label: str | None = None) -> None: + """Override to filter episodes after deterministic split selection. + + Supports two mutually exclusive modes: + - ``max_episodes``: keep the first N episodes **globally** across all + shards (not per-shard). This method is called once per shard, so we + track how many episodes have already been kept via + ``len(self._episode_records)`` before and after the base-class call. + - ``episode_ids``: keep only episodes whose dataset-level episode + index appears in the given list. + """ + records_before = len(self._episode_records) + if self._intra_episode_val_ratio > 0.0 and self._split in {"train", "val"}: + episode_ids = split_episode_ids( + total_episodes=meta.total_episodes, + seed=self._split_seed, + val_ratio=self._split_val_ratio, + split="train", + ) + episode_spans, _, sample_count = build_episode_spans( + episodes=meta.episodes, + episode_ids=episode_ids, + chunk_length=self._chunk_length, + ) + + # Prevent train/val leakage caused by overlapping chunk windows. + # A gap of chunk_length sample-starts ensures no frame overlap between splits. + non_overlap_gap = self._chunk_length + valid_count = 0 + for episode_id, sample_start, valid_len in episode_spans: + if valid_len <= 0: + continue + num_val = int(round(valid_len * self._intra_episode_val_ratio)) + val_start_offset = max(0, valid_len - num_val) # take tail for validation + + if self._split == "train": + split_start = sample_start + split_len = max(0, val_start_offset - non_overlap_gap) + else: + split_start = sample_start + val_start_offset + split_len = max(0, valid_len - val_start_offset) + + if split_len <= 0: + continue + + self._episode_records.append((ds_idx, split_start, split_len, episode_id)) + self._num_valid_indices += split_len + self._episode_cum_ends.append(self._num_valid_indices) + valid_count += split_len + + class_name = self.__class__.__name__ + label = f" [{dataset_label}]" if dataset_label else "" + log.info( + f"{class_name}{label}: intra-episode split enabled " + f"(ratio={self._intra_episode_val_ratio:.3f}, split={self._split}, gap={non_overlap_gap})" + ) + if sample_count > 0: + log.info( + f"{class_name}{label}: kept {valid_count} / {sample_count} " + f"({100 * valid_count / sample_count:.2f} %) samples" + ) + else: + super()._append_index_records(meta=meta, ds_idx=ds_idx, dataset_label=dataset_label) + + new_records = self._episode_records[records_before:] + if not new_records: + return + + if self._episode_ids is not None: + keep_set = set(self._episode_ids) + kept: list[tuple[int, int, int, int]] = [] + removed_frames = 0 + for rec in new_records: + if rec[3] in keep_set: + kept.append(rec) + else: + removed_frames += rec[2] + self._episode_records = self._episode_records[:records_before] + kept + self._num_valid_indices -= removed_frames + self._episode_cum_ends = self._episode_cum_ends[:records_before] + running = self._episode_cum_ends[-1] if records_before > 0 else 0 + for rec in kept: + running += rec[2] + self._episode_cum_ends.append(running) + kept_ids = [rec[3] for rec in kept] + log.info( + f"HandPoseDataset: episode_ids filter — " + f"kept {len(kept)}/{len(new_records)} episodes " + f"({self._num_valid_indices} valid indices), " + f"retained episode IDs: {kept_ids}" + ) + elif self._max_episodes is not None: + global_total = len(self._episode_records) + remaining_budget = self._max_episodes - records_before + if remaining_budget <= 0: + self._episode_records = self._episode_records[:records_before] + self._episode_cum_ends = self._episode_cum_ends[:records_before] + removed_frames = sum(rec[2] for rec in new_records) + self._num_valid_indices -= removed_frames + log.info( + f"HandPoseDataset: max_episodes={self._max_episodes} — " + f"global budget exhausted, dropped all {len(new_records)} episodes " + f"from shard ds_idx={ds_idx}" + ) + elif len(new_records) > remaining_budget: + keep = records_before + remaining_budget + removed = self._episode_records[keep:] + self._episode_records = self._episode_records[:keep] + self._episode_cum_ends = self._episode_cum_ends[:keep] + removed_frames = sum(rec[2] for rec in removed) + self._num_valid_indices -= removed_frames + if keep > 0: + self._episode_cum_ends[-1] = self._num_valid_indices + + retained = self._episode_records[records_before:keep] + retained_ids = [rec[3] for rec in retained] + retained_frames = sum(rec[2] for rec in retained) + log.info( + f"HandPoseDataset: max_episodes={self._max_episodes} — " + f"shard ds_idx={ds_idx}: kept {remaining_budget}/{len(new_records)} episodes, " + f"removed {removed_frames} frames, " + f"retained {retained_frames} valid indices, " + f"episode IDs: {retained_ids}, " + f"global total: {len(self._episode_records)} episodes" + ) + else: + retained_ids = [rec[3] for rec in new_records] + log.info( + f"HandPoseDataset: max_episodes={self._max_episodes} — " + f"shard ds_idx={ds_idx}: kept all {len(new_records)} episodes " + f"(global total: {len(self._episode_records)}/{self._max_episodes}), " + f"episode IDs: {retained_ids}" + ) + else: + all_ids = [rec[3] for rec in new_records] + log.info( + f"HandPoseDataset: using all {len(new_records)} episodes ({self._num_valid_indices} valid indices)" + ) + + if self._snap_to_subtask: + self._rebuild_snap_indices(ds_idx=ds_idx, meta=meta, records_before=records_before) + + # ------------------------------------------------------------------------- + # Action building + # ------------------------------------------------------------------------- + + def _build_action(self, sample: dict[str, Any]) -> torch.Tensor: + """Build the action tensor from a LeRobot sample. + + Both modes share the same three-stage pipeline: + 1. Compute absolute SE(3) poses: camera (already world-frame), + wrists (converted from per-frame camera space to world space + via ``P_world = P_c2w @ P_cam``). + 2. Compute finger positions in per-frame wrist frame (valid in + camera space since both are in the same frame at each timestep). + 3. Convert world-frame absolute poses to relative (anchored or + frame-wise) for camera and wrists. + + Layout: ``[camera, R_wrist, R_fingers, L_wrist, L_fingers]`` + + Returns: + Action tensor of shape ``(T, raw_action_dim)``. + """ + pose_convention: Literal["backward_anchored", "backward_framewise"] = self._pose_convention # type: ignore[assignment] + + def _to_np(t: torch.Tensor) -> np.ndarray: + return t.detach().cpu().numpy() + + # -- Stage 1: absolute SE(3) poses ------------------------------------ + # Camera pose (world frame): c2w transforms + cam_pos = _to_np(sample[CAM_POSITION_KEY]) # (T+1, 3) + cam_rot_q = _to_np(sample[CAM_ROTATION_KEY]) # (T+1, 4) + cam_c2w = build_abs_pose_from_components(cam_pos, cam_rot_q, "quat_xyzw") # (T+1, 4, 4) + + # Wrist poses in camera frame (right first, then left) + right_pos_all = _to_np(sample[self._position_keys[1]]) # (T+1, 63) — hand_right_cam + right_rot_all = _to_np(sample[self._rotation_keys[1]]) # (T+1, 84) + left_pos_all = _to_np(sample[self._position_keys[0]]) # (T+1, 63) — hand_left_cam + left_rot_all = _to_np(sample[self._rotation_keys[0]]) # (T+1, 84) + + right_wrist_pos, right_wrist_quat = self._extract_wrist_pose_components(right_pos_all, right_rot_all) + left_wrist_pos, left_wrist_quat = self._extract_wrist_pose_components(left_pos_all, left_rot_all) + + # Wrist poses in camera frame, aligned to the unified cross-domain convention. + right_wrist_cam = build_abs_pose_from_components(right_wrist_pos, right_wrist_quat, "quat_xyzw") + left_wrist_cam = build_abs_pose_from_components(left_wrist_pos, left_wrist_quat, "quat_xyzw") + + if "embodiment_a" in self._root.lower(): + right_wrist_cam = right_wrist_cam @ WRIST_FRAME_ALIGN_EMBODIMENT_A + left_wrist_cam = left_wrist_cam @ WRIST_FRAME_ALIGN_EMBODIMENT_A + + # Wrist poses in camera frame → world frame: P_world = P_c2w @ P_cam + right_wrist_world = cam_c2w @ right_wrist_cam # (T+1, 4, 4) + left_wrist_world = cam_c2w @ left_wrist_cam # (T+1, 4, 4) + + # -- Stage 2: finger positions in per-frame wrist frame --------------- + # (Correct as-is: both finger positions and wrist pose are in the same + # camera frame at each timestep, so the transform to wrist-local is valid. + # The alignment rotation only changes the wrist-local axes; finger positions + # in camera space are unchanged, so their wrist-local coordinates rotate + # correspondingly to match the unified convention.) + right_fingers = self._build_fingers_in_wrist_frame(right_pos_all, right_wrist_cam) + left_fingers = self._build_fingers_in_wrist_frame(left_pos_all, left_wrist_cam) + + # -- Stage 3: convert world-frame absolute → relative poses ----------- + cam_rel = pose_abs_to_rel(cam_c2w, rotation_format=self._rotation_format, pose_convention=pose_convention) + right_wrist_rel = pose_abs_to_rel( + right_wrist_world, + rotation_format=self._rotation_format, + pose_convention=pose_convention, + ) + left_wrist_rel = pose_abs_to_rel( + left_wrist_world, + rotation_format=self._rotation_format, + pose_convention=pose_convention, + ) + + # -- Assemble: [camera, R_wrist, R_fingers, L_wrist, L_fingers] ------- + return torch.from_numpy( + np.concatenate([cam_rel, right_wrist_rel, right_fingers, left_wrist_rel, left_fingers], axis=-1) + ).float() + + # -- Shared action helpers ------------------------------------------------ + + def _extract_wrist_pose_components( + self, + pos_data: np.ndarray, + rot_quat: np.ndarray, + ) -> tuple[np.ndarray, np.ndarray]: + """Extract wrist translation and wrist quaternion from full-hand arrays.""" + wrist_pos = pos_data[:, :3] + wrist_quat = rot_quat.reshape(pos_data.shape[0], NUM_JOINTS, QUAT_DIM_PER_JOINT)[:, WRIST_JOINT_IDX] + return wrist_pos, wrist_quat + + def _build_fingers_in_wrist_frame( + self, + pos_data: np.ndarray, + wrist_poses_abs: np.ndarray, + ) -> np.ndarray: + """Express selected non-wrist keypoints in the per-frame wrist coordinate frame.""" + future_pos = pos_data[1:].astype(np.float32, copy=False) + T = future_pos.shape[0] + + selected_non_wrist = [j for j in self._position_joint_indices if j != WRIST_JOINT_IDX] + if not selected_non_wrist: + return np.empty((T, 0), dtype=np.float32) + + pos_3d = future_pos.reshape(T, NUM_JOINTS, 3) + finger_pos = pos_3d[:, selected_non_wrist, :] + finger_pos_h = np.concatenate( + [finger_pos, np.ones((*finger_pos.shape[:-1], 1), dtype=np.float32)], + axis=-1, + ) + wrist_inv = np.linalg.inv(wrist_poses_abs[1:]) + finger_pos_wrist = np.einsum("tij,tnj->tni", wrist_inv, finger_pos_h)[..., :3] + return finger_pos_wrist.reshape(T, -1) + + def _get_snapped_video_frame_count(self, subtask_native_len: int, fps_ratio: int) -> int: + """Return tokenizer-compatible video frame count for a snapped subtask. + + The tokenizer keeps one conditioning frame and consumes the remaining + temporal dimension in groups of four, so snapped clips must be ``1 + 4N`` + frames long. + """ + capped_video_frames = min(subtask_native_len // fps_ratio, self._chunk_length + 1) + if capped_video_frames < 5: + return 0 + return 1 + 4 * ((capped_video_frames - 1) // 4) + + # ------------------------------------------------------------------------- + # Caption helpers + # ------------------------------------------------------------------------- + + def _load_task_labels(self) -> None: + """Load task and subtask name mappings from meta parquet files. + + Subtask indices are **per-shard** (each shard's indices start at 0 and + the name lists differ across shards), so we must load and store subtask + names independently for every shard to avoid cross-shard caption + mismatch. + """ + if self._enable_fast_init: + self._load_task_labels_fast() + return + + tasks_path = os.path.join(self._all_shard_roots[0], "meta", "tasks.parquet") + if os.path.exists(tasks_path): + tasks_df = pd.read_parquet(tasks_path) + for _, row in tasks_df.iterrows(): + self._task_names[int(row["task_index"])] = str(row.name) + log.info(f"HandPoseDataset: loaded {len(self._task_names)} task labels") + + if self._load_subtasks: + for shard_idx, shard_root in enumerate(self._all_shard_roots): + shard_names: dict[int, str] = {} + plain_path = os.path.join(shard_root, "meta", "subtasks.parquet") + + if os.path.exists(plain_path): + df = pd.read_parquet(plain_path) + for _, row in df.iterrows(): + shard_names[int(row["subtask_index"])] = str(row.name) + log.info(f"HandPoseDataset: shard {shard_idx}: loaded {len(shard_names)} subtask labels") + + self._per_shard_subtask_names[shard_idx] = shard_names + + if self._skip_no_action: + for shard_idx, shard_root in enumerate(self._all_shard_roots): + orig_subtasks_path = os.path.join(shard_root, "meta", "subtasks.parquet") + if os.path.exists(orig_subtasks_path): + shard_names = {} + orig_df = pd.read_parquet(orig_subtasks_path) + for _, row in orig_df.iterrows(): + shard_names[int(row["subtask_index"])] = str(row.name) + self._per_shard_orig_subtask_names[shard_idx] = shard_names + log.info( + f"HandPoseDataset: shard {shard_idx}: loaded {len(shard_names)} " + f"original subtask labels (from {shard_root})" + ) + + def _load_task_labels_fast(self) -> None: + """Parallel version of ``_load_task_labels`` — bit-exact output.""" + tasks_path = os.path.join(self._all_shard_roots[0], "meta", "tasks.parquet") + if os.path.exists(tasks_path): + tasks_df = pd.read_parquet(tasks_path) + for _, row in tasks_df.iterrows(): + self._task_names[int(row["task_index"])] = str(row.name) + log.info(f"HandPoseDataset: loaded {len(self._task_names)} task labels") + + if not (self._load_subtasks or self._skip_no_action): + return + + def _read(shard_root: str) -> dict[int, str] | None: + path = os.path.join(shard_root, "meta", "subtasks.parquet") + if not os.path.exists(path): + return None + df = pd.read_parquet(path) + idx_arr = df["subtask_index"].to_numpy() + name_arr = df.index.to_numpy() + return dict(zip(idx_arr.astype(np.int64).tolist(), name_arr.astype(str).tolist())) + + roots = self._all_shard_roots + all_names = _parallel_map( + _read, + roots, + max_workers=max(1, min(self._fast_init_max_workers, len(roots))), + label="HandPoseDataset: _load_task_labels", + ) + # Match the serial version's set-membership: _per_shard_subtask_names + # is always set (``{}`` when missing); _per_shard_orig_subtask_names + # only when the file exists. + if self._load_subtasks: + for shard_idx, names in enumerate(all_names): + self._per_shard_subtask_names[shard_idx] = names if names is not None else {} + if self._skip_no_action: + for shard_idx, names in enumerate(all_names): + if names is not None: + self._per_shard_orig_subtask_names[shard_idx] = names + + def _register_sources(self, indices: list[int] | None = None) -> None: + """Register shard sources + HandPose-specific ``subtask_index`` prefetch. + + ``indices`` is the subset of ``_all_shard_roots`` assigned to this + caller (e.g. one slice per DataLoader worker under + ``shard_across_workers=True``). ``_ds_idx_to_shard_idx`` maps the + local ``ds_idx`` back to the global shard index used to key + ``_per_shard_subtask_names``, so caption lookups keep working. + + The base class owns the generic fast-init (parallel + ``LeRobotDatasetMetadata`` prefetch + serial ``_register_source`` + append loop). When ``enable_fast_init=True`` *and* snap/skip + flags are on, this override additionally prefetches the + ``subtask_index`` parquet column for each assigned shard in a + thread pool and caches it into ``_cached_subtask_col`` / + ``_orig_subtask_index``, so ``_rebuild_snap_indices`` and + ``_load_orig_subtask_index`` hit the cache instead of re-scanning + ``data/*.parquet``. + """ + if indices is None: + indices = list(range(len(self._all_shard_roots))) + self._ds_idx_to_shard_idx = list(indices) + if not indices: + return + + # Snapshot before ``super()._register_sources`` appends to + # ``_datasets``; each new ds_idx is ``base_ds_idx + offset``. + base_ds_idx = len(self._datasets) + roots = [self._all_shard_roots[i] for i in indices] + + super()._register_sources(indices) + + if not (self._enable_fast_init and (self._snap_to_subtask or self._skip_no_action)): + return + + import pyarrow.dataset as pa_ds + + def _read_subtask_col(root: str) -> np.ndarray: + data_dir = Path(root) / "data" + if not data_dir.exists(): + return np.array([], dtype=np.int64) + # Explicit ``*.parquet`` glob avoids picking up partial-write + # residues like ``file-000.parquet``. + parquet_files = sorted(str(f) for f in data_dir.rglob("*.parquet")) + if not parquet_files: + return np.array([], dtype=np.int64) + # ``use_threads=False`` avoids oversubscription; the outer + # pool already saturates lustre at the configured workers. + table = pa_ds.dataset(parquet_files, format="parquet").to_table( + columns=["subtask_index"], use_threads=False + ) + return table.column("subtask_index").to_numpy(zero_copy_only=False).astype(np.int64, copy=False) + + workers = max(1, min(self._fast_init_max_workers, len(roots))) + subtask_cols = _parallel_map( + _read_subtask_col, + roots, + max_workers=workers, + label="HandPoseDataset: subtask_index prefetch", + ) + for offset, col in enumerate(subtask_cols): + self._cached_subtask_col[base_ds_idx + offset] = col + self._orig_subtask_index[base_ds_idx + offset] = col + + def _resolve_shard_idx(self, ds_idx: int) -> int: + """Map a worker-local ``ds_idx`` to the global shard index.""" + if self._ds_idx_to_shard_idx is not None: + return self._ds_idx_to_shard_idx[ds_idx] + return ds_idx + + def _get_subtask_names_for_ds(self, ds_idx: int) -> dict[int, str]: + """Return the subtask name mapping for the shard that owns *ds_idx*.""" + return self._per_shard_subtask_names.get(self._resolve_shard_idx(ds_idx), {}) + + def _get_orig_subtask_names_for_ds(self, ds_idx: int) -> dict[int, str]: + """Return the original subtask name mapping for the shard that owns *ds_idx*.""" + return self._per_shard_orig_subtask_names.get(self._resolve_shard_idx(ds_idx), {}) + + def _load_episode_intrinsics(self, root: str) -> None: + """Load per-episode camera intrinsics from dataset metadata.""" + from lerobot.datasets.lerobot_dataset import LeRobotDatasetMetadata + + meta = LeRobotDatasetMetadata(repo_id="local", root=root) + eps = meta.episodes + if "camera_intrinsics" not in eps.column_names: + log.warning( + "HandPoseDataset: 'camera_intrinsics' not found in metadata; skeleton overlay will be unavailable." + ) + return + for ep in eps: + self._episode_intrinsics[int(ep["episode_index"])] = np.asarray(ep["camera_intrinsics"], dtype=np.float32) + log.info(f"HandPoseDataset: loaded intrinsics for {len(self._episode_intrinsics)} episodes") + + def _get_chunk_caption( + self, + ds_idx: int, + row_idx: int, + ep_idx: int, + sample: dict[str, Any], + effective_chunk_length: int | None = None, + ) -> str: + """Build a caption covering all subtasks that overlap the chunk window. + + When ``_load_subtasks`` is enabled, finds every subtask whose frame + range intersects ``[row_idx, row_idx + fps_ratio * chunk_length]`` and + concatenates their descriptions. Falls back to anchor-only task name + otherwise. + + Args: + effective_chunk_length: If provided, overrides ``self._chunk_length`` + for computing the chunk window (used by ``snap_to_subtask`` to + restrict the caption to the snapped subtask). + """ + task_name: str | None = None + task_idx = sample.get("task_index") + if task_idx is not None: + ti = int(task_idx) if not isinstance(task_idx, int) else task_idx + task_name = self._task_names.get(ti) + + if not self._load_subtasks: + if task_name: + return task_name.replace("_", " ") + return "Human hand manipulation task" + + transitions = self._get_subtask_transitions(ds_idx, ep_idx) + if not transitions: + if task_name: + return task_name.replace("_", " ") + return "Human hand manipulation task" + + ds = self._get_dataset(ds_idx) + fps_ratio = round(ds.meta.fps / self._fps) + cl = effective_chunk_length if effective_chunk_length is not None else self._chunk_length + chunk_end_row = row_idx + fps_ratio * cl + + row_starts = [t[0] for t in transitions] + first_ti = max(0, bisect_right(row_starts, row_idx) - 1) + + subtask_parts: list[str] = [] + seen: set[int] = set() + for ti in range(first_ti, len(transitions)): + seg_start, si = transitions[ti] + if seg_start >= chunk_end_row: + break + if si in seen: + continue + seen.add(si) + name = self._get_subtask_names_for_ds(ds_idx).get(si) + if name: + subtask_parts.append(name.replace("_", " ")) + + if subtask_parts: + return " Then, ".join(subtask_parts) + if task_name: + return task_name.replace("_", " ") + return "Human hand manipulation task" + + # ------------------------------------------------------------------------- + # Data quality filters + # ------------------------------------------------------------------------- + + def _has_unannotated_frames(self, sample: dict[str, Any]) -> bool: + """Return True if any frame in the sampled chunk appears unannotated. + + A frame is considered unannotated when both hands' position vectors are + effectively all-zero under ``self._unannotated_pos_l1_threshold``. + """ + left = sample[self._position_keys[0]] # (T+1, 63) + right = sample[self._position_keys[1]] # (T+1, 63) + left_l1 = left.abs().sum(dim=-1) + right_l1 = right.abs().sum(dim=-1) + missing_mask = (left_l1 <= self._unannotated_pos_l1_threshold) & ( + right_l1 <= self._unannotated_pos_l1_threshold + ) + return bool(missing_mask.any()) + + def _has_nan_action(self, action: torch.Tensor) -> bool: + """Return True if the action tensor contains any NaN or Inf values.""" + return bool(torch.isnan(action).any() or torch.isinf(action).any()) + + # ------------------------------------------------------------------------- + # Dataset interface + # ------------------------------------------------------------------------- + + def _snap_to_subtask_bounds(self, ds_idx: int, row_idx: int, ep_idx: int) -> tuple[int, int | None]: + """Snap ``row_idx`` to the start of its subtask and return its native-fps duration. + + Returns: + (snapped_row_idx, subtask_native_len) where subtask_native_len is + the number of native-fps rows in the subtask, or ``None`` if + transitions are unavailable. + """ + transitions = self._get_subtask_transitions(ds_idx, ep_idx) + if not transitions: + return row_idx, None + row_starts = [t[0] for t in transitions] + ti = bisect_right(row_starts, row_idx) - 1 + if ti < 0: + return row_idx, None + snapped = row_starts[ti] + if ti + 1 < len(transitions): + subtask_len = row_starts[ti + 1] - snapped + else: + ds = self._get_dataset(ds_idx) + ep_end: int = ds.meta.episodes[ep_idx]["dataset_to_index"] + subtask_len = ep_end - snapped + return snapped, subtask_len + + def _choose_mode(self) -> str: + """Resolve mode with biased sampling toward forward_dynamics in joint mode. + Overrides the uniform sampling in BaseActionLeRobotDataset. + """ + if self._mode == "joint": + return random.choices( + ("forward_dynamics", "inverse_dynamics", "policy"), + weights=(0.8, 0.1, 0.1), + k=1, + )[0] + return self._mode + + def __getitem__(self, idx: int) -> dict[str, Any]: + mode = self._choose_mode() + sample: dict[str, Any] | None = None + action: torch.Tensor | None = None + episode_idx: int = -1 + final_ds_idx: int = -1 + final_row_idx: int = -1 + effective_len: int | None = None + last_error: Exception | None = None + current_idx = idx + ds_idx = -1 + row_idx = -1 + frame_offset = -1 + + for _attempt in range(self._max_item_retries + 1): + try: + effective_len = None + ds_idx, row_idx, ep_idx, frame_offset = self._resolve_index(current_idx) + if self._snap_to_subtask: + subtask_info = self._subtask_start_indices.get((ds_idx, ep_idx)) + if subtask_info is not None and len(subtask_info) > 0: + si = frame_offset + if si < len(subtask_info): + row_idx, subtask_native_len = subtask_info[si] + ds = self._get_dataset(ds_idx) + fps_ratio = round(ds.meta.fps / self._fps) + snapped_video_frames = self._get_snapped_video_frame_count(subtask_native_len, fps_ratio) + if snapped_video_frames == 0: + last_error = RuntimeError("Snapped subtask is too short for tokenizer; resampling.") + current_idx = random.randrange(len(self)) + continue + effective_len = snapped_video_frames - 1 + else: + last_error = RuntimeError("Invalid subtask index; resampling.") + current_idx = random.randrange(len(self)) + continue + else: + last_error = RuntimeError("Invalid subtask index; resampling.") + current_idx = random.randrange(len(self)) + continue + elif self._skip_no_action: + skipped_row = self._skip_no_action_subtask(ds_idx, row_idx, ep_idx) + if skipped_row is None: + last_error = RuntimeError("All remaining original subtasks are 'No action'; resampling.") + current_idx = random.randrange(len(self)) + continue + row_idx = skipped_row + candidate = self._get_dataset(ds_idx)[row_idx] + except RuntimeError as error: + if self._is_video_decode_error(error): + last_error = error + self._log_bad_video_decode( + ds_idx=ds_idx, + row_idx=row_idx, + ep_idx=ep_idx, + frame_offset=frame_offset, + idx=idx, + attempt=_attempt, + error=error, + ) + current_idx = random.randrange(len(self)) + continue + raise + except (AssertionError, IndexError) as error: + if isinstance(error, AssertionError) and "violate the tolerance" not in str(error): + raise + last_error = error + if not self._warned_tolerance_once: + log.warning( + f"HandPoseDataset: encountered timestamp-tolerance mismatch; resampling index. Failed episode: {ep_idx}, row_idx: {row_idx}" + ) + self._warned_tolerance_once = True + current_idx = random.randrange(len(self)) + continue + + if self._drop_unannotated_edge_frames and self._has_unannotated_frames(candidate): + last_error = RuntimeError("Chunk contains all-zero hand annotation frame(s).") + if not self._warned_unannotated_once: + log.warning("HandPoseDataset: detected zero-filled annotation frames; resampling index.") + self._warned_unannotated_once = True + current_idx = random.randrange(len(self)) + continue + + try: + candidate_action = self._build_action(candidate) + except (ValueError, RuntimeError) as error: + last_error = error + if not self._warned_build_action_error_once: + log.warning(f"HandPoseDataset: _build_action failed at index {current_idx} ({error}); resampling.") + self._warned_build_action_error_once = True + current_idx = random.randrange(len(self)) + continue + if self._has_nan_action(candidate_action): + last_error = RuntimeError("Action contains NaN/Inf values.") + if not self._warned_nan_action_once: + log.warning(f"HandPoseDataset: NaN/Inf in action at index {current_idx}; resampling.") + self._warned_nan_action_once = True + current_idx = random.randrange(len(self)) + continue + + sample = candidate + action = candidate_action + episode_idx = ep_idx + final_ds_idx = ds_idx + final_row_idx = row_idx + break + + if sample is None or action is None: + raise RuntimeError( + "HandPoseDataset failed to sample a valid chunk after retries " + f"(max_item_retries={self._max_item_retries})." + ) from last_error + + ai_caption = self._get_chunk_caption( + final_ds_idx, + final_row_idx, + episode_idx, + sample, + effective_chunk_length=effective_len, + ) + if self._skip_video_loading: + raw_video = None + video = None + else: + # [T+1,C,H,W] float from LeRobot; _build_result converts to uint8 [C,T+1,H,W]. + raw_video = sample[self._video_key] + video = raw_video + + if effective_len is not None and effective_len < self._chunk_length: + if video is not None: + video = video[: effective_len + 1] # [T+1,C,H,W] + action = action[:effective_len] + + extras: dict[str, Any] = { + "__episode_id__": episode_idx, + "__row_idx__": final_row_idx, + "__dataset_root__": str(self._get_dataset(final_ds_idx).root), + "__index__": idx, + } + if self._return_overlay_data and not self._skip_video_loading: + extras["raw_cam_left_3d"] = sample[HAND_LEFT_POSITION_KEY] # (T+1, 63) + extras["raw_cam_right_3d"] = sample[HAND_RIGHT_POSITION_KEY] # (T+1, 63) + extras["raw_cam_left_rot"] = sample[HAND_LEFT_ROTATION_KEY] # (T+1, 84) + extras["raw_cam_right_rot"] = sample[HAND_RIGHT_ROTATION_KEY] # (T+1, 84) + extras["raw_cam_position"] = sample[CAM_POSITION_KEY] # (T+1, 3) + extras["raw_cam_rotation"] = sample[CAM_ROTATION_KEY] # (T+1, 4) + extras["orig_video_hw"] = torch.tensor([raw_video.shape[-2], raw_video.shape[-1]], dtype=torch.long) + intrinsics = self._episode_intrinsics.get(episode_idx) + if intrinsics is not None: + extras["camera_intrinsics"] = torch.from_numpy(intrinsics) + + return self._build_result(mode=mode, video=video, action=action, ai_caption=ai_caption, **extras) + + @staticmethod + def _is_video_decode_error(error: RuntimeError) -> bool: + msg = str(error) + common_err = "Requested next frame while there are no more frames left to decode" + return common_err in msg + + def _log_bad_video_decode( + self, + *, + ds_idx: int, + row_idx: int, + ep_idx: int, + frame_offset: int, + idx: int, + attempt: int, + error: RuntimeError, + ) -> None: + """Log decode failures with enough identifiers to locate bad videos later.""" + key = (ds_idx, ep_idx, row_idx) + if key in self._logged_decode_failures: + return + self._logged_decode_failures.add(key) + + ds = self._get_dataset(ds_idx) + dataset_root = str(getattr(ds, "root", "unknown")) + repo_id = getattr(getattr(ds, "meta", None), "repo_id", "unknown") + log.critical( + "HandPoseDataset video decode failure detected. " + f"idx={idx}, attempt={attempt}, split={self._split}, mode={self._mode}, " + f"dataset_idx={ds_idx}, dataset_root={dataset_root}, repo_id={repo_id}, " + f"episode_id={ep_idx}, row_idx={row_idx}, frame_offset={frame_offset}, " + f"video_key={self._video_key}, chunk_length={self._chunk_length}, fps={self._fps}, " + f"error={error!r}" + ) + + @property + def action_dim(self) -> int: + """Raw action dimensionality before padding.""" + return self._raw_action_dim diff --git a/cosmos_framework/data/vfm/action/hand_pose_dataset_config.py b/cosmos_framework/data/vfm/action/hand_pose_dataset_config.py new file mode 100644 index 0000000..0dab3f3 --- /dev/null +++ b/cosmos_framework/data/vfm/action/hand_pose_dataset_config.py @@ -0,0 +1,103 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Hand-pose dataset constants: data roots, dataset keys, and skeleton topology.""" + +import numpy as np + +# --------------------------------------------------------------------------- +# Embodiment_a LeRobot parquet keys +# --------------------------------------------------------------------------- +HAND_RIGHT_POSITION_KEY = "observation.state.hand_right_cam" +HAND_RIGHT_ROTATION_KEY = "observation.state.hand_right_cam_rotation" +HAND_LEFT_POSITION_KEY = "observation.state.hand_left_cam" +HAND_LEFT_ROTATION_KEY = "observation.state.hand_left_cam_rotation" +CAM_POSITION_KEY = "observation.state.camera_position" +CAM_ROTATION_KEY = "observation.state.camera_rotation" + +# --------------------------------------------------------------------------- +# Skeleton topology +# --------------------------------------------------------------------------- +NUM_JOINTS = 21 +QUAT_DIM_PER_JOINT = 4 # quaternion (qx, qy, qz, qw) as stored in parquet +MAT_DIM_PER_JOINT = 9 # flattened 3x3 rotation matrix (internal working format) +WRIST_JOINT_IDX = 0 +FINGERTIP_JOINT_IDXS = (4, 8, 12, 16, 20) + +ROTATION_FORMAT_DIM: dict[str, int] = {"rot9d": 9, "rot6d": 6, "euler_xyz": 3} + +# --------------------------------------------------------------------------- +# Subtask-name filters for ``skip_no_action`` (exact / prefix / substring). +# Mirrors FilterConfig in +# pipelines/customers/dht/action_gen/run_extract_subtask_clips.py. +# Matching is done against the normalized name (underscores → spaces, +# stripped, lowercased). +# --------------------------------------------------------------------------- +NO_ACTION_SKIP_LABELS: tuple[str, ...] = ("no action", "no actions") +NO_ACTION_SKIP_LABEL_PREFIXES: tuple[str, ...] = ("hold", "adjust") +NO_ACTION_SKIP_LABEL_SUBSTRINGS: tuple[str, ...] = ("idle",) + +# --------------------------------------------------------------------------- +# Wrist-frame alignment (Embodiment_a) +# --------------------------------------------------------------------------- +# 90° CCW rotation about local Z so that the wrist-local frame becomes: +# X = thumb→pinky, Y = palm normal (outward), Z = wrist→fingertips +WRIST_FRAME_ALIGN_EMBODIMENT_A = np.array( + [[0, 1, 0, 0], [-1, 0, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]], + dtype=np.float32, +) + +WRIST_FRAME_ALIGN_EMBODIMENT_A_IDENTITY = np.eye(4, dtype=np.float32) +# WRIST_FRAME_ALIGN_EMBODIMENT_A = np.array( +# [[0, 1, 0, 0], [-1, 0, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]], +# dtype=np.float32, +# ) +# WRIST_FRAME_ALIGN_EMBODIMENT_A = WRIST_FRAME_ALIGN_EMBODIMENT_A_IDENTITY + +# --------------------------------------------------------------------------- +# Embodiment_a sharded dataset roots +# --------------------------------------------------------------------------- +_EMBODIMENT_A_BASE = "" +EMBODIMENT_A_FEB08_500HR = [f"{_EMBODIMENT_A_BASE}/feb_08_500hr_lerobot_no_bframes/shard_{i:02d}" for i in range(10)] +EMBODIMENT_A_FEB15_1500HR = [f"{_EMBODIMENT_A_BASE}/feb_15_1500hr_lerobot_no_bframes/shard_{i:02d}" for i in range(31)] +EMBODIMENT_A_FEB23_1000HR = [f"{_EMBODIMENT_A_BASE}/feb_23_1000hr_lerobot_no_bframes/shard_{i:02d}" for i in range(21)] +EMBODIMENT_A_MAR02_1000HR = [f"{_EMBODIMENT_A_BASE}/mar_02_1000hr_lerobot/shard_{i:02d}" for i in range(21)] +EMBODIMENT_A_MAR09_4000HR = [f"{_EMBODIMENT_A_BASE}/mar_09_4000hr_lerobot/shard_{i:02d}" for i in range(81)] +EMBODIMENT_A_MAR16_7000HR = [f"{_EMBODIMENT_A_BASE}/mar_16_7000hr_lerobot/shard_{i:02d}" for i in range(140)] +EMBODIMENT_A_MAR30_9000HR = [f"{_EMBODIMENT_A_BASE}/mar_30_9000hr_lerobot/shard_{i:02d}" for i in range(180)] +EMBODIMENT_A_APR03 = [f"{_EMBODIMENT_A_BASE}/apr_03_lerobot/shard_{i:02d}" for i in range(181)] +EMBODIMENT_A_APR06_10000HR = [f"{_EMBODIMENT_A_BASE}/apr_06_10000hr_lerobot/shard_{i:02d}" for i in range(202)] +EMBODIMENT_A_ALL = ( + EMBODIMENT_A_FEB08_500HR + + EMBODIMENT_A_FEB15_1500HR + + EMBODIMENT_A_FEB23_1000HR + + EMBODIMENT_A_MAR02_1000HR + + EMBODIMENT_A_MAR09_4000HR + + EMBODIMENT_A_MAR16_7000HR + + EMBODIMENT_A_MAR30_9000HR + + EMBODIMENT_A_APR03 + + EMBODIMENT_A_APR06_10000HR +) + +# --------------------------------------------------------------------------- +# Registry of all hand-pose datasets for hand-only experiments +# --------------------------------------------------------------------------- +HAND_POSE_DATASETS: dict[str, str | list[str]] = { + "embodiment_a_feb08_500hr": EMBODIMENT_A_FEB08_500HR, + "embodiment_a_feb15_1500hr": EMBODIMENT_A_FEB15_1500HR, + "embodiment_a_feb23_1000hr": EMBODIMENT_A_FEB23_1000HR, + "embodiment_a_mar02_1000hr": EMBODIMENT_A_MAR02_1000HR, + "embodiment_a_mar09_4000hr": EMBODIMENT_A_MAR09_4000HR, + "embodiment_a_mar16_7000hr": EMBODIMENT_A_MAR16_7000HR, + "embodiment_a_mar30_9000hr": EMBODIMENT_A_MAR30_9000HR, + "embodiment_a_apr03": EMBODIMENT_A_APR03, + "embodiment_a_apr06_10000hr": EMBODIMENT_A_APR06_10000HR, + "embodiment_a_all": EMBODIMENT_A_ALL, + "embodiment_a_500hr_legacy_single": "", + "vitra_ego4d": [ + f"/{res}" + for res in ("810x1080", "1440x1080", "1920x1080", "1920x1440", "2560x1440", "2560x1920") + ], + "hwb_egoverse_eval_set_v0p1": "", + "hwb_egoverse_eval_set_v0p2": "", +} diff --git a/cosmos_framework/data/vfm/action/json_formatter.py b/cosmos_framework/data/vfm/action/json_formatter.py new file mode 100644 index 0000000..201a76e --- /dev/null +++ b/cosmos_framework/data/vfm/action/json_formatter.py @@ -0,0 +1,294 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from __future__ import annotations + +import math + +import torch + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.viewpoint_utils import DEFAULT_VIEWPOINT_TEMPLATES +from cosmos_framework.data.vfm.utils import VIDEO_RES_SIZE_INFO + + +def _should_append_idle_frame_info(mode: object) -> bool: + """Return whether idle-frame prompt metadata should be surfaced.""" + return mode != "inverse_dynamics" + + +class ActionPromptJsonFormatter: + """Format action prompts into a structured JSON-compatible dictionary. + + JSON fields are emitted in this order: ``cinematography``, ``actions``, + ``duration``, ``fps``, ``resolution``, then ``aspect_ratio``. Like video JSON + prompts, ``cinematography`` is a dictionary, duration is truncated to an + integer-second string such as ``"2s"``, and aspect ratio is stored as a + comma-separated string such as ``"16,9"``. If ``data_dict["mode"]`` is + ``"inverse_dynamics"``, idle-frame metadata is omitted from the prompt. + """ + + def __init__( + self, + caption_key: str = "ai_caption", + viewpoint_key: str = "viewpoint", + video_key: str = "video", + fps_key: str = "conditioning_fps", + image_size_key: str = "image_size", + idle_frames_key: str = "idle_frames", + total_frames_key: str = "idle_frames_total", + action_key: str = "action", + viewpoint_templates: dict[str, str] | None = None, + ) -> None: + self.caption_key: str = caption_key + self.viewpoint_key: str = viewpoint_key + self.video_key: str = video_key + self.fps_key: str = fps_key + self.image_size_key: str = image_size_key + self.idle_frames_key: str = idle_frames_key + self.total_frames_key: str = total_frames_key + self.action_key: str = action_key + self.viewpoint_templates: dict[str, str] = ( + viewpoint_templates if viewpoint_templates is not None else DEFAULT_VIEWPOINT_TEMPLATES + ) + + def __call__(self, data_dict: dict) -> dict: + """Replace the caption with the action JSON prompt structure.""" + additional_view_description = data_dict.pop("additional_view_description", None) + caption = data_dict.get(self.caption_key) + if not isinstance(caption, str) or caption == "": + return data_dict + + height, width = self._get_resolution(data_dict) + fps = self._get_scalar_float(data_dict.get(self.fps_key), self.fps_key) + if fps <= 0: + raise ValueError(f"ActionPromptJsonFormatter: '{self.fps_key}' must be positive, got {fps}") + + video = data_dict.get(self.video_key) + if not isinstance(video, torch.Tensor) or video.ndim < 2: + raise ValueError( + f"ActionPromptJsonFormatter: expected '{self.video_key}' to be a video tensor with shape " + f"(C, T, H, W), got {type(video).__name__}" + ) + duration_seconds = video.shape[1] / fps + duration = self._truncate_seconds(duration_seconds) + action_end_time = self._round_time_seconds(duration_seconds) + + prompt = { + "cinematography": { + "framing": self._get_viewpoint_caption(data_dict, additional_view_description), + }, + "actions": [ + { + "time": f"0:00-{self._format_time_mss(action_end_time)}", + "description": self._ensure_sentence(caption), + "idle_frame": self._get_idle_frame_info(data_dict), + } + ], + "duration": f"{duration}s", + "fps": float(fps), + "resolution": {"H": height, "W": width}, + "aspect_ratio": self._get_aspect_ratio(width, height), + } + cleaned_prompt = self._drop_empty_fields(prompt) + self._raise_if_empty_fields(cleaned_prompt) + data_dict[self.caption_key] = cleaned_prompt + return data_dict + + def _truncate_seconds(self, seconds: float) -> int: + """Truncate duration to integer seconds, matching video JSON-caption augmentors.""" + if seconds < 0 or not math.isfinite(seconds): + return 0 + return int(seconds) + + def _round_time_seconds(self, seconds: float) -> int: + """Round an action timestamp to integer seconds, matching video captioning.""" + if seconds < 0 or not math.isfinite(seconds): + return 0 + return round(seconds) + + def _format_time_mss(self, seconds: int) -> str: + """Format integer seconds as M:SS for JSON prompt time ranges.""" + minutes, remaining_seconds = divmod(seconds, 60) + return f"{minutes}:{remaining_seconds:02d}" + + def _get_aspect_ratio(self, width: int, height: int) -> str: + """Return the canonical width,height aspect ratio string when known.""" + for aspect_ratio_sizes in VIDEO_RES_SIZE_INFO.values(): + for aspect_ratio, (candidate_w, candidate_h) in aspect_ratio_sizes.items(): + if width == candidate_w and height == candidate_h: + return aspect_ratio + + divisor = math.gcd(width, height) + if divisor == 0: + raise ValueError( + f"ActionPromptJsonFormatter: width and height must be non-zero, got width={width}, height={height}." + ) + return f"{width // divisor},{height // divisor}" + + def _get_viewpoint_caption(self, data_dict: dict, additional_view_description: object | None) -> str | None: + """Resolve the viewpoint text used in the ``cinematography`` field.""" + viewpoint = data_dict.get(self.viewpoint_key) + template = self.viewpoint_templates.get(viewpoint) if isinstance(viewpoint, str) else None + + if template is None: + if viewpoint is not None: + log.warning( + f"ActionPromptJsonFormatter: unrecognized viewpoint {viewpoint!r}. " + f"Known viewpoints: {sorted(self.viewpoint_templates.keys())}. " + f"Using additional view description when available.", + rank0_only=False, + ) + return self._get_optional_text(additional_view_description) + + if additional_view_description: + separator = " " if template.endswith(".") else ". " + template = template + separator + str(additional_view_description).rstrip() + return template + + def _get_resolution(self, data_dict: dict) -> tuple[int, int]: + """Resolve ``(height, width)`` from the post-padding image size.""" + image_size = data_dict.get(self.image_size_key) + if image_size is None: + raise ValueError(f"ActionPromptJsonFormatter: missing '{self.image_size_key}' in data_dict.") + + if isinstance(image_size, torch.Tensor): + if image_size.numel() < 2: + raise ValueError( + f"ActionPromptJsonFormatter: expected '{self.image_size_key}' to contain at least " + f"height and width, got shape {tuple(image_size.shape)}" + ) + return int(image_size[0].item()), int(image_size[1].item()) + + try: + return int(image_size[0]), int(image_size[1]) + except (TypeError, ValueError, IndexError) as e: + raise ValueError( + f"ActionPromptJsonFormatter: expected '{self.image_size_key}' to contain height and width." + ) from e + + def _get_scalar_float(self, value: object, key: str) -> float: + """Parse a required scalar float from a tensor or Python value.""" + if value is None: + raise ValueError(f"ActionPromptJsonFormatter: missing '{key}' in data_dict.") + + if isinstance(value, torch.Tensor): + if value.numel() != 1: + raise ValueError( + f"ActionPromptJsonFormatter: expected scalar tensor at '{key}', got shape {tuple(value.shape)}" + ) + return float(value.item()) + + if isinstance(value, (str, int, float)): + try: + return float(value) + except ValueError as e: + raise ValueError( + f"ActionPromptJsonFormatter: expected scalar float-compatible value at '{key}'." + ) from e + raise ValueError(f"ActionPromptJsonFormatter: expected scalar float-compatible value at '{key}'.") + + def _get_optional_scalar_int(self, value: object, key: str) -> int | None: + """Parse an optional scalar integer metadata value.""" + if value is None: + return None + + if isinstance(value, torch.Tensor): + if value.numel() != 1: + log.warning( + f"ActionPromptJsonFormatter: expected scalar tensor at '{key}', got shape " + f"{tuple(value.shape)}. Skipping.", + rank0_only=False, + ) + return None + return int(value.item()) + + if isinstance(value, (str, int, float)): + try: + return int(value) + except ValueError: + pass + log.warning( + f"ActionPromptJsonFormatter: expected integer-compatible value at " + f"'{key}', got {type(value).__name__}. Skipping.", + rank0_only=False, + ) + return None + + def _get_total_frames(self, data_dict: dict) -> int | None: + """Resolve the total action-frame count for idle-frame text.""" + total_frames = self._get_optional_scalar_int(data_dict.get(self.total_frames_key), self.total_frames_key) + if total_frames is not None: + return total_frames + + action = data_dict.get(self.action_key) + if isinstance(action, torch.Tensor): + if action.ndim == 0: + log.warning( + f"ActionPromptJsonFormatter: expected action tensor at " + f"'{self.action_key}' to have a frame dimension. Skipping total frames.", + rank0_only=False, + ) + return None + return int(action.shape[0]) + + try: + return len(action) if action is not None else None + except TypeError: + return None + + def _get_idle_frame_info(self, data_dict: dict) -> str | None: + """Build the idle-frame string for the action object.""" + if not _should_append_idle_frame_info(data_dict.get("mode")): + return None + + idle_frames = self._get_optional_scalar_int(data_dict.get(self.idle_frames_key), self.idle_frames_key) + total_frames = self._get_total_frames(data_dict) + + if idle_frames is not None and total_frames is not None: + return f"{idle_frames} out of {total_frames}." + if idle_frames is not None: + return f"{idle_frames}." + return None + + def _ensure_sentence(self, text: str) -> str: + """Return text with terminal sentence punctuation.""" + text = text.strip() + if text.endswith((".", "!", "?")): + return text + return f"{text}." + + def _get_optional_text(self, value: object) -> str | None: + """Return stripped text, leaving empty optional text for the final prune pass.""" + if value is None: + return None + text = str(value).rstrip() + return text if text else None + + def _drop_empty_fields(self, value: object) -> object: + """Recursively remove empty strings, dictionaries, lists, and ``None`` values.""" + if isinstance(value, dict): + return { + key: cleaned + for key, item in value.items() + if not self._is_empty(cleaned := self._drop_empty_fields(item)) + } + if isinstance(value, list): + return [cleaned for item in value if not self._is_empty(cleaned := self._drop_empty_fields(item))] + return value + + def _is_empty(self, value: object) -> bool: + """Return whether a JSON field should be dropped.""" + return value is None or value == "" or value == [] or value == {} + + def _raise_if_empty_fields(self, value: object, path: str = "prompt") -> None: + """Validate that no empty JSON fields remain after pruning.""" + if self._is_empty(value): + raise ValueError(f"ActionPromptJsonFormatter: empty field remains at {path}.") + + if isinstance(value, dict): + for key, item in value.items(): + self._raise_if_empty_fields(item, f"{path}.{key}") + elif isinstance(value, list): + for index, item in enumerate(value): + self._raise_if_empty_fields(item, f"{path}[{index}]") diff --git a/cosmos_framework/data/vfm/action/libero_action_stats_10k.json b/cosmos_framework/data/vfm/action/libero_action_stats_10k.json new file mode 100644 index 0000000..7b12d29 --- /dev/null +++ b/cosmos_framework/data/vfm/action/libero_action_stats_10k.json @@ -0,0 +1,101 @@ +{ + "metadata": { + "action_dim": 7, + "chunk_length": 1, + "num_samples": 10000, + "total_action_frames": 10000, + "processing_time_s": 11.293463945388794, + "config_path": "cosmos_framework/configs/base/config.py", + "split": "train", + "experiment": "libero_exp", + "config_overrides": [ + "--", + "experiment=libero_exp" + ] + }, + "global": { + "mean": [ + 0.010700089146313277, + 0.059342142708553126, + -0.08804116051685972, + 0.003358964275650197, + 0.0031843214033986456, + -0.0034359285722253877, + 0.45609999999999973 + ], + "std": [ + 0.29001801614779354, + 0.3642332438293914, + 0.4413559193152232, + 0.0468960901544023, + 0.0627405349745355, + 0.08982732429699941, + 0.49809396679740553 + ], + "min": [ + -0.9375, + -0.9375, + -0.9375, + -0.2978571355342865, + -0.302142858505249, + -0.375, + 0.0 + ], + "max": [ + 0.9375, + 0.9375, + 0.9375, + 0.3257142901420593, + 0.3546428680419922, + 0.3750000298023224, + 1.0 + ], + "count": 10000 + }, + "per_timestep": { + "mean": [ + [ + 0.010700089146313277, + 0.059342142708553126, + -0.08804116051685972, + 0.003358964275650197, + 0.0031843214033986456, + -0.0034359285722253877, + 0.45609999999999973 + ] + ], + "std": [ + [ + 0.29001801614779354, + 0.3642332438293914, + 0.4413559193152232, + 0.0468960901544023, + 0.0627405349745355, + 0.08982732429699941, + 0.49809396679740553 + ] + ], + "min": [ + [ + -0.9375, + -0.9375, + -0.9375, + -0.2978571355342865, + -0.302142858505249, + -0.375, + 0.0 + ] + ], + "max": [ + [ + 0.9375, + 0.9375, + 0.9375, + 0.3257142901420593, + 0.3546428680419922, + 0.3750000298023224, + 1.0 + ] + ] + } +} diff --git a/cosmos_framework/data/vfm/action/libero_dataset.py b/cosmos_framework/data/vfm/action/libero_dataset.py new file mode 100644 index 0000000..91d2413 --- /dev/null +++ b/cosmos_framework/data/vfm/action/libero_dataset.py @@ -0,0 +1,611 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""LIBERO dataset for training from local storage, supporting multiple dataset roots.""" + +import random +from pathlib import Path +from typing import Literal + +import torch +import torchvision.transforms.functional as F +from lerobot.datasets.lerobot_dataset import LeRobotDataset +from torch.utils.data import Dataset + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.action_normalization import ( + load_action_stats, + normalize_action, +) +from cosmos_framework.data.vfm.action.action_spec import ( + Gripper, + Pos, + Rot, + build_action_spec, +) +from cosmos_framework.data.vfm.action.domain_utils import get_domain_id +from cosmos_framework.data.vfm.action.libero_pose_utils import ( + libero_action_dim, + libero_rotation_format, +) +from cosmos_framework.data.vfm.action.pose_utils import ( + compute_idle_frames, + convert_rotation, +) + +LIBERO_ROOTS: list[str] = [ + "", + "", + "", + "", + "", +] + + +class LIBERODataset(Dataset): + """ + A Dataset wrapper for LeRobot LIBERO dataset(s) designed for training from local storage. + + This dataset: + - Loads data from local storage using LeRobotDataset + - Supports multiple dataset roots that are concatenated into one dataset + - Supports configurable camera modes (image, wrist_image, or concat_view) + - Filters episodes for train/val split + - Filters frames at episode boundaries (to avoid padding issues with delta timestamps) + - Uses task descriptions from meta/tasks.parquet for ai_caption + """ + + _NORMALIZERS_DIR = Path(__file__).parent / "normalizers" + + def __init__( + self, + repo_id: str | list[str] = "lerobot/libero_90", + root: str | list[str] | None = LIBERO_ROOTS, + image_size: int = 256, + chunk_length: int = 16, # must be divisible by 4 + fps: int = 10, # IMPORTANT! LIBERO is at 20fps. If using frame_wise_relative in policy mode, we have to match the fps. + mode: str = "policy", + video_backend: str | None = "torchcodec", + download_videos: bool = False, + force_cache_sync: bool = False, + tolerance_s: float = 1e-4, + split: str = "train", + val_ratio: float = 0.01, + seed: int = 0, + # Camera configuration + camera_mode: str = "image", # 'image', 'wrist_image', or 'concat_view' + # Action configuration + action_space: str = "frame_wise_relative", # "absolute" or "relative" or "frame_wise_relative" + # rotation_space + rotation_space: Literal["9d", "6d", "3d"] = "3d", + # Native simulator frame or shared OpenCV-style EE frame used by midtraining. + pose_coordinate_frame: Literal["native", "opencv"] = "native", + # domain-aware configuration + embodiment_type: str = "libero", + action_normalization: Literal["quantile", "quantile_rot", "meanstd", "minmax"] | None = None, + action_stats_path: str | None = None, + skip_video_loading: bool = False, + ): + super().__init__() + self._embodiment_type = embodiment_type + self.domain_id = get_domain_id(embodiment_type) + self.image_size = image_size + self.chunk_length = chunk_length + assert self.chunk_length % 4 == 0, "chunk_length must be divisible by 4" + self.fps = fps + self.mode = mode + self.split = split.lower().strip() + self.val_ratio = val_ratio + self.seed = seed + self.camera_mode = camera_mode.lower().strip() + self.action_space = action_space + self.action_normalization = action_normalization + self.rotation_space = rotation_space.lower().strip() + self.pose_coordinate_frame = pose_coordinate_frame + self._pose_convention = self.action_space + self._rotation_format = libero_rotation_format(self.rotation_space) + # When True, skip video decoding entirely: drop image keys from + # delta_timestamps so LeRobot never touches the mp4, and return + # ``video=None`` in __getitem__. Must be set at construction time + # because LeRobotDataset is eagerly built in __init__. + self._skip_video_loading = bool(skip_video_loading) + + # Load action normalization stats. ``action_min`` / ``action_range`` are + # retained for older LIBERO eval code that knows how to invert a + # range-style [-1, 1] normalization. + self._norm_stats: dict[str, torch.Tensor] | None = None + self.action_min: torch.Tensor | None = None + self.action_max: torch.Tensor | None = None + self.action_range: torch.Tensor | None = None + if self.action_normalization is not None: + stats_path = self._resolve_action_stats_path(action_stats_path) + stats_key = "global_raw" if self.action_normalization == "quantile_rot" else "global" + raw_stats = load_action_stats(str(stats_path), stats_key=stats_key) + self._norm_stats = {} + for key, value in raw_stats.items(): + self._norm_stats[key] = torch.from_numpy(value).float() # [D] + self._set_range_denormalization_stats() + log.info( + f"Loaded LIBERO action stats from {stats_path} with action_normalization={self.action_normalization}" + ) + + # Validate camera mode + if self.camera_mode not in {"image", "wrist_image", "concat_view"}: + raise ValueError(f"Unsupported camera_mode={camera_mode!r}. Use 'image', 'wrist_image', or 'concat_view'.") + + # Validate split + if self.split not in {"train", "val", "valid", "validation", "eval", "test", "full"}: + raise ValueError(f"Unsupported {split=}. Use train/val/full.") + + # Build delta timestamps based on camera mode + dt = 1.0 / self.fps + + if self.fps != 20: + log.warning( + f"LIBERO is at 20fps. If using frame_wise_relative for policy mode training, we have to match the fps. fps={self.fps}" + ) + + # Determine which image keys to use + if self.camera_mode == "image": + self.image_keys = ["observation.images.image"] + elif self.camera_mode == "wrist_image": + self.image_keys = ["observation.images.wrist_image"] + else: # concat_view + self.image_keys = ["observation.images.image", "observation.images.wrist_image"] + + # Build delta_timestamps for all keys (same convention as PushT: 0 to chunk_length) + self.delta_timestamps: dict[str, list[float]] = {} + if not self._skip_video_loading: + for key in self.image_keys: + self.delta_timestamps[key] = [i * dt for i in range(0, chunk_length + 1)] + self.delta_timestamps["observation.state"] = [i * dt for i in range(0, chunk_length + 1)] + self.delta_timestamps["action"] = [i * dt for i in range(0, chunk_length + 1)] + + # Normalize repo_id and root to lists + repo_id_list: list[str] = [repo_id] if isinstance(repo_id, str) else list(repo_id) + root_list: list[str | None] + if root is None: + root_list = [None for _ in repo_id_list] + elif isinstance(root, str): + root_list = [root] + else: + root_list = [r for r in root] + + if len(repo_id_list) != len(root_list): + raise ValueError( + f"Length mismatch: repo_id has {len(repo_id_list)} items, root has {len(root_list)} items." + ) + + # Load all datasets + self.datasets: list[LeRobotDataset] = [] + self.tasks_dfs: list = [] # Store tasks DataFrames for each dataset + for rid, r in zip(repo_id_list, root_list): + dataset = LeRobotDataset( + repo_id=rid, + root=r, + delta_timestamps=self.delta_timestamps, # type: ignore + tolerance_s=tolerance_s, + force_cache_sync=force_cache_sync, + download_videos=download_videos, + video_backend=video_backend, + episodes=None, # Load full dataset, filter later + ) + self.datasets.append(dataset) + self.tasks_dfs.append(dataset.meta.tasks) + + # Build index mapping: list of (dataset_idx, local_idx) for valid frames + self.index_map: list[tuple[int, int, int]] = [] # (dataset_idx, local_idx, episode_idx) + self._episode_boundaries: list[dict[int, tuple[int, int]]] = [] + self._episode_splits: list[tuple[set[int], set[int]]] = [] + + total_episodes = 0 + total_frames = 0 + for ds_idx, dataset in enumerate(self.datasets): + # Compute episode splits for this dataset + train_eps, val_eps = self._compute_episode_splits_for_dataset(dataset) + self._episode_splits.append((train_eps, val_eps)) + + # Get episodes for current split + split_episodes = self._get_split_episodes_for_dataset(ds_idx) + + # Build episode boundaries + boundaries = self._build_episode_boundaries_for_dataset(dataset) + self._episode_boundaries.append(boundaries) + + # Filter indices + indices = self._filter_indices_for_dataset(ds_idx, dataset, split_episodes, boundaries) + self.index_map.extend(indices) + + total_episodes += dataset.num_episodes + total_frames += len(dataset) + + log.info( + f"Loaded LIBERO dataset with {len(repo_id_list)} source(s) split={self.split!r} " + f"camera_mode={self.camera_mode!r} " + f"total_episodes={total_episodes} " + f"total_frames={total_frames} " + f"valid_indices={len(self.index_map)}" + ) + + def _compute_episode_splits_for_dataset(self, dataset: LeRobotDataset) -> tuple[set[int], set[int]]: + """Compute train/val episode splits deterministically for a single dataset.""" + total_episodes = int(dataset.meta.total_episodes) + + if not (0.0 < self.val_ratio < 1.0): + raise ValueError(f"{self.val_ratio=} must be in (0, 1).") + + n_val = max(1, int(round(total_episodes * self.val_ratio))) + # val_eps = set(range(n_val)) + # train_eps = set(range(n_val, total_episodes)) + + # Yihuai: Randomly select validation episodes instead of the first n_val episodes (otherwise task will be repeated) + rng = random.Random(self.seed) # To ensure validation episodes are the same on all ranks + val_eps = set(rng.sample(range(total_episodes), n_val)) + train_eps = set(range(total_episodes)) - val_eps + + log.info(f"train_eps={train_eps}, val_eps={val_eps}") + + return train_eps, val_eps + + def _get_split_episodes_for_dataset(self, ds_idx: int) -> set[int]: + """Get the episode set for the current split for a specific dataset.""" + train_eps, val_eps = self._episode_splits[ds_idx] + if self.split in {"val", "valid", "validation", "eval", "test"}: + return val_eps + elif self.split == "train": + return train_eps + else: # full + return train_eps | val_eps + + def _build_episode_boundaries_for_dataset(self, dataset: LeRobotDataset) -> dict[int, tuple[int, int]]: + """Build a dict of episode_index -> (start_frame, end_frame) for a single dataset.""" + boundaries: dict[int, tuple[int, int]] = {} + for ep in dataset.meta.episodes: + ep_idx = int(ep["episode_index"]) # type: ignore[index] + start = int(ep["dataset_from_index"]) # type: ignore[index] + end = int(ep["dataset_to_index"]) # type: ignore[index] + boundaries[ep_idx] = (start, end) + return boundaries + + def _filter_indices_for_dataset( + self, + ds_idx: int, + dataset: LeRobotDataset, + split_episodes: set[int], + boundaries: dict[int, tuple[int, int]], + ) -> list[tuple[int, int, int]]: + """Filter valid indices for a single dataset, returning (dataset_idx, local_idx, episode_idx).""" + index_map: list[tuple[int, int, int]] = [] + all_meta = list(dataset.meta.episodes) + + for ep_idx in split_episodes: + if ep_idx >= len(all_meta): + continue + ep = all_meta[ep_idx] + + ep_start = int(ep["dataset_from_index"]) # type: ignore[index] + ep_end = int(ep["dataset_to_index"]) # type: ignore[index] + + # Valid range: [start, end - chunk_length - 1] inclusive + # We drop chunk_length frames at end to ensure we can query up to delta=chunk_length. + start = ep_start + end = ep_end - self.chunk_length - 1 + + if end >= start: + for local_idx in range(start, end + 1): + index_map.append((ds_idx, local_idx, ep_idx)) + + return index_map + + def __len__(self) -> int: + return len(self.index_map) + + def _get_task_description(self, ds_idx: int, item: dict) -> str: + """Get task description for the current item from meta/tasks.parquet. + + The tasks.parquet has task descriptions as the DataFrame index (row labels) + and task_index as an integer column. We look up by task_index and return + the corresponding index name (the actual task description string). + """ + task_idx = item.get("task_index") + if task_idx is not None: + if isinstance(task_idx, torch.Tensor): + task_idx = task_idx.item() + task_idx = int(task_idx) + tasks_df = self.tasks_dfs[ds_idx] + if task_idx in tasks_df["task_index"].values: + row = tasks_df[tasks_df["task_index"] == task_idx].iloc[0] + # The task description is the index name (row label), not a column value + return str(row.name) + raise ValueError(f"Task index {task_idx} not found in tasks.parquet for dataset {ds_idx}") + + def _compute_anchored_actions( + self, + state_raw: torch.Tensor, + action_raw: torch.Tensor, + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """Compute anchored relative actions (batched). + + Converts frame-wise relative actions to anchored relative actions where each + action[t] represents the target pose (after applying action[t] to state[t]) + expressed in state 0's local coordinate frame. + + Mathematical formulation: + 1. Compute target in world frame (LIBERO convention): + - p_{t+1} = p_t + delta_p[t] (position addition in world frame) + - R_{t+1} = R_delta[t] @ R_t (rotation composition, delta first) + 2. Compute anchored (left-multiply by T_0^{-1}): + - anchored_pos[t] = R_0^T @ (p_{t+1} - p_0) + - anchored_rot[t] = R_0^T @ R_{t+1} + + Args: + state_raw: State tensor of shape (T+1, 8): [x, y, z, ax, ay, az, grip1, grip2] + where (ax, ay, az) is axis-angle rotation. + action_raw: Action tensor of shape (T+1, 7): [dx, dy, dz, dax, day, daz, grip] + where (dax, day, daz) is axis-angle rotation delta. + + Returns: + anchored_translation: (T, 3) - position in state_0's local frame + anchored_rotation_9d: (T, 9) - rotation relative to state_0 as flattened 3x3 matrix + gripper: (T, 1) - original gripper commands (unchanged) + """ + # Extract positions and rotations from states + p_states = state_raw[:, :3] # [T+1,3] + rotvec_states = state_raw[:, 3:6] # [T+1,3] - axis-angle + + # Extract deltas from actions (use first T actions) + delta_p = action_raw[:-1, :3] # [T,3] + delta_rotvec = action_raw[:-1, 3:6] # [T,3] - axis-angle delta + gripper = action_raw[:-1, 6:7] # [T,1] + + # Convert all axis-angle to rotation matrices (batched) + R_states = convert_rotation(rotvec_states, input_format="axisangle", output_format="matrix") # [T+1,3,3] + R_deltas = convert_rotation(delta_rotvec, input_format="axisangle", output_format="matrix") # [T,3,3] + + # Initial pose (state 0) + p_0 = p_states[0] # [3] + R_0 = R_states[0] # [3,3] + R_0_T = R_0.T # [3,3] - transpose for inverse rotation + + # Current states for t = 0..T-1 + p_t = p_states[:-1] # [T,3] + R_t = R_states[:-1] # [T,3,3] + + # Step 1: Compute target poses in world frame (LIBERO convention) + # p_target = p_t + delta_p + p_target = p_t + delta_p # [T,3] + + # R_target = R_delta @ R_t (batched matrix multiply) + R_target = torch.bmm(R_deltas, R_t) # [T,3,3] + + # Step 2: Compute anchored (in state_0's local frame) + # anchored_p = R_0^T @ (p_target - p_0) + displacement = p_target - p_0 # [T,3] + anchored_p = (R_0_T @ displacement.T).T # [T,3] + + # anchored_R = R_0^T @ R_target (batched) + R_0_T_expanded = R_0_T.unsqueeze(0).expand(R_target.shape[0], -1, -1) # [T,3,3] + anchored_R = torch.bmm(R_0_T_expanded, R_target) # [T,3,3] + + return anchored_p, anchored_R, gripper + + def _convert_rotation_to_repr(self, rotation_matrix: torch.Tensor) -> torch.Tensor: + """Convert rotation matrix to the desired representation. + + Args: + rotation_matrix: Rotation matrices of shape (T, 3, 3). + + Returns: + Rotation in the configured ``rotation_space`` format. + """ + return convert_rotation(rotation_matrix, "matrix", libero_rotation_format(self.rotation_space)) + + def _normalizer_filename(self) -> str: + rotation_suffix = { + "3d": "3d", + "6d": "rot6d", + "9d": "rot9d", + }.get(self.rotation_space) + if rotation_suffix is None: + raise ValueError(f"Unsupported rotation_space={self.rotation_space!r}.") + action_space = self.action_space.replace("-", "_") + return f"{self._embodiment_type}_{action_space}_{rotation_suffix}.json" + + def _resolve_action_stats_path(self, action_stats_path: str | None) -> Path: + if action_stats_path is None: + stats_path = self._NORMALIZERS_DIR / self._normalizer_filename() + if stats_path.exists(): + return stats_path + raise FileNotFoundError( + f"Could not find bundled LIBERO action stats at {stats_path}. " + "Pass action_stats_path explicitly or regenerate stats with compute_action_stats.py." + ) + + stats_path = Path(action_stats_path) + if stats_path.is_absolute(): + if stats_path.exists(): + return stats_path + raise FileNotFoundError(f"Could not find action_stats_path={action_stats_path!r}.") + + module_dir = Path(__file__).resolve().parent + candidates: list[Path] = [] + for parent in module_dir.parents: + candidates.append(parent / stats_path) + candidates.append(self._NORMALIZERS_DIR / stats_path.name) + candidates.append(module_dir / stats_path.name) + for candidate in candidates: + if candidate.exists(): + return candidate + raise FileNotFoundError( + f"Could not resolve action_stats_path={action_stats_path!r}; tried: {[str(c) for c in candidates]}" + ) + + def _set_range_denormalization_stats(self) -> None: + if self._norm_stats is None: + return + + if self.action_normalization == "minmax": + lo_key, hi_key = "min", "max" + elif self.action_normalization in ("quantile", "quantile_rot"): + lo_key, hi_key = "q01", "q99" + else: + return + + if lo_key not in self._norm_stats or hi_key not in self._norm_stats: + raise ValueError( + f"Action stats for {self.action_normalization!r} normalization require " + f"{lo_key!r} and {hi_key!r} entries." + ) + self.action_min = self._norm_stats[lo_key] # [D] + self.action_max = self._norm_stats[hi_key] # [D] + action_range = self.action_max - self.action_min # [D] + self.action_range = torch.clamp(action_range, min=1e-6) # [D] + + def __getitem__(self, idx: int, _retry_count: int = 0) -> dict[str, torch.Tensor | str]: + """Get a single item from the dataset.""" + max_retries = 10 + ds_idx, local_idx, ep_idx = self.index_map[idx] + dataset = self.datasets[ds_idx] + try: + item = dataset[local_idx] + except Exception as e: + log.warning( + f"Error loading item (retry {_retry_count}/{max_retries}): idx={idx}, ds_idx={ds_idx}, " + f"local_idx={local_idx}, ep_idx={ep_idx}, repo_id={dataset.meta.repo_id}, error={e}" + ) + if _retry_count >= max_retries: + raise RuntimeError(f"Failed to load data after {max_retries} retries") from e + new_idx = random.randint(0, len(self) - 1) + return self.__getitem__(new_idx, _retry_count + 1) + + if self.mode == "joint": + mode = random.choice(["forward_dynamics", "inverse_dynamics", "policy", "image2video"]) + else: + mode = self.mode + + # Get task description for ai_caption + task_description = self._get_task_description(ds_idx, item) + + # Process video based on camera mode (skipped entirely when + # skip_video_loading=True; image keys are also absent from + # delta_timestamps so LeRobot never decoded them). + video: torch.Tensor | None + if self._skip_video_loading: + video = None + else: + if self.camera_mode == "concat_view": + # Load both cameras and concatenate horizontally + video_1: torch.Tensor = item["observation.images.image"] + video_2: torch.Tensor = item["observation.images.wrist_image"] + + # Resize each if needed + if video_1.shape[-1] != self.image_size or video_1.shape[-2] != self.image_size: + video_1 = F.resize(video_1, [self.image_size, self.image_size]) + if video_2.shape[-1] != self.image_size or video_2.shape[-2] != self.image_size: + video_2 = F.resize(video_2, [self.image_size, self.image_size]) + + # Concatenate along width dimension (last dim for TCHW) + video_tchw = torch.cat([video_1, video_2], dim=-1) # (T, C, H, W*2) + else: + # Single camera mode + image_key = self.image_keys[0] + video_tchw = item[image_key] + + # Resize if needed + if video_tchw.shape[-1] != self.image_size or video_tchw.shape[-2] != self.image_size: + video_tchw = F.resize(video_tchw, [self.image_size, self.image_size]) + + # Convert to uint8 and transpose to (C, T, H, W) + video = (video_tchw * 255).clamp(0, 255).to(torch.uint8).permute(1, 0, 2, 3) + + # Action (raw): LIBERO actions are 7D (6 DoF + gripper) + action_raw: torch.Tensor = item["action"] + # State (raw): LIBERO state is 8D (6 DoF + 2 gripper states) + state_raw: torch.Tensor = item["observation.state"] + + # Action: (T+1, D) -> (T, D) + # Take all but last action + # LIBERO action format: [x, y, z, ax, ay, az, gripper] (7D) where (ax,ay,az) is axis-angle + + if self.action_space == "relative": + # Compute anchored relative actions + # Returns: translation (T, 3), rotation_matrix (T, 3, 3), gripper (T, 1) + translation, rotation_matrix, gripper = self._compute_anchored_actions(state_raw, action_raw.clone()) + elif self.action_space == "frame_wise_relative": + action = action_raw[:-1].clone() # [T,7] + translation = action[:, :3] # [T,3] + rotation_rotvec = action[:, 3:6] # [T,3] + gripper = action[:, 6:] # [T,1] + rotation_matrix = convert_rotation( + rotation_rotvec, input_format="axisangle", output_format="matrix" + ) # [T,3,3] + else: + raise ValueError(f"Unsupported action space: {self.action_space}") + + rotation = self._convert_rotation_to_repr(rotation_matrix) # [T,rot_dim] + action = torch.cat([translation, rotation, gripper], dim=-1) # [T,action_dim] + + # Compute idle_frames from the raw (un-normalized) action, only when the + # action layout has correct per-frame idle semantics (frame_wise_relative + # ⇔ backward_framewise). The other action_spaces ("relative", + # "absolute") encode per-frame motion differently and would not give + # meaningful idle counts under the same threshold check. + idle_frames: torch.Tensor | None = None + if self.action_space == "frame_wise_relative": + try: + spec = build_action_spec(Pos(), Rot(libero_rotation_format(self.rotation_space)), Gripper()) + n = compute_idle_frames(action, spec) + idle_frames = torch.tensor(n, dtype=torch.long) + except (ValueError, TypeError): + idle_frames = None + + if self.action_normalization is not None and self._norm_stats is not None and self.action_min is not None: + if action.shape[-1] != self.action_min.shape[0]: + raise ValueError( + f"Action dimension {action.shape[-1]} does not match stats dimension " + f"{self.action_min.shape[0]}. Recompute stats for the current " + f"rotation_space={self.rotation_space!r} and action_space={self.action_space!r}." + ) + method = "quantile" if self.action_normalization == "quantile_rot" else self.action_normalization + action = normalize_action(action, method, self._norm_stats) # [T,D] + + # Index + key = torch.tensor([local_idx], dtype=torch.long) + + if self.camera_mode == "image": + viewpoint = "third_person_view" + elif self.camera_mode == "wrist_image": + viewpoint = "wrist_view" + else: + viewpoint = "concat_view" + + result: dict[str, torch.Tensor | str] = { + "source_repo_id": dataset.meta.repo_id, + "video": video, + "action": action, + "action_raw": action_raw, + "conditioning_fps": torch.tensor(self.fps, dtype=torch.long), + "prompt": task_description, + "ai_caption": task_description, + "mode": mode, + "state": state_raw, + "action_space": self.action_space, + "rotation_space": self.rotation_space, + "pose_coordinate_frame": self.pose_coordinate_frame, + "__key__": key, + "domain_id": torch.tensor(self.domain_id, dtype=torch.long), + "viewpoint": viewpoint, + } + if idle_frames is not None: + result["idle_frames"] = idle_frames + + if self.camera_mode == "concat_view" and not self._skip_video_loading: + result["additional_view_description"] = ( + "The left half shows the third-person view; the right half shows the wrist-mounted camera." + ) + + return result + + @property + def action_dim(self) -> int: + return libero_action_dim(self.rotation_space) diff --git a/cosmos_framework/data/vfm/action/libero_pose_utils.py b/cosmos_framework/data/vfm/action/libero_pose_utils.py new file mode 100644 index 0000000..5cc9fff --- /dev/null +++ b/cosmos_framework/data/vfm/action/libero_pose_utils.py @@ -0,0 +1,69 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Small LIBERO pose helpers shared by training and closed-loop eval.""" + +from __future__ import annotations + +import numpy as np +import torch + +from cosmos_framework.data.vfm.action.pose_utils import ( + RotationConvention, + build_abs_pose_from_components, +) + +# Same local-frame post-rotation pattern used by DROID/Bridge/Fractal: +# R_opencv = R_native @ *_TO_OPENCV. +LIBERO_TO_OPENCV: np.ndarray = np.array( + [[0.0, -1.0, 0.0], [1.0, 0.0, 0.0], [0.0, 0.0, 1.0]], + dtype=np.float32, +) + +LIBERO_ROTATION_FORMATS: dict[str, RotationConvention] = { + "3d": "axisangle", + "6d": "rot6d", + "9d": "rot9d", +} +LIBERO_ACTION_DIMS: dict[str, int] = {"3d": 7, "6d": 10, "9d": 13} + + +def libero_rotation_format(rotation_space: str) -> RotationConvention: + """Return the shared ``pose_utils`` rotation format for a LIBERO setting.""" + rotation_format = LIBERO_ROTATION_FORMATS.get(rotation_space) + if rotation_format is None: + raise ValueError(f"Unsupported rotation_space={rotation_space!r}. Use 3d/6d/9d.") + return rotation_format + + +def libero_action_dim(rotation_space: str) -> int: + """Return ``[xyz, rotation, gripper]`` action width for LIBERO.""" + action_dim = LIBERO_ACTION_DIMS.get(rotation_space) + if action_dim is None: + raise ValueError(f"Unsupported rotation_space={rotation_space!r}. Use 3d/6d/9d.") + return action_dim + + +def libero_rotation_space_from_action_dim(action_dim: int) -> str: + """Infer LIBERO rotation space from unpadded action width.""" + for rotation_space, dim in LIBERO_ACTION_DIMS.items(): + if dim == action_dim: + return rotation_space + raise ValueError(f"Unable to infer rotation_space from action_dim={action_dim}.") + + +def build_libero_abs_pose(state_raw: torch.Tensor | np.ndarray, *, to_opencv: bool) -> np.ndarray: + """Build absolute LIBERO EE poses from state rows. + + ``state_raw`` is ``[x,y,z,axisangle(3),gripper(2)]``. When requested, the + local EE frame is post-rotated into the shared OpenCV-style action frame. + """ + if isinstance(state_raw, torch.Tensor): + state_np = state_raw.detach().cpu().numpy().astype(np.float32, copy=False) + else: + state_np = np.asarray(state_raw, dtype=np.float32) + + poses_abs = build_abs_pose_from_components(state_np[:, :3], state_np[:, 3:6], "axisangle") + if to_opencv: + poses_abs[:, :3, :3] = poses_abs[:, :3, :3] @ LIBERO_TO_OPENCV + return poses_abs diff --git a/cosmos_framework/data/vfm/action/normalizers/README.md b/cosmos_framework/data/vfm/action/normalizers/README.md new file mode 100644 index 0000000..78c9e54 --- /dev/null +++ b/cosmos_framework/data/vfm/action/normalizers/README.md @@ -0,0 +1,79 @@ +# Action Normalizers + +This directory contains committed action-normalization statistics used by action +datasets during training and inference. + +There are two formats: + +- `BaseActionLeRobotDataset` stats JSONs, keyed by `global.mean/std/min/max/q01/q99`. + Files generated with skipped rotation dimensions may also include `global_raw`, + which preserves the raw pre-masking stats for `action_normalization="quantile_rot"`. + These are produced by `projects/cosmos3/vfm/datasets/action/compute_action_stats.py` + and are used by LeRobot-backed action datasets such as Bridge, DROID, + Fractal, RoboMIND, HandPose, Embodiment_b, and AgiBot. +- UMI field normalizers, keyed by output field name with per-field + `scale`/`offset` values. Normalization is `(x - offset) / scale`. + +## LeRobot-backed datasets + +Regenerate LeRobot-backed normalizers with `compute_action_stats.py`. By +default, output filenames match `BaseActionLeRobotDataset._normalizer_filename()`. +The script disables video loading for supported datasets; use fast init for +multi-shard datasets such as AgiBot: + +```bash +PYTHONPATH=. python cosmos_framework/data/vfm/action/compute_action_stats.py \ + --config cosmos_framework/configs/base/config.py \ + --split train \ + --enable-fast-init \ + --reservoir-size 5000000 \ + -- experiment=embodiment_c_gripper +``` + +For Embodiment C gripper, the committed file is: + +```text +embodiment_c_gripper_backward_framewise_rot6d.json +``` + +This normalizer is shared by: + +- `embodiment_c_gripper` +- `embodiment_c_gripper_ext` +- `agibotworld_beta` + +The AgiBot FK-pose action layout is 29D: + +```text +[head(9), right_wrist(9), right_gripper(1), left_wrist(9), left_gripper(1)] +``` + +Rotation dims are left unnormalized by writing identity stats for the rot6d +blocks when using `action_normalization="quantile"`. Use +`action_normalization="quantile_rot"` to load `global_raw` and normalize +rotation dimensions as well. + +## UMI datasets + +UMI normalizers are produced by `UMISingleTrajDataset.fit_normalizer()`, which +computes min/max statistics over the dataset. To regenerate: + +```python +from cosmos_framework.data.vfm.action.umi_dataset import get_umi_dataset + +dataset = get_umi_dataset( + dataset_name="", + dataset_type="single_task", + is_val=False, +) +dataset.fit_normalizer() +``` + +This writes `_normalizer.json` into the configured `normalizer_dir`. Copy the result into this directory. + +## Regenerate When + +- Changing action representation, pose convention, or rotation format. +- Changing gripper scaling or FK/action construction. +- Changing UMI `relative_pose_mode`, `use_relative_gripper_width`, or `eef_z_offset`. +- Adding a new task dataset or significantly expanding data with out-of-distribution actions. diff --git a/cosmos_framework/data/vfm/action/normalizers/bridge_orig_lerobot_backward_framewise_rot6d.json b/cosmos_framework/data/vfm/action/normalizers/bridge_orig_lerobot_backward_framewise_rot6d.json new file mode 100644 index 0000000..30ac5ab --- /dev/null +++ b/cosmos_framework/data/vfm/action/normalizers/bridge_orig_lerobot_backward_framewise_rot6d.json @@ -0,0 +1,33 @@ +{ + "metadata": { + "embodiment_type": "bridge_orig_lerobot", + "pose_convention": "backward_framewise", + "rotation_format": "rot6d", + "action_dim": 10, + "skip_rotation_dims": [3, 4, 5, 6, 7, 8], + "chunk_length": 16, + "sample_stride": 16, + "dataset_name": "bridge_20260416", + "dataset_class": "BridgeOrigLeRobotDataset", + "dataset_root": "", + "split": "train", + "num_samples_stats": 83036, + "reservoir_size": 5000000 + }, + "global": { + "mean": [-0.000094, -0.000394, 0.001623, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.582683], + "std": [ 0.013297, 0.009985, 0.012079, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.489959], + "min": [-0.309451, -0.074740, -0.082767, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000], + "max": [ 0.127018, 0.414660, 0.493186, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000], + "q01": [-0.038884, -0.028667, -0.037840, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000], + "q99": [ 0.039722, 0.029068, 0.026702, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000] + }, + "global_raw": { + "mean": [-0.000094, -0.000394, 0.001623, 0.998307, -0.001371, 0.000061, 0.001414, 0.998226, -0.000154, 0.582683], + "std": [ 0.013297, 0.009985, 0.012079, 0.004630, 0.050168, 0.029018, 0.050165, 0.004328, 0.031742, 0.489959], + "min": [-0.309451, -0.074740, -0.082767, -0.845782, -0.636628, -0.401535, -0.590214, -0.217448, -0.979635, 0.000000], + "max": [ 0.127018, 0.414660, 0.493186, 1.000000, 0.362611, 0.601211, 0.619479, 1.000000, 0.365993, 1.000000], + "q01": [-0.038884, -0.028667, -0.037840, 0.976292, -0.163098, -0.081545, -0.160193, 0.976322, -0.078872, 0.000000], + "q99": [ 0.039722, 0.029068, 0.026702, 1.000000, 0.160195, 0.081655, 0.163227, 1.000000, 0.095189, 1.000000] + } +} diff --git a/cosmos_framework/data/vfm/action/normalizers/droid_lerobot_backward_framewise_rot6d.json b/cosmos_framework/data/vfm/action/normalizers/droid_lerobot_backward_framewise_rot6d.json new file mode 100644 index 0000000..7f18bec --- /dev/null +++ b/cosmos_framework/data/vfm/action/normalizers/droid_lerobot_backward_framewise_rot6d.json @@ -0,0 +1,33 @@ +{ + "metadata": { + "embodiment_type": "droid_lerobot", + "pose_convention": "backward_framewise", + "rotation_format": "rot6d", + "action_dim": 10, + "skip_rotation_dims": [3, 4, 5, 6, 7, 8], + "chunk_length": 16, + "sample_stride": 16, + "dataset_name": "droid_20260418", + "dataset_class": "DROIDLeRobotDataset", + "dataset_root": "", + "split": "train", + "num_samples_stats": 1321153, + "reservoir_size": 5000000 + }, + "global": { + "mean": [-0.000017, -0.000612, 0.000568, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.588911], + "std": [ 0.004539, 0.004054, 0.004999, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.441186], + "min": [-0.075397, -0.057288, -0.056677, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000], + "max": [ 0.073107, 0.082187, 0.077080, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000], + "q01": [-0.014200, -0.013416, -0.015206, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000], + "q99": [ 0.014515, 0.011517, 0.014520, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000] + }, + "global_raw": { + "mean": [-0.000017, -0.000612, 0.000568, 0.999830, 0.000227, -0.000152, -0.000222, 0.999818, 0.000417, 0.588911], + "std": [ 0.004539, 0.004054, 0.004999, 0.000336, 0.014924, 0.010784, 0.014927, 0.000351, 0.011903, 0.441186], + "min": [-0.075397, -0.057288, -0.056677, 0.695640, -0.220599, -0.195892, -0.697421, 0.600468, -0.154176, 0.000000], + "max": [ 0.073107, 0.082187, 0.077080, 1.000000, 0.698449, 0.168089, 0.220605, 1.000000, 0.391206, 1.000000], + "q01": [-0.014200, -0.013416, -0.015206, 0.998459, -0.047659, -0.034774, -0.047609, 0.998428, -0.035553, 0.000000], + "q99": [ 0.014515, 0.011517, 0.014520, 1.000000, 0.047596, 0.034660, 0.047654, 1.000000, 0.038888, 1.000000] + } +} diff --git a/cosmos_framework/data/vfm/action/normalizers/embodiment_b.json b/cosmos_framework/data/vfm/action/normalizers/embodiment_b.json new file mode 100644 index 0000000..acc9cab --- /dev/null +++ b/cosmos_framework/data/vfm/action/normalizers/embodiment_b.json @@ -0,0 +1,25 @@ +{ + "metadata": { + "embodiment_type": "embodiment_b", + "pose_convention": null, + "rotation_format": null, + "action_dim": 30, + "skip_rotation_dims": [], + "chunk_length": 16, + "sample_stride": 16, + "dataset_name": "embodiment_b", + "dataset_class": "Embodiment_bDataset", + "dataset_root": "", + "split": "train", + "num_samples_stats": 29959, + "reservoir_size": 5000000 + }, + "global": { + "mean": [ 1.736306, -1.439579, -0.668389, -2.180986, -0.140066, -0.108423, 0.082355, -0.875710, 1.008860, 0.201179, 1.680600, 0.491681, -0.079254, -0.216104, 0.388113, 0.266642, 0.840022, 0.032036, 0.015728, -0.023622, 0.994667, 0.589821, -0.233003, 0.887161, -0.020604, 0.056817, 0.174483, 0.859159, 0.298373, 0.466469], + "std": [ 0.258256, 0.155732, 0.166345, 0.259065, 0.150113, 0.188906, 0.151725, 0.735972, 0.454291, 0.585101, 0.423016, 0.331052, 0.294947, 0.426356, 0.082411, 0.049379, 0.054863, 0.042942, 0.047117, 0.065842, 0.020142, 0.140435, 0.134750, 0.086622, 0.241199, 0.112985, 0.267914, 0.291596, 0.199299, 0.378883], + "min": [-0.210359, -1.590618, -2.304231, -2.450943, -1.152912, -0.820512, -0.360174, -3.004326, -1.608063, -1.690435, 0.127842, -1.044611, -0.735398, -1.531176, 0.247331, -0.017434, 0.461093, -0.075683, -0.261053, -0.638773, 0.162428, 0.251857, -0.709606, 0.437299, -0.793154, -0.370741, -0.701581, -0.499994, 0.166667, 0.166667], + "max": [ 2.898517, 1.205229, 1.096718, -0.532735, 0.288580, 0.516904, 0.841676, 1.171752, 1.608063, 2.833071, 2.567994, 1.932145, 0.822665, 1.028385, 0.918418, 0.457046, 1.316623, 0.793569, 0.204932, 0.131193, 0.999951, 0.985773, 0.198002, 1.260103, 0.889235, 0.491383, 0.829481, 0.999999, 1.000000, 1.000000], + "q01": [ 0.855471, -1.590571, -0.868137, -2.450919, -0.492864, -0.820464, -0.221557, -2.603151, -1.029259, -1.337718, 0.634284, -0.325527, -0.703343, -1.367646, 0.295801, 0.164838, 0.697646, -0.027419, -0.120393, -0.156538, 0.968709, 0.311899, -0.534542, 0.757548, -0.630892, -0.165377, -0.629816, -0.378646, 0.166667, 0.166667], + "q99": [ 2.018381, -1.090716, -0.124196, -1.152713, 0.110422, 0.177702, 0.344905, 0.813681, 1.608063, 1.769150, 2.396629, 1.414085, 0.645255, 0.542724, 0.589946, 0.352675, 0.947880, 0.122770, 0.125947, 0.115921, 0.999723, 0.887231, 0.080622, 1.114126, 0.784541, 0.386300, 0.704707, 0.999596, 0.583333, 1.000000] + } +} diff --git a/cosmos_framework/data/vfm/action/normalizers/embodiment_c_gripper_backward_framewise_rot6d.json b/cosmos_framework/data/vfm/action/normalizers/embodiment_c_gripper_backward_framewise_rot6d.json new file mode 100644 index 0000000..0622e1c --- /dev/null +++ b/cosmos_framework/data/vfm/action/normalizers/embodiment_c_gripper_backward_framewise_rot6d.json @@ -0,0 +1,33 @@ +{ + "metadata": { + "embodiment_type": "embodiment_c_gripper", + "pose_convention": "backward_framewise", + "rotation_format": "rot6d", + "action_dim": 29, + "skip_rotation_dims": [3, 4, 5, 6, 7, 8, 12, 13, 14, 15, 16, 17, 22, 23, 24, 25, 26, 27], + "chunk_length": 16, + "sample_stride": 16, + "dataset_name": "embodiment_c_gripper", + "dataset_class": "EmbodimentCGripperDataset", + "dataset_root": "", + "split": "train", + "num_samples_stats": 4602075, + "reservoir_size": 5000000 + }, + "global": { + "mean": [-0.000001, -0.000053, -0.000039, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000067, 0.000060, 0.000031, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.490037, -0.000099, 0.000038, 0.000047, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.556796], + "std": [ 0.001810, 0.004119, 0.010410, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.006703, 0.009197, 0.006997, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.396319, 0.006079, 0.008763, 0.006617, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.402953], + "min": [-0.607455, -0.465921, -1.314688, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.205045, -1.253672, -1.082791, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000, -1.235187, -1.328355, -1.081424, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000], + "max": [ 0.607457, 0.465930, 1.314692, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.205044, 1.253679, 1.097739, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.238890, 1.322212, 1.089373, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000], + "q01": [-0.000167, -0.007272, -0.014935, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -0.012912, -0.017163, -0.017614, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000, -0.011640, -0.015508, -0.013880, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000], + "q99": [ 0.000164, 0.004822, 0.013706, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.013182, 0.016960, 0.016101, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.010890, 0.015347, 0.012968, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000] + }, + "global_raw": { + "mean": [-0.000001, -0.000053, -0.000039, 0.999995, 0.000001, -0.000002, -0.000001, 0.999975, 0.000081, 0.000067, 0.000060, 0.000031, 0.999687, 0.000133, -0.000151, -0.000152, 0.999683, -0.000060, 0.490037, -0.000099, 0.000038, 0.000047, 0.999815, -0.000118, 0.000219, 0.000132, 0.999816, -0.000041, 0.556796], + "std": [ 0.001810, 0.004119, 0.010410, 0.000093, 0.001281, 0.003053, 0.001279, 0.000182, 0.007016, 0.006703, 0.009197, 0.006997, 0.001695, 0.019522, 0.015552, 0.019496, 0.001879, 0.015844, 0.396319, 0.006079, 0.008763, 0.006617, 0.001385, 0.015094, 0.011857, 0.015060, 0.001785, 0.011771, 0.402953], + "min": [-0.607455, -0.465921, -1.314688, 0.985691, -0.066926, -0.159641, -0.068475, 0.951378, -0.308000, -1.205045, -1.253672, -1.082791, -0.501766, -0.909665, -0.743807, -0.901858, -0.763505, -0.903163, 0.000000, -1.235187, -1.328355, -1.081424, -0.193770, -0.811308, -0.944745, -0.828973, -0.818795, -0.773241, 0.000000], + "max": [ 0.607457, 0.465930, 1.314692, 1.000000, 0.069006, 0.146651, 0.065107, 1.000000, 0.268938, 1.205044, 1.253679, 1.097739, 1.000000, 0.820798, 0.962229, 0.831849, 1.000000, 0.837535, 1.000000, 1.238890, 1.322212, 1.089373, 1.000000, 0.795667, 0.686249, 0.796464, 1.000000, 0.696757, 1.000000], + "q01": [-0.000167, -0.007272, -0.014935, 0.999999, -0.000306, -0.000594, -0.000260, 0.999227, -0.025516, -0.012912, -0.017163, -0.017614, 0.994613, -0.064506, -0.053231, -0.066267, 0.994383, -0.051163, 0.000000, -0.011640, -0.015508, -0.013880, 0.996511, -0.050126, -0.040305, -0.047330, 0.996618, -0.038303, 0.000000], + "q99": [ 0.000164, 0.004822, 0.013706, 1.000000, 0.000240, 0.000703, 0.000278, 1.000000, 0.030090, 0.013182, 0.016960, 0.016101, 1.000000, 0.066268, 0.053905, 0.064357, 1.000000, 0.052547, 1.000000, 0.010890, 0.015347, 0.012968, 1.000000, 0.047482, 0.042217, 0.050173, 1.000000, 0.041428, 1.000000] + } +} diff --git a/cosmos_framework/data/vfm/action/normalizers/fractal_backward_framewise_rot6d.json b/cosmos_framework/data/vfm/action/normalizers/fractal_backward_framewise_rot6d.json new file mode 100644 index 0000000..c4fa8ec --- /dev/null +++ b/cosmos_framework/data/vfm/action/normalizers/fractal_backward_framewise_rot6d.json @@ -0,0 +1,33 @@ +{ + "metadata": { + "embodiment_type": "fractal", + "pose_convention": "backward_framewise", + "rotation_format": "rot6d", + "action_dim": 10, + "skip_rotation_dims": [3, 4, 5, 6, 7, 8], + "chunk_length": 16, + "sample_stride": 16, + "dataset_name": "fractal_20260413", + "dataset_class": "FractalLeRobotDataset", + "dataset_root": "", + "split": "train", + "num_samples_stats": 166961, + "reservoir_size": 5000000 + }, + "global": { + "mean": [ 0.002259, 0.000721, 0.009372, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.526947], + "std": [ 0.014178, 0.016428, 0.022554, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.499273], + "min": [-0.151886, -0.176424, -0.194576, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000], + "max": [ 0.130892, 0.190835, 0.193839, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000], + "q01": [-0.039816, -0.049270, -0.056266, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000], + "q99": [ 0.043860, 0.050352, 0.072505, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000] + }, + "global_raw": { + "mean": [ 0.002259, 0.000721, 0.009372, 0.998347, 0.001789, 0.002694, -0.001861, 0.997811, 0.016366, 0.526947], + "std": [ 0.014178, 0.016428, 0.022554, 0.003377, 0.043416, 0.037369, 0.043211, 0.004566, 0.047057, 0.499273], + "min": [-0.151886, -0.176424, -0.194576, 0.520558, -0.676280, -0.822475, -0.460521, 0.736643, -0.517041, 0.000000], + "max": [ 0.130892, 0.190835, 0.193839, 1.000000, 0.461026, 0.403940, 0.671708, 1.000000, 0.505528, 1.000000], + "q01": [-0.039816, -0.049270, -0.056266, 0.983667, -0.134543, -0.107048, -0.126518, 0.977277, -0.091363, 0.000000], + "q99": [ 0.043860, 0.050352, 0.072505, 1.000000, 0.127404, 0.107273, 0.134140, 1.000000, 0.179731, 1.000000] + } +} diff --git a/cosmos_framework/data/vfm/action/normalizers/hand_pose_backward_framewise_rot6d.json b/cosmos_framework/data/vfm/action/normalizers/hand_pose_backward_framewise_rot6d.json new file mode 100644 index 0000000..5c68f61 --- /dev/null +++ b/cosmos_framework/data/vfm/action/normalizers/hand_pose_backward_framewise_rot6d.json @@ -0,0 +1,35 @@ +{ + "metadata": { + "embodiment_type": "hand_pose", + "pose_convention": "backward_framewise", + "rotation_format": "rot6d", + "action_dim": 57, + "skip_rotation_dims": [3, 4, 5, 6, 7, 8, 12, 13, 14, 15, 16, 17, 36, 37, 38, 39, 40, 41], + "chunk_length": 1, + "sample_stride": 16, + "dataset_name": "hand_pose_20260413", + "dataset_class": "HandPoseDataset", + "dataset_root": [], + "split": "train", + "num_samples_stats": 1027904, + "reservoir_size": 5000000, + "max_samples": 1000000, + "sampling_seed": 42 + }, + "global": { + "mean": [-0.000051, -0.000536, 0.000271, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, -0.000250, 0.000039, 0.000358, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, -0.020389, 0.067672, 0.095163, -0.007768, 0.060786, 0.120434, 0.014182, 0.060810, 0.106781, 0.030128, 0.057963, 0.096013, 0.043033, 0.045557, 0.083792, 0.000213, 0.000032, 0.000431, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.025732, 0.066837, 0.093646, 0.009652, 0.056993, 0.125456, -0.013350, 0.057552, 0.117093, -0.030613, 0.054797, 0.107642, -0.045114, 0.042600, 0.092532], + "std": [ 0.008032, 0.006073, 0.004838, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.015564, 0.013365, 0.013550, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.020526, 0.015765, 0.014673, 0.010769, 0.016233, 0.027204, 0.009681, 0.015045, 0.031945, 0.012000, 0.015039, 0.031826, 0.012330, 0.013215, 0.025054, 0.012756, 0.011769, 0.011749, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.021840, 0.017342, 0.015389, 0.010482, 0.018389, 0.028098, 0.009773, 0.017744, 0.032972, 0.011621, 0.017690, 0.032064, 0.012763, 0.015870, 0.024854], + "min": [-0.135277, -0.178123, -0.132330, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -0.303401, -0.308792, -0.285116, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -0.169384, -0.057983, 0.004869, -0.080825, -0.066215, 0.031035, -0.062484, -0.074770, 0.021586, -0.033416, -0.068450, 0.013296, -0.018355, -0.051769, 0.009567, -0.286551, -0.261433, -0.304940, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -0.073763, -0.052037, -0.012102, -0.045693, -0.059251, 0.034208, -0.061135, -0.072159, 0.029027, -0.093798, -0.058713, 0.019525, -0.109188, -0.044517, 0.013484], + "max": [ 0.162729, 0.092829, 0.113989, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.412223, 0.342286, 0.258529, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.053027, 0.143626, 0.196154, 0.048750, 0.110958, 0.216498, 0.071694, 0.109664, 0.226691, 0.093366, 0.106893, 0.219522, 0.104191, 0.094833, 0.192419, 0.288662, 0.238981, 0.266988, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.180267, 0.152062, 0.183739, 0.098497, 0.114275, 0.216822, 0.064544, 0.107233, 0.233817, 0.021220, 0.107390, 0.233983, -0.000990, 0.096846, 0.206103], + "q01": [-0.026688, -0.024154, -0.013269, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -0.048362, -0.040973, -0.040898, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -0.080012, 0.029713, 0.059227, -0.038851, 0.010826, 0.062254, -0.007452, 0.010061, 0.047149, 0.005736, 0.006631, 0.036123, 0.018989, 0.001386, 0.033526, -0.037843, -0.037665, -0.036427, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -0.013628, 0.024588, 0.053585, -0.012045, 0.000813, 0.065182, -0.037355, -0.001436, 0.052279, -0.058561, -0.004175, 0.042882, -0.078014, -0.007496, 0.039113], + "q99": [ 0.026573, 0.015575, 0.015514, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.044917, 0.040128, 0.043628, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.016559, 0.105274, 0.129951, 0.014097, 0.089857, 0.188346, 0.037904, 0.087388, 0.192120, 0.059029, 0.083990, 0.177890, 0.074478, 0.069913, 0.145329, 0.041177, 0.036800, 0.039655, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.087733, 0.107510, 0.128318, 0.040024, 0.088640, 0.191875, 0.008532, 0.086990, 0.195138, -0.005823, 0.083213, 0.180562, -0.019596, 0.069257, 0.148124] + }, + "global_raw": { + "mean": [-0.000051, -0.000536, 0.000271, 0.999819, -0.000087, -0.000014, 0.000086, 0.999810, -0.000149, -0.000250, 0.000039, 0.000358, 0.990760, -0.000167, 0.000539, -0.000625, 0.991384, 0.000669, -0.020389, 0.067672, 0.095163, -0.007768, 0.060786, 0.120434, 0.014182, 0.060810, 0.106781, 0.030128, 0.057963, 0.096013, 0.043033, 0.045557, 0.083792, 0.000213, 0.000032, 0.000431, 0.993515, -0.000283, -0.000113, 0.000924, 0.993799, 0.000744, 0.025732, 0.066837, 0.093646, 0.009652, 0.056993, 0.125456, -0.013350, 0.057552, 0.117093, -0.030613, 0.054797, 0.107642, -0.045114, 0.042600, 0.092532], + "std": [ 0.008032, 0.006073, 0.004838, 0.000911, 0.017357, 0.007726, 0.017367, 0.000866, 0.008817, 0.015564, 0.013365, 0.013550, 0.043221, 0.102472, 0.077623, 0.102379, 0.039789, 0.071362, 0.020526, 0.015765, 0.014673, 0.010769, 0.016233, 0.027204, 0.009681, 0.015045, 0.031945, 0.012000, 0.015039, 0.031826, 0.012330, 0.013215, 0.025054, 0.012756, 0.011769, 0.011749, 0.034683, 0.087015, 0.064440, 0.086931, 0.032113, 0.061431, 0.021840, 0.017342, 0.015389, 0.010482, 0.018389, 0.028098, 0.009773, 0.017744, 0.032972, 0.011621, 0.017690, 0.032064, 0.012763, 0.015870, 0.024854], + "min": [-0.135277, -0.178123, -0.132330, 0.922451, -0.352303, -0.233175, -0.363654, 0.931059, -0.182811, -0.303401, -0.308792, -0.285116, -0.998424, -0.999706, -0.994105, -0.999480, -0.997971, -0.987817, -0.169384, -0.057983, 0.004869, -0.080825, -0.066215, 0.031035, -0.062484, -0.074770, 0.021586, -0.033416, -0.068450, 0.013296, -0.018355, -0.051769, 0.009567, -0.286551, -0.261433, -0.304940, -0.998922, -0.999765, -0.998850, -0.996613, -0.999843, -0.978575, -0.073763, -0.052037, -0.012102, -0.045693, -0.059251, 0.034208, -0.061135, -0.072159, 0.029027, -0.093798, -0.058713, 0.019525, -0.109188, -0.044517, 0.013484], + "max": [ 0.162729, 0.092829, 0.113989, 1.000000, 0.355478, 0.210337, 0.344650, 1.000000, 0.169127, 0.412223, 0.342286, 0.258529, 1.000000, 0.995610, 0.977012, 0.998188, 1.000000, 0.981797, 0.053027, 0.143626, 0.196154, 0.048750, 0.110958, 0.216498, 0.071694, 0.109664, 0.226691, 0.093366, 0.106893, 0.219522, 0.104191, 0.094833, 0.192419, 0.288662, 0.238981, 0.266988, 1.000000, 0.999539, 0.994357, 0.999825, 1.000000, 0.999127, 0.180267, 0.152062, 0.183739, 0.098497, 0.114275, 0.216822, 0.064544, 0.107233, 0.233817, 0.021220, 0.107390, 0.233983, -0.000990, 0.096846, 0.206103], + "q01": [-0.026688, -0.024154, -0.013269, 0.996827, -0.056513, -0.023720, -0.055905, 0.996923, -0.026957, -0.048362, -0.040973, -0.040898, 0.869093, -0.316019, -0.240748, -0.327180, 0.879926, -0.213020, -0.080012, 0.029713, 0.059227, -0.038851, 0.010826, 0.062254, -0.007452, 0.010061, 0.047149, 0.005736, 0.006631, 0.036123, 0.018989, 0.001386, 0.033526, -0.037843, -0.037665, -0.036427, 0.901690, -0.278892, -0.200027, -0.267251, 0.906997, -0.187048, -0.013628, 0.024588, 0.053585, -0.012045, 0.000813, 0.065182, -0.037355, -0.001436, 0.052279, -0.058561, -0.004175, 0.042882, -0.078014, -0.007496, 0.039113], + "q99": [ 0.026573, 0.015575, 0.015514, 1.000000, 0.055889, 0.023992, 0.056601, 1.000000, 0.027849, 0.044917, 0.040128, 0.043628, 0.999997, 0.322927, 0.238815, 0.311405, 0.999997, 0.222693, 0.016559, 0.105274, 0.129951, 0.014097, 0.089857, 0.188346, 0.037904, 0.087388, 0.192120, 0.059029, 0.083990, 0.177890, 0.074478, 0.069913, 0.145329, 0.041177, 0.036800, 0.039655, 0.999998, 0.271390, 0.203172, 0.282521, 0.999998, 0.193682, 0.087733, 0.107510, 0.128318, 0.040024, 0.088640, 0.191875, 0.008532, 0.086990, 0.195138, -0.005823, 0.083213, 0.180562, -0.019596, 0.069257, 0.148124] + } +} diff --git a/cosmos_framework/data/vfm/action/normalizers/libero_native_frame_wise_relative_rot6d.json b/cosmos_framework/data/vfm/action/normalizers/libero_native_frame_wise_relative_rot6d.json new file mode 100644 index 0000000..6cde670 --- /dev/null +++ b/cosmos_framework/data/vfm/action/normalizers/libero_native_frame_wise_relative_rot6d.json @@ -0,0 +1,37 @@ +{ + "metadata": { + "embodiment_type": "libero", + "pose_convention": "frame_wise_relative", + "pose_coordinate_frame": "native", + "rotation_format": "6d", + "action_dim": 10, + "skip_rotation_dims": [3, 4, 5, 6, 7, 8], + "chunk_length": 16, + "sample_stride": null, + "dataset_name": "libero", + "dataset_class": "LIBERODataset", + "dataset_root": ["outputs/libero_datasets/libero_10", "outputs/libero_datasets/libero_object", "outputs/libero_datasets/libero_spatial", "outputs/libero_datasets/libero_goal"], + "_comment": "Dataset paths are placeholders; the statistics values are independent of local dataset location.", + "split": "train", + "num_samples_stats": 10000, + "reservoir_size": 50000, + "max_samples": 10000, + "sampling_seed": 42 + }, + "global": { + "mean": [ 0.050704, 0.097407, -0.094833, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.476725], + "std": [ 0.333621, 0.387175, 0.457140, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.499460], + "min": [-0.937500, -0.937500, -0.937500, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000], + "max": [ 0.937500, 0.937500, 0.937500, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000], + "q01": [-0.723214, -0.808929, -0.937500, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000], + "q99": [ 0.937500, 0.870536, 0.937500, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000] + }, + "global_raw": { + "mean": [ 0.050704, 0.097407, -0.094833, 0.994873, -0.004579, -0.004288, 0.004389, 0.996104, 0.001109, 0.476725], + "std": [ 0.333621, 0.387175, 0.457140, 0.010807, 0.077802, 0.063386, 0.078571, 0.009994, 0.038504, 0.499460], + "min": [-0.937500, -0.937500, -0.937500, 0.902028, -0.356085, -0.367416, -0.370434, 0.921907, -0.255000, 0.000000], + "max": [ 0.937500, 0.937500, 0.937500, 1.000000, 0.368853, 0.341214, 0.356395, 1.000000, 0.348251, 1.000000], + "q01": [-0.723214, -0.808929, -0.937500, 0.934955, -0.223431, -0.189878, -0.334735, 0.938516, -0.107736, 0.000000], + "q99": [ 0.937500, 0.870536, 0.937500, 1.000000, 0.331000, 0.163153, 0.226216, 1.000000, 0.127158, 1.000000] + } +} diff --git a/cosmos_framework/data/vfm/action/normalizers/robomind-franka-dual_backward_framewise_rot6d.json b/cosmos_framework/data/vfm/action/normalizers/robomind-franka-dual_backward_framewise_rot6d.json new file mode 100644 index 0000000..6190715 --- /dev/null +++ b/cosmos_framework/data/vfm/action/normalizers/robomind-franka-dual_backward_framewise_rot6d.json @@ -0,0 +1,33 @@ +{ + "metadata": { + "embodiment_type": "robomind-franka-dual", + "pose_convention": "backward_framewise", + "rotation_format": "rot6d", + "action_dim": 20, + "skip_rotation_dims": [3, 4, 5, 6, 7, 8, 13, 14, 15, 16, 17, 18], + "chunk_length": 16, + "sample_stride": 16, + "dataset_name": "robomind_franka_dual_20260414", + "dataset_class": "RoboMINDFrankaDataset", + "dataset_root": "", + "split": "train", + "num_samples_stats": 21410, + "reservoir_size": 5000000 + }, + "global": { + "mean": [ 0.000231, 0.000179, -0.000319, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.638652, 0.000148, -0.000377, -0.000241, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.815273], + "std": [ 0.014881, 0.008081, 0.014371, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.464058, 0.010628, 0.005868, 0.007900, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.366049], + "min": [-0.115093, -0.096415, -0.112595, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000, -0.091252, -0.052148, -0.113650, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000], + "max": [ 0.114941, 0.063433, 0.098721, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.123908, 0.077951, 0.080229, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.999628], + "q01": [-0.051367, -0.031964, -0.046482, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000, -0.035108, -0.021212, -0.029788, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000], + "q99": [ 0.043729, 0.021737, 0.036738, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.047581, 0.021270, 0.025712, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.995443] + }, + "global_raw": { + "mean": [ 0.000231, 0.000179, -0.000319, 0.999350, 0.000318, -0.000135, -0.000286, 0.999586, -0.000051, 0.638652, 0.000148, -0.000377, -0.000241, 0.999294, 0.000343, 0.000431, -0.000147, 0.999580, -0.000570, 0.815273], + "std": [ 0.014881, 0.008081, 0.014371, 0.002235, 0.020196, 0.029781, 0.020185, 0.001125, 0.020484, 0.464058, 0.010628, 0.005868, 0.007900, 0.002664, 0.025404, 0.027550, 0.025193, 0.001657, 0.014210, 0.366049], + "min": [-0.115093, -0.096415, -0.112595, 0.944314, -0.271877, -0.325264, -0.254808, 0.962274, -0.227188, 0.000000, -0.091252, -0.052148, -0.113650, 0.941406, -0.265241, -0.273484, -0.290840, 0.954990, -0.264631, 0.000000], + "max": [ 0.114941, 0.063433, 0.098721, 1.000000, 0.258475, 0.270230, 0.271943, 1.000000, 0.221936, 1.000000, 0.123908, 0.077951, 0.080229, 1.000000, 0.296517, 0.333596, 0.269131, 1.000000, 0.139695, 0.999628], + "q01": [-0.051367, -0.031964, -0.046482, 0.988101, -0.053179, -0.128603, -0.075432, 0.994427, -0.059973, 0.000000, -0.035108, -0.021212, -0.029788, 0.986086, -0.098043, -0.111441, -0.093441, 0.991492, -0.058030, 0.000000], + "q99": [ 0.043729, 0.021737, 0.036738, 1.000000, 0.075612, 0.102791, 0.053223, 1.000000, 0.077057, 1.000000, 0.047581, 0.021270, 0.025712, 1.000000, 0.095525, 0.126049, 0.098778, 1.000000, 0.041914, 0.995443] + } +} diff --git a/cosmos_framework/data/vfm/action/normalizers/robomind-franka_backward_framewise_rot6d.json b/cosmos_framework/data/vfm/action/normalizers/robomind-franka_backward_framewise_rot6d.json new file mode 100644 index 0000000..b99966e --- /dev/null +++ b/cosmos_framework/data/vfm/action/normalizers/robomind-franka_backward_framewise_rot6d.json @@ -0,0 +1,33 @@ +{ + "metadata": { + "embodiment_type": "robomind-franka", + "pose_convention": "backward_framewise", + "rotation_format": "rot6d", + "action_dim": 10, + "skip_rotation_dims": [3, 4, 5, 6, 7, 8], + "chunk_length": 16, + "sample_stride": 16, + "dataset_name": "robomind_franka_20260414", + "dataset_class": "RoboMINDFrankaDataset", + "dataset_root": "", + "split": "train", + "num_samples_stats": 141658, + "reservoir_size": 5000000 + }, + "global": { + "mean": [ 0.000241, 0.000073, -0.000597, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.630501], + "std": [ 0.020545, 0.010725, 0.022054, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.434021], + "min": [-0.184377, -0.130924, -0.183947, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000], + "max": [ 0.227682, 0.134118, 0.133222, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000], + "q01": [-0.065029, -0.030683, -0.075321, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000], + "q99": [ 0.068546, 0.036309, 0.051772, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000] + }, + "global_raw": { + "mean": [ 0.000241, 0.000073, -0.000597, 0.998782, 0.000605, 0.000003, -0.000592, 0.998245, -0.000508, 0.630501], + "std": [ 0.020545, 0.010725, 0.022054, 0.004056, 0.043101, 0.023671, 0.043102, 0.004948, 0.040306, 0.434021], + "min": [-0.184377, -0.130924, -0.183947, 0.837403, -0.525301, -0.384252, -0.543663, 0.801190, -0.490979, 0.000000], + "max": [ 0.227682, 0.134118, 0.133222, 1.000000, 0.543800, 0.389145, 0.522029, 1.000000, 0.414190, 1.000000], + "q01": [-0.065029, -0.030683, -0.075321, 0.981664, -0.137429, -0.069593, -0.140220, 0.976885, -0.140399, 0.000000], + "q99": [ 0.068546, 0.036309, 0.051772, 1.000000, 0.140290, 0.079942, 0.137529, 1.000000, 0.113651, 1.000000] + } +} diff --git a/cosmos_framework/data/vfm/action/normalizers/robomind-ur_backward_framewise_rot6d.json b/cosmos_framework/data/vfm/action/normalizers/robomind-ur_backward_framewise_rot6d.json new file mode 100644 index 0000000..e424f7c --- /dev/null +++ b/cosmos_framework/data/vfm/action/normalizers/robomind-ur_backward_framewise_rot6d.json @@ -0,0 +1,33 @@ +{ + "metadata": { + "embodiment_type": "robomind-ur", + "pose_convention": "backward_framewise", + "rotation_format": "rot6d", + "action_dim": 10, + "skip_rotation_dims": [3, 4, 5, 6, 7, 8], + "chunk_length": 16, + "sample_stride": 16, + "dataset_name": "robomind_ur_20260501", + "dataset_class": "RoboMINDURDataset", + "dataset_root": "", + "split": "train", + "num_samples_stats": 211078, + "reservoir_size": 5000000 + }, + "global": { + "mean": [-0.000470, 0.000182, -0.000724, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.688744], + "std": [ 0.014480, 0.011243, 0.013583, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 0.371170], + "min": [-0.204206, -0.132684, -0.164591, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000], + "max": [ 0.400660, 0.354529, 0.098898, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000], + "q01": [-0.050087, -0.032055, -0.045222, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 0.000000], + "q99": [ 0.040385, 0.036577, 0.032376, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000, 1.000000] + }, + "global_raw": { + "mean": [-0.000470, 0.000182, -0.000724, 0.998910, 0.001998, 0.000529, -0.001703, 0.998614, -0.000210, 0.688744], + "std": [ 0.014480, 0.011243, 0.013583, 0.003004, 0.037484, 0.027562, 0.037337, 0.003758, 0.036869, 0.371170], + "min": [-0.204206, -0.132684, -0.164591, -0.202474, -0.937542, -0.284120, -0.671888, -0.110612, -0.442271, 0.000000], + "max": [ 0.400660, 0.354529, 0.098898, 1.000000, 0.707452, 0.399441, 0.944798, 1.000000, 0.398811, 1.000000], + "q01": [-0.050087, -0.032055, -0.045222, 0.986554, -0.114343, -0.074432, -0.128455, 0.982298, -0.132509, 0.000000], + "q99": [ 0.040385, 0.036577, 0.032376, 1.000000, 0.129744, 0.095558, 0.115080, 1.000000, 0.121987, 1.000000] + } +} diff --git a/cosmos_framework/data/vfm/action/normalizers/uva_umi_single_task_anchored_normalizer.json b/cosmos_framework/data/vfm/action/normalizers/uva_umi_single_task_anchored_normalizer.json new file mode 100644 index 0000000..2c68ef6 --- /dev/null +++ b/cosmos_framework/data/vfm/action/normalizers/uva_umi_single_task_anchored_normalizer.json @@ -0,0 +1,85 @@ +{ + "robot0_main_camera": { + "type": "identity", + "scale": 1.0, + "offset": 0.0 + }, + "camera_poses": { + "type": "clamped_range", + "scale": [ + 0.25000008940696716, + 0.25000008940696716, + 0.25000008940696716, + 0.0750000849366188, + 0.25000008940696716, + 0.25000008940696716, + 0.25000008940696716, + 0.0750000849366188, + 0.25000008940696716 + ], + "offset": [ + 0.0, + 0.0, + 0.0, + 0.925000011920929, + 0.0, + 0.0, + 0.0, + 0.925000011920929, + 0.0 + ] + }, + "eef_poses": { + "type": "clamped_range", + "scale": [ + 0.25000008940696716, + 0.25000008940696716, + 0.25000008940696716, + 0.0750000849366188, + 0.25000008940696716, + 0.25000008940696716, + 0.25000008940696716, + 0.0750000849366188, + 0.25000008940696716 + ], + "offset": [ + 0.0, + 0.0, + 0.0, + 0.925000011920929, + 0.0, + 0.0, + 0.0, + 0.925000011920929, + 0.0 + ] + }, + "eef_commands": { + "type": "clamped_range", + "scale": [ + 0.05000010132789612 + ], + "offset": [ + 0.05000000074505806 + ] + }, + "grasp_states": { + "type": "clamped_range", + "scale": [ + 0.1, + 0.1, + 0.1, + 0.1, + 0.1, + 0.1 + ], + "offset": [ + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0 + ] + } +} diff --git a/cosmos_framework/data/vfm/action/normalizers/uva_umi_single_task_frame_wise_no_rot_normalizer.json b/cosmos_framework/data/vfm/action/normalizers/uva_umi_single_task_frame_wise_no_rot_normalizer.json new file mode 100644 index 0000000..2cdaaa9 --- /dev/null +++ b/cosmos_framework/data/vfm/action/normalizers/uva_umi_single_task_frame_wise_no_rot_normalizer.json @@ -0,0 +1,85 @@ +{ + "robot0_main_camera": { + "type": "identity", + "scale": 1.0, + "offset": 0.0 + }, + "camera_poses": { + "type": "clamped_range", + "scale": [ + 0.08000008940696716, + 0.08000008940696716, + 0.08000008940696716, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0 + ], + "offset": [ + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0 + ] + }, + "eef_poses": { + "type": "clamped_range", + "scale": [ + 0.08000008940696716, + 0.08000008940696716, + 0.08000008940696716, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0 + ], + "offset": [ + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0 + ] + }, + "eef_commands": { + "type": "clamped_range", + "scale": [ + 0.10000010132789612 + ], + "offset": [ + 0.10000000074505806 + ] + }, + "grasp_states": { + "type": "clamped_range", + "scale": [ + 0.1, + 0.1, + 0.1, + 0.1, + 0.1, + 0.1 + ], + "offset": [ + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0 + ] + } +} diff --git a/cosmos_framework/data/vfm/action/normalizers/uva_umi_single_task_frame_wise_normalizer.json b/cosmos_framework/data/vfm/action/normalizers/uva_umi_single_task_frame_wise_normalizer.json new file mode 100644 index 0000000..3674e26 --- /dev/null +++ b/cosmos_framework/data/vfm/action/normalizers/uva_umi_single_task_frame_wise_normalizer.json @@ -0,0 +1,85 @@ +{ + "robot0_main_camera": { + "type": "identity", + "scale": 1.0, + "offset": 0.0 + }, + "camera_poses": { + "type": "clamped_range", + "scale": [ + 0.08000008940696716, + 0.08000008940696716, + 0.08000008940696716, + 0.0050000849366188, + 0.10000008940696716, + 0.10000008940696716, + 0.10000008940696716, + 0.0050000849366188, + 0.10000008940696716 + ], + "offset": [ + 0.0, + 0.0, + 0.0, + 0.995000011920929, + 0.0, + 0.0, + 0.0, + 0.995000011920929, + 0.0 + ] + }, + "eef_poses": { + "type": "clamped_range", + "scale": [ + 0.08000008940696716, + 0.08000008940696716, + 0.08000008940696716, + 0.0050000849366188, + 0.10000008940696716, + 0.10000008940696716, + 0.10000008940696716, + 0.0050000849366188, + 0.10000008940696716 + ], + "offset": [ + 0.0, + 0.0, + 0.0, + 0.995000011920929, + 0.0, + 0.0, + 0.0, + 0.995000011920929, + 0.0 + ] + }, + "eef_commands": { + "type": "clamped_range", + "scale": [ + 0.10000010132789612 + ], + "offset": [ + 0.10000000074505806 + ] + }, + "grasp_states": { + "type": "clamped_range", + "scale": [ + 0.1, + 0.1, + 0.1, + 0.1, + 0.1, + 0.1 + ], + "offset": [ + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0 + ] + } +} diff --git a/cosmos_framework/data/vfm/action/normalizers/uva_umi_single_task_normalizer.json b/cosmos_framework/data/vfm/action/normalizers/uva_umi_single_task_normalizer.json new file mode 100644 index 0000000..2c68ef6 --- /dev/null +++ b/cosmos_framework/data/vfm/action/normalizers/uva_umi_single_task_normalizer.json @@ -0,0 +1,85 @@ +{ + "robot0_main_camera": { + "type": "identity", + "scale": 1.0, + "offset": 0.0 + }, + "camera_poses": { + "type": "clamped_range", + "scale": [ + 0.25000008940696716, + 0.25000008940696716, + 0.25000008940696716, + 0.0750000849366188, + 0.25000008940696716, + 0.25000008940696716, + 0.25000008940696716, + 0.0750000849366188, + 0.25000008940696716 + ], + "offset": [ + 0.0, + 0.0, + 0.0, + 0.925000011920929, + 0.0, + 0.0, + 0.0, + 0.925000011920929, + 0.0 + ] + }, + "eef_poses": { + "type": "clamped_range", + "scale": [ + 0.25000008940696716, + 0.25000008940696716, + 0.25000008940696716, + 0.0750000849366188, + 0.25000008940696716, + 0.25000008940696716, + 0.25000008940696716, + 0.0750000849366188, + 0.25000008940696716 + ], + "offset": [ + 0.0, + 0.0, + 0.0, + 0.925000011920929, + 0.0, + 0.0, + 0.0, + 0.925000011920929, + 0.0 + ] + }, + "eef_commands": { + "type": "clamped_range", + "scale": [ + 0.05000010132789612 + ], + "offset": [ + 0.05000000074505806 + ] + }, + "grasp_states": { + "type": "clamped_range", + "scale": [ + 0.1, + 0.1, + 0.1, + 0.1, + 0.1, + 0.1 + ], + "offset": [ + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0 + ] + } +} diff --git a/cosmos_framework/data/vfm/action/pose_utils.py b/cosmos_framework/data/vfm/action/pose_utils.py new file mode 100644 index 0000000..12ea51b --- /dev/null +++ b/cosmos_framework/data/vfm/action/pose_utils.py @@ -0,0 +1,747 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Rotation and pose utilities for action datasets. + +This module centralizes three related responsibilities used across the action +dataset stack: + +1. Converting rotations between the conventions used by the datasets and the + action model (`euler_xyz`, quaternion, axis-angle, rot6d, rot9d, matrix). +2. Building absolute homogeneous poses of shape ``(T, 4, 4)`` from per-frame + translation and rotation components. +3. Converting trajectories between absolute-pose form and the relative-pose + action vectors consumed by the datasets. + + The relative-pose action vectors always follow the shared layout + ``[translation(3), rotation(...)]``. The rotation block is encoded with the + requested rotation output convention, and `convert_rotation()` is the + canonical public entrypoint for representation conversion. +""" + +import math +from typing import Literal + +import numpy as np +import torch +from scipy.spatial.transform import Rotation as R + +PoseConvention = Literal["absolute", "backward_anchored", "backward_framewise"] +RotationConvention = Literal["matrix", "euler_xyz", "quat_xyzw", "quat_wxyz", "rot6d", "axisangle", "rot9d"] + + +def _to_numpy_float32(array: torch.Tensor | np.ndarray) -> np.ndarray: + """Convert an input array to a NumPy ``float32`` array. + + Args: + array: A torch tensor or NumPy array with arbitrary leading dimensions. + + Returns: + A NumPy array with dtype ``float32``. Torch tensors are moved to CPU + before conversion. NumPy inputs are converted with ``copy=False`` + semantics when possible. + + Raises: + ValueError: If a torch tensor with ``requires_grad=True`` is passed. + These utilities are non-differentiable; callers must explicitly + detach tensors before conversion. + """ + if isinstance(array, torch.Tensor): + if array.requires_grad: + raise ValueError( + "pose_utils conversion is non-differentiable; call `.detach()` " + "explicitly before passing tensors with requires_grad=True" + ) + return array.cpu().numpy().astype(np.float32, copy=False) + return np.asarray(array, dtype=np.float32) + + +def _normalize_rotation_matrices(rot_matrices: np.ndarray) -> np.ndarray: + """Project approximate matrices onto valid rotation matrices. + + This helper uses an SVD-based projection onto ``SO(3)``. It is mainly used + when decoding rotations from network-like representations such as rot6d or rot9d + where the input may not already be perfectly orthonormal. + + Args: + rot_matrices: Array of shape ``(..., 3, 3)`` containing one or more + approximate rotation matrices. + + Returns: + Array of shape ``(..., 3, 3)`` whose trailing matrices are proper + rotation matrices with determinant ``+1``. + + Raises: + ValueError: If the input does not have trailing shape ``(3, 3)``. + """ + matrices = np.asarray(rot_matrices, dtype=np.float32) + if matrices.ndim < 2 or matrices.shape[-2:] != (3, 3): + raise ValueError(f"Rotation matrices must have shape (..., 3, 3), got {matrices.shape}") + + original_shape = matrices.shape[:-2] + matrices_flat = matrices.reshape(-1, 3, 3) + + # Batched SVD projection to SO(3). + U, _, Vt = np.linalg.svd(matrices_flat) + normalized = U @ Vt + + # Ensure determinant is +1 (proper rotations, no reflections). + det = np.linalg.det(normalized) + reflection_mask = det < 0 + if np.any(reflection_mask): + U_reflect = U.copy() + U_reflect[reflection_mask, :, -1] *= -1 + normalized[reflection_mask] = U_reflect[reflection_mask] @ Vt[reflection_mask] + + return normalized.astype(np.float32, copy=False).reshape(*original_shape, 3, 3) + + +def convert_rotation( + rotation: torch.Tensor | np.ndarray, + input_format: RotationConvention, + output_format: RotationConvention, + normalize_matrix: bool = False, +) -> torch.Tensor | np.ndarray: + """Convert rotations between the conventions used by action datasets. + + The function first maps the input representation to rotation matrices and + then emits the requested output convention. It is the single conversion seam + used by the public pose helpers so that all code paths share the same + convention handling. + + Supported input conventions: + - ``matrix``: rotation matrices with shape ``(..., 3, 3)`` + - ``euler_xyz``: Euler xyz angles in radians with shape ``(..., 3)`` + - ``quat_xyzw``: quaternions in SciPy's xyzw order with shape ``(..., 4)`` + - ``quat_wxyz``: quaternions in wxyz order with shape ``(..., 4)`` + - ``rot6d``: column-based 6D representation with shape ``(..., 6)`` + - ``rot9d``: flattened rotation matrices with shape ``(..., 9)`` + - ``axisangle``: axis-angle vectors with shape ``(..., 3)`` + + Supported output conventions: + - ``matrix`` + - ``euler_xyz`` + - ``quat_xyzw`` + - ``quat_wxyz`` + - ``rot6d`` + - ``axisangle`` + - ``rot9d`` + + Args: + rotation: Input rotations in the representation specified by + ``input_format``. + input_format: Convention used by ``rotation``. + output_format: Convention to return. + normalize_matrix: Whether to project intermediate matrices to a valid + rotation before returning. This is most useful when decoding from + approximate ``rot6d``/``rot9d`` inputs or non-unit quaternions. + + Returns: + Rotations with the same leading shape as the input, expressed in the + requested output convention. Torch inputs return torch outputs on the + same device with the same dtype; NumPy inputs return NumPy arrays. + + Raises: + ValueError: If the input shape is incompatible with ``input_format`` or + if either format is unsupported. + """ + input_is_tensor = isinstance(rotation, torch.Tensor) + input_dtype = rotation.dtype if input_is_tensor else None + input_device = rotation.device if input_is_tensor else None + rotation_np = _to_numpy_float32(rotation) + + if input_format == "matrix": + if rotation_np.ndim < 2 or rotation_np.shape[-2:] != (3, 3): + raise ValueError(f"matrix rotation must have shape (..., 3, 3), got {rotation_np.shape}") + original_shape = rotation_np.shape[:-2] + matrices_flat = rotation_np.reshape(-1, 3, 3) + if normalize_matrix: + matrices_flat = _normalize_rotation_matrices(matrices_flat).reshape(-1, 3, 3) + elif input_format == "euler_xyz": + if rotation_np.ndim < 1 or rotation_np.shape[-1] != 3: + raise ValueError(f"{input_format} rotation must have shape (..., 3), got {rotation_np.shape}") + original_shape = rotation_np.shape[:-1] + matrices_flat = R.from_euler("xyz", rotation_np.reshape(-1, 3), degrees=False).as_matrix().astype(np.float32) + elif input_format in ("quat_xyzw", "quat_wxyz"): + if rotation_np.ndim < 1 or rotation_np.shape[-1] != 4: + raise ValueError(f"{input_format} rotation must have shape (..., 4), got {rotation_np.shape}") + original_shape = rotation_np.shape[:-1] + quaternions = rotation_np.reshape(-1, 4) + if input_format == "quat_wxyz": + quaternions = quaternions[:, [1, 2, 3, 0]] + norms = np.linalg.norm(quaternions, axis=-1) + if np.any(norms < 1e-8): + raise ValueError(f"Found zero-norm quaternion(s) (min norm={norms.min():.2e}).") + if normalize_matrix: + quaternions = quaternions / norms[:, None] + matrices_flat = R.from_quat(quaternions).as_matrix().astype(np.float32) + elif input_format == "rot6d": + if rotation_np.ndim < 1 or rotation_np.shape[-1] != 6: + raise ValueError(f"{input_format} rotation must have shape (..., 6), got {rotation_np.shape}") + original_shape = rotation_np.shape[:-1] + rot6d_flat = rotation_np.reshape(-1, 6) + col0 = rot6d_flat[:, :3] + col1 = rot6d_flat[:, 3:] + col2 = np.cross(col0, col1, axis=-1) + matrices_flat = np.stack((col0, col1, col2), axis=-1).astype(np.float32) + if normalize_matrix: + matrices_flat = _normalize_rotation_matrices(matrices_flat).reshape(-1, 3, 3) + elif input_format == "rot9d": + if rotation_np.ndim < 1 or rotation_np.shape[-1] != 9: + raise ValueError(f"rot9d rotation must have shape (..., 9), got {rotation_np.shape}") + original_shape = rotation_np.shape[:-1] + matrices_flat = rotation_np.reshape(-1, 3, 3) + if normalize_matrix: + matrices_flat = _normalize_rotation_matrices(matrices_flat).reshape(-1, 3, 3) + elif input_format == "axisangle": + if rotation_np.ndim < 1 or rotation_np.shape[-1] != 3: + raise ValueError(f"axisangle rotation must have shape (..., 3), got {rotation_np.shape}") + original_shape = rotation_np.shape[:-1] + matrices_flat = R.from_rotvec(rotation_np.reshape(-1, 3)).as_matrix().astype(np.float32) + else: + raise ValueError(f"Unsupported input_format: {input_format!r}") + + if output_format == "matrix": + converted = matrices_flat.reshape(*original_shape, 3, 3).astype(np.float32) + elif output_format == "rot9d": + converted = matrices_flat.reshape(-1, 9) + elif output_format == "rot6d": + converted = matrices_flat[:, :, :2].transpose(0, 2, 1).reshape(-1, 6) + elif output_format == "quat_xyzw": + converted = R.from_matrix(matrices_flat).as_quat().astype(np.float32) + elif output_format == "quat_wxyz": + converted = R.from_matrix(matrices_flat).as_quat().astype(np.float32) + converted = converted[:, [3, 0, 1, 2]] + elif output_format == "euler_xyz": + converted = R.from_matrix(matrices_flat).as_euler("xyz", degrees=False).astype(np.float32) + elif output_format == "axisangle": + converted = R.from_matrix(matrices_flat).as_rotvec().astype(np.float32) + else: + raise ValueError(f"Unsupported output_format: {output_format!r}") + + if output_format != "matrix": + converted = converted.reshape(*original_shape, converted.shape[-1]) + + if input_is_tensor: + return torch.from_numpy(np.ascontiguousarray(converted)).to(dtype=input_dtype, device=input_device) + return converted + + +# ----------------------------------------------------------------------------- +# Absolute pose construction +# ----------------------------------------------------------------------------- + + +def build_abs_pose_from_components( + xyz: torch.Tensor | np.ndarray, + rotation: torch.Tensor | np.ndarray, + rotation_input_format: Literal["euler_xyz", "quat_xyzw", "quat_wxyz", "axisangle"], + translation_scale: float | None = None, +) -> np.ndarray: + """Build absolute homogeneous poses from per-frame translation and rotation. + + This is the canonical helper for turning dataset-provided pose components + into a sequence of rigid transforms. Each output pose is a homogeneous + transform whose top-left ``3 x 3`` block stores rotation and whose last + column stores translation. + + Args: + xyz: Per-frame translations with shape ``(T, 3)``. + rotation: Per-frame rotations with shape ``(T, 3)`` for ``euler_xyz`` + and ``axisangle``, or ``(T, 4)`` for quaternion conventions. + rotation_input_format: Convention used by ``rotation``. Supported values + are ``euler_xyz``, ``quat_xyzw``, ``quat_wxyz``, and ``axisangle``. + translation_scale: Optional factor used to divide translations before + inserting them into the output poses. This is useful when upstream + data stores translations in a scaled unit. + + Returns: + Absolute poses with shape ``(T, 4, 4)`` and dtype ``float32``. + + Raises: + ValueError: If the translation and rotation arrays have incompatible + lengths or unsupported shapes, or if ``translation_scale`` is zero. + """ + xyz_np = _to_numpy_float32(xyz) + rotation_np = _to_numpy_float32(rotation) + + if xyz_np.ndim != 2 or xyz_np.shape[1] != 3: + raise ValueError(f"xyz must have shape (T, 3), got {xyz_np.shape}") + if rotation_np.ndim != 2: + raise ValueError(f"rotation must be 2D, got {rotation_np.shape}") + if rotation_np.shape[0] != xyz_np.shape[0]: + raise ValueError( + f"xyz and rotation must have the same length, got {xyz_np.shape[0]} and {rotation_np.shape[0]}" + ) + + rot_mats = np.asarray( + convert_rotation(rotation_np, input_format=rotation_input_format, output_format="matrix"), + dtype=np.float32, + ) + + if translation_scale is not None: + if translation_scale == 0: + raise ValueError("translation_scale must be non-zero") + xyz_np = xyz_np / float(translation_scale) + + poses_abs = np.eye(4, dtype=np.float32)[None].repeat(xyz_np.shape[0], axis=0) + poses_abs[:, :3, :3] = rot_mats.astype(np.float32) + poses_abs[:, :3, 3] = xyz_np + return poses_abs + + +# ----------------------------------------------------------------------------- +# Relative pose conversions +# ----------------------------------------------------------------------------- + + +def _delta_transform_to_pose_vector( + delta_T: np.ndarray, + rotation_output_format: RotationConvention, + translation_scale: float = 1.0, + rotation_scale: float = 1.0, +) -> np.ndarray: + """Encode a relative transform as an action vector. + + The shared action-vector layout is always ``[translation(3), rotation(...)]``. + The translation block is multiplied by ``translation_scale`` before concatenation, + and the rotation block is multiplied by ``rotation_scale``. + + Args: + delta_T: Relative transform of shape ``(4, 4)``. + rotation_output_format: Concrete convention used for the output rotation + block. + translation_scale: Scalar multiplier applied to the translation block. + rotation_scale: Scalar multiplier applied to the rotation block. Used to + match the loss scale of the rotation block to the translation block. + The decoder must divide by the same factor before reconstructing the + rotation matrix. + + Returns: + A ``float32`` action vector whose first three values are translation and + whose remaining values are the rotation in ``rotation_output_format``. + """ + delta_np = np.asarray(delta_T, dtype=np.float32) + if delta_np.shape != (4, 4): + raise ValueError(f"delta_T must have shape (4, 4), got {delta_np.shape}") + + translation = delta_np[:3, 3] * translation_scale + rotation = np.asarray( + convert_rotation(delta_np[:3, :3], input_format="matrix", output_format=rotation_output_format), + dtype=np.float32, + ) + rotation = rotation * rotation_scale + return np.concatenate([translation, rotation]).astype(np.float32) + + +def _pose_vector_to_delta_transform( + pose_vector: np.ndarray, + rotation_input_format: RotationConvention, + translation_scale: float, + normalize_rotation: bool, + rotation_scale: float = 1.0, +) -> np.ndarray: + """Decode an action vector back into a relative homogeneous transform. + + This is the inverse of `_delta_transform_to_pose_vector()` when the same + rotation convention and scale are used. + + Args: + pose_vector: Relative-pose action vector with layout + ``[translation(3), rotation(...)]``. + rotation_input_format: Concrete convention used by the rotation block. + translation_scale: Scalar used to undo the translation scaling applied during + encoding. + normalize_rotation: Whether to project the decoded rotation to a valid + matrix before assembling the transform. + rotation_scale: Scalar used to undo the rotation scaling applied during + encoding. Must match the value used by + `_delta_transform_to_pose_vector()`. + + Returns: + A relative homogeneous transform with shape ``(4, 4)`` and dtype + ``float32``. + """ + pose_vector_np = np.asarray(pose_vector, dtype=np.float32) + rotation_block = pose_vector_np[3:] / rotation_scale + + rotation_matrix = np.asarray( + convert_rotation( + rotation_block, + input_format=rotation_input_format, + output_format="matrix", + normalize_matrix=normalize_rotation, + ), + dtype=np.float32, + ) + + delta_T = np.eye(4, dtype=np.float32) + delta_T[:3, 3] = pose_vector_np[:3] / translation_scale + delta_T[:3, :3] = rotation_matrix + return delta_T + + +def _get_relative_delta_transform( + poses_abs: np.ndarray, + inv_poses_abs: np.ndarray, + frame_idx: int, + pose_convention: PoseConvention, +) -> np.ndarray: + """Compute one relative transform from an absolute-pose trajectory. + + Args: + poses_abs: Absolute poses of shape ``(T, 4, 4)``. + inv_poses_abs: Precomputed inverses of ``poses_abs`` with the same shape. + frame_idx: Index of the step to encode, in ``[0, T - 2]``. + pose_convention: Pose convention controlling which two poses + define the delta and whether it is framewise or anchored. + + Returns: + The relative transform ``delta_T`` with shape ``(4, 4)`` for the + requested step and convention. + """ + if pose_convention == "backward_framewise": + return inv_poses_abs[frame_idx] @ poses_abs[frame_idx + 1] + if pose_convention == "backward_anchored": + return inv_poses_abs[0] @ poses_abs[frame_idx + 1] + raise ValueError( + f"Unsupported pose_convention={pose_convention!r}. Expected one of: backward_framewise, backward_anchored." + ) + + +def _apply_relative_delta_transform( + current_pose: np.ndarray, + initial_pose: np.ndarray, + delta_T: np.ndarray, + pose_convention: PoseConvention, +) -> np.ndarray: + """Recover the next absolute pose from a decoded relative transform. + + Args: + current_pose: The current reconstructed pose for framewise modes. + initial_pose: The anchor pose used by anchored modes. + delta_T: Relative transform for the current step. + pose_convention: Pose convention controlling how ``delta_T`` + should be composed back into an absolute pose. + + Returns: + The next absolute pose with shape ``(4, 4)``. + """ + if pose_convention == "backward_framewise": + return current_pose @ delta_T + if pose_convention == "backward_anchored": + return initial_pose @ delta_T + raise ValueError( + f"Unsupported pose_convention={pose_convention!r}. Expected one of: backward_framewise, backward_anchored." + ) + + +def pose_abs_to_rel( + poses_abs: np.ndarray, + rotation_format: RotationConvention = "rot9d", + pose_convention: PoseConvention = "backward_framewise", + translation_scale: float = 1.0, + rotation_scale: float = 1.0, +) -> np.ndarray: + """Convert an absolute-pose trajectory into relative-pose action vectors. + + Args: + poses_abs: Absolute poses with shape ``(T, 4, 4)``. These are typically + object-in-world or camera-to-world transforms. + rotation_format: Rotation convention used for the output rotation block. + Supported values are ``rot9d``, ``rot6d``, ``quat_xyzw``, and + ``euler_xyz``. + pose_convention: Pose convention: + - ``backward_framewise``: ``delta_T = T_i^{-1} @ T_{i+1}`` + - ``backward_anchored``: ``delta_T = T_0^{-1} @ T_{i+1}`` + translation_scale: Scalar multiplier applied to the translation block of each + encoded action vector. + rotation_scale: Scalar multiplier applied to the rotation block of each + encoded action vector. Use this to match the loss scale of rotation + and translation. `pose_rel_to_abs()` must be called with the same + value to invert the scaling. + + Returns: + An array of shape ``(T - 1, D)`` where ``D = 3 + rotation_dim``. + + Raises: + AssertionError: If fewer than two absolute poses are provided. + """ + num_frames = len(poses_abs) + assert num_frames > 1, "At least 2 frames are required to compute relative poses" + + # Compute inverse poses + inv_poses_abs = np.linalg.inv(poses_abs) + + poses_rel = [] + # We produce num_frames - 1 relative poses + for i in range(num_frames - 1): + delta_T = _get_relative_delta_transform(poses_abs, inv_poses_abs, i, pose_convention) + poses_rel.append( + _delta_transform_to_pose_vector( + delta_T, + rotation_output_format=rotation_format, + translation_scale=translation_scale, + rotation_scale=rotation_scale, + ) + ) + + return np.stack(poses_rel).astype(np.float32) # [T-1,D] + + +def pose_rel_to_abs( + poses_rel: np.ndarray, + rotation_format: RotationConvention = "rot9d", + pose_convention: PoseConvention = "backward_framewise", + initial_pose: np.ndarray | None = None, + normalize_rotation: bool = True, + translation_scale: float = 1.0, + rotation_scale: float = 1.0, +) -> np.ndarray: + """Reconstruct an absolute-pose trajectory from relative-pose action vectors. + + Args: + poses_rel: Relative-pose action vectors with shape ``(T - 1, D)`` and + layout ``[translation(3), rotation(...)]``. + rotation_format: Convention used by the rotation block of ``poses_rel``. + pose_convention: Pose convention used when the vectors were + encoded. This must match the convention passed to `pose_abs_to_rel()`. + initial_pose: Absolute pose for the first frame. If ``None``, the + identity transform is used. + normalize_rotation: Whether to project decoded rotations onto ``SO(3)`` + before composing them back into the trajectory. + translation_scale: Scalar used to undo the translation scaling applied during + `pose_abs_to_rel()`. + rotation_scale: Scalar used to undo the rotation scaling applied during + `pose_abs_to_rel()`. Must match the value passed there. + + Returns: + Absolute poses with shape ``(T, 4, 4)`` where ``T = len(poses_rel) + 1``. + """ + if initial_pose is None: + initial_pose = np.eye(4) + + poses_abs = [initial_pose] + current_pose = initial_pose + + num_poses_rel = poses_rel.shape[0] + + for i in range(num_poses_rel): + delta_T = _pose_vector_to_delta_transform( + poses_rel[i], + rotation_input_format=rotation_format, + translation_scale=translation_scale, + normalize_rotation=normalize_rotation, + rotation_scale=rotation_scale, + ) + next_pose = _apply_relative_delta_transform(current_pose, initial_pose, delta_T, pose_convention) + + poses_abs.append(next_pose) + current_pose = next_pose + + return np.stack(poses_abs) # [T,4,4] + + +# ----------------------------------------------------------------------------- +# Idle-frame detection +# ----------------------------------------------------------------------------- + + +def _identity_rotation_vector(rotation_format: RotationConvention) -> np.ndarray: + """Return the identity-rotation vector for a given rotation convention. + + Used by :func:`compute_idle_frames` to test whether a rotation block is + close to "no rotation" in its current encoding. + """ + if rotation_format in ("matrix", "rot9d"): + return np.array([1, 0, 0, 0, 1, 0, 0, 0, 1], dtype=np.float32) + if rotation_format == "rot6d": + return np.array([1, 0, 0, 0, 1, 0], dtype=np.float32) + if rotation_format == "quat_xyzw": + return np.array([0, 0, 0, 1], dtype=np.float32) + if rotation_format == "quat_wxyz": + return np.array([1, 0, 0, 0], dtype=np.float32) + if rotation_format in ("euler_xyz", "axisangle"): + return np.array([0, 0, 0], dtype=np.float32) + raise ValueError(f"Unsupported rotation_format={rotation_format!r}") + + +def _rotation_angle_per_arm(rotations: np.ndarray, rotation_format: str) -> np.ndarray: + """Geodesic angle (rad) from identity for each arm at each frame. + + ``rotations`` has shape ``(T, n_arms, n_per_arm)``; the returned array has + shape ``(T, n_arms)``. The angle is rotation-format aware so a fixed + ``eps_r`` threshold has consistent geometric meaning across formats: + + - ``rot6d`` → reconstruct ``trace(R)`` in closed form from the two stored + columns ``a, b`` (already unit-orthogonal as they came from a valid + rotation matrix). The third column is ``a × b``, so + ``trace(R) = a[0] + b[1] + a[0]·b[1] - a[1]·b[0]``. + ``angle = arccos(clip((trace - 1) / 2, -1, 1))``. + - ``rot9d`` → reshape to ``(..., 3, 3)`` and use + ``trace(R) = R[0,0] + R[1,1] + R[2,2]``. + - ``quat_xyzw`` / ``quat_wxyz`` → ``angle = 2 · arccos(|q_w|)``; the + absolute value handles the double cover (``q`` and ``-q`` represent the + same rotation). + - ``axisangle`` → the magnitude of the axis-angle vector *is* the angle. + - ``euler_xyz`` → no closed-form angle; use ``‖euler‖`` as a conservative + upper bound (exact for single-axis rotations, an overestimate for + composed ones — fine for idle detection where small angles are the + regime of interest). + """ + if rotation_format == "rot6d": + a = rotations[..., :3] + b = rotations[..., 3:6] + trace = a[..., 0] + b[..., 1] + a[..., 0] * b[..., 1] - a[..., 1] * b[..., 0] + return np.arccos(np.clip((trace - 1.0) / 2.0, -1.0, 1.0)) + if rotation_format == "rot9d": + mat = rotations.reshape(*rotations.shape[:-1], 3, 3) + trace = mat[..., 0, 0] + mat[..., 1, 1] + mat[..., 2, 2] + return np.arccos(np.clip((trace - 1.0) / 2.0, -1.0, 1.0)) + if rotation_format in ("quat_xyzw", "quat_wxyz"): + qw = rotations[..., 3] if rotation_format == "quat_xyzw" else rotations[..., 0] + return 2.0 * np.arccos(np.clip(np.abs(qw), 0.0, 1.0)) + if rotation_format == "axisangle": + return np.linalg.norm(rotations, axis=-1) + if rotation_format == "euler_xyz": + # Exact for single-axis rotations, overestimate for composed ones — + # safe for idle thresholds since overestimation can only mark a frame + # as non-idle, never spuriously idle. + return np.linalg.norm(rotations, axis=-1) + raise ValueError(f"Unsupported rotation_format={rotation_format!r}") + + +def _consecutive_streaks(idle: np.ndarray, min_streak: int) -> np.ndarray: + """Zero out idle bits not belonging to a run of ``>= min_streak`` Trues. + + Pure-numpy two-pointer scan. ``min_streak <= 1`` is a no-op (returns the + input mask unchanged). + """ + if min_streak <= 1: + return idle + out = np.zeros_like(idle) + n = len(idle) + i = 0 + while i < n: + if not idle[i]: + i += 1 + continue + j = i + while j < n and idle[j]: + j += 1 + if j - i >= min_streak: + out[i:j] = True + i = j + return out + + +def compute_idle_frames( + action_raw: torch.Tensor | np.ndarray, + spec: "ActionSpec", # noqa: F821 — forward ref, real import is in action_spec.py + *, + eps_t: float = 1e-3, + eps_r: float = math.radians(5.0), + eps_g: float = 1e-2, + joint_threshold: float = 5e-4, + min_streak: int = 3, +) -> int: + """Count idle frames in a raw (un-normalized) action chunk. + + Idle detection runs per-DimType (driven by ``spec.types``); a frame is + *raw-idle* iff every relevant type group is idle on that frame, and + counts toward the final tally only if it belongs to a run of at least + ``min_streak`` consecutive raw-idle frames. The streak filter rejects + isolated low-motion frames (instantaneous slowdowns) which carry weak + physical meaning and add noise to the IdleFrames training signal. + + DimType branches: + + - ``POS`` → combined ``‖action[pos_idx]‖`` (L2 across all POS dims) + < ``eps_t``. For single-arm specs (3 dims) this is the standard ``‖t‖`` + check; for multi-arm specs the combined norm is slightly stricter than + a per-arm check. + - ``ROT`` → per-arm geodesic rotation angle (rad) from identity + < ``eps_r``. The angle is computed in a rotation-format aware way (see + :func:`_rotation_angle_per_arm`) so the threshold has consistent + geometric meaning regardless of the encoding. + - ``GRIPPER`` → ``max |action[t] - action[t-1]| < eps_g``. ``np.diff`` + with ``prepend=action[0]`` makes step 0 ``|0|`` (treated as "no change"); + with the streak filter this can no longer create a spurious single-frame + idle event. + - ``JOINT`` → same frame-diff scheme as gripper with + ``joint_threshold`` (rad / step). + - ``RESERVED`` → ignored. + + Defaults (in the units of the un-normalized action): + + - ``eps_t = 1e-3`` → 1 mm per-frame translation + - ``eps_r = 5°`` → 5° per-frame rotation (geodesic angle) + - ``eps_g = 1e-2`` → 1 % gripper command change + - ``joint_threshold = 5e-4`` → ~0.03° / step joint angle change + - ``min_streak = 3`` → require a run of >= 3 consecutive idle frames + + The input must be **un-normalized** so the identity transform sits at + known coordinates (translation ≈ 0, rotation ≈ identity). The action + vector is also assumed to be encoded in a per-step / framewise convention + (e.g. ``backward_framewise``); anchored conventions (``backward_anchored``) + accumulate over the chunk and would silently break the POS/ROT idle + checks. Callers (e.g. the LeRobot base class) gate on pose convention + before calling this function. + """ + if isinstance(action_raw, torch.Tensor): + action = action_raw.detach().cpu().numpy().astype(np.float32, copy=False) + else: + action = np.asarray(action_raw, dtype=np.float32) + + if action.ndim != 2: + raise ValueError(f"action_raw must be 2-D (T, D); got shape {action.shape}") + num_frames, action_dim = action.shape + if num_frames == 0: + return 0 + if action_dim != len(spec.types): + raise ValueError(f"action_dim={action_dim} does not match spec.dim={len(spec.types)}") + + # Import locally to avoid a circular import at module load time + # (action_spec.py imports RotationConvention from this file). + from cosmos_framework.data.vfm.action.action_spec import DimType + + pos_idx = [i for i, t in enumerate(spec.types) if t == DimType.POS] + rot_idx = [i for i, t in enumerate(spec.types) if t == DimType.ROT] + grip_idx = [i for i, t in enumerate(spec.types) if t == DimType.GRIPPER] + joint_idx = [i for i, t in enumerate(spec.types) if t == DimType.JOINT] + + idle = np.ones(num_frames, dtype=bool) + + # POS: combined L2 norm across all translation dims. + if pos_idx: + idle &= np.linalg.norm(action[:, pos_idx], axis=1) < eps_t + + # ROT: per-arm geodesic angle (rad). + if rot_idx: + rot_id = _identity_rotation_vector(spec.rotation_format) + n_per_arm = rot_id.shape[0] + if len(rot_idx) % n_per_arm != 0: + raise ValueError( + f"ROT dims ({len(rot_idx)}) not a multiple of " + f"rotation_format={spec.rotation_format!r} dim ({n_per_arm})" + ) + rotations = action[:, rot_idx].reshape(num_frames, -1, n_per_arm) + angles = _rotation_angle_per_arm(rotations, spec.rotation_format) # (T, n_arms) + idle &= angles.max(axis=1) < eps_r + + # GRIPPER: max |Δgripper| across all gripper dims; step 0's diff is 0. + if grip_idx: + gripper = action[:, grip_idx] + diff = np.abs(np.diff(gripper, axis=0, prepend=gripper[:1])) + idle &= diff.max(axis=1) < eps_g + + # JOINT: same frame-diff scheme with joint_threshold. + if joint_idx: + joints = action[:, joint_idx] + diff = np.abs(np.diff(joints, axis=0, prepend=joints[:1])) + idle &= diff.max(axis=1) < joint_threshold + + if min_streak > 1: + idle = _consecutive_streaks(idle, min_streak) + + return int(idle.sum()) diff --git a/cosmos_framework/data/vfm/action/pose_utils_test.py b/cosmos_framework/data/vfm/action/pose_utils_test.py new file mode 100644 index 0000000..93f9791 --- /dev/null +++ b/cosmos_framework/data/vfm/action/pose_utils_test.py @@ -0,0 +1,209 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import numpy as np +import pytest +import torch +from scipy.spatial.transform import Rotation as R + +from cosmos_framework.data.vfm.action.pose_utils import ( + _normalize_rotation_matrices, + _to_numpy_float32, + build_abs_pose_from_components, + convert_rotation, + pose_abs_to_rel, + pose_rel_to_abs, +) + + +def _make_example_poses_abs() -> np.ndarray: + xyz = np.array( + [ + [1.0, -0.5, 0.25], + [1.5, 0.0, 0.75], + [2.0, 0.5, 1.5], + [2.5, 1.0, 2.0], + ], + dtype=np.float32, + ) + euler = np.array( + [ + [0.0, 0.0, 0.0], + [0.1, -0.2, 0.3], + [0.2, 0.15, -0.1], + [-0.25, 0.05, 0.4], + ], + dtype=np.float32, + ) + return build_abs_pose_from_components(xyz, euler, "euler_xyz") + + +@pytest.mark.L0 +def test_to_numpy_float32_raises_on_requires_grad_tensor() -> None: + """Tensor inputs with gradients must be explicitly detached by callers.""" + x = torch.randn(2, 3, requires_grad=True) + with pytest.raises(ValueError, match="non-differentiable"): + _to_numpy_float32(x) + + +@pytest.mark.L0 +def test_build_abs_pose_from_components_supports_quat_wxyz() -> None: + """AV-style wxyz quaternions should produce the same matrices as xyzw.""" + xyz = np.array([[0.0, 0.0, 0.0], [1.0, 2.0, 3.0]], dtype=np.float32) + quat_xyzw = np.array( + [ + [0.0, 0.0, 0.0, 1.0], + [0.0, 0.0, np.sin(np.pi / 4), np.cos(np.pi / 4)], + ], + dtype=np.float32, + ) + quat_wxyz = quat_xyzw[:, [3, 0, 1, 2]] + + poses_xyzw = build_abs_pose_from_components(xyz, quat_xyzw, "quat_xyzw") + poses_wxyz = build_abs_pose_from_components(xyz, quat_wxyz, "quat_wxyz") + + np.testing.assert_allclose(poses_xyzw, poses_wxyz, atol=1e-6) + + +@pytest.mark.L0 +def test_build_abs_pose_from_components_matches_manual_euler_conversion() -> None: + """Euler component helper should match the previous matrix-building pattern.""" + xyz = np.array( + [ + [0.0, 0.0, 0.0], + [1.0, 0.0, 0.0], + [1.0, 2.0, 0.0], + ], + dtype=np.float32, + ) + euler = np.array( + [ + [0.0, 0.0, 0.0], + [0.0, 0.0, np.pi / 2], + [0.0, np.pi / 4, np.pi / 2], + ], + dtype=np.float32, + ) + + poses_abs = build_abs_pose_from_components(xyz, euler, "euler_xyz") + manual_poses_abs = np.tile(np.eye(4, dtype=np.float32), (xyz.shape[0], 1, 1)) + manual_poses_abs[:, :3, :3] = R.from_euler("xyz", euler).as_matrix() + manual_poses_abs[:, :3, 3] = xyz + + actual = pose_abs_to_rel(poses_abs, rotation_format="rot6d", pose_convention="backward_framewise") + expected = pose_abs_to_rel(manual_poses_abs, rotation_format="rot6d", pose_convention="backward_framewise") + + np.testing.assert_allclose(actual, expected, atol=1e-6) + + +@pytest.mark.L0 +def test_build_abs_pose_from_components_applies_translation_scale() -> None: + """Explicit translation scaling should be applied before building pose matrices.""" + xyz = np.array( + [ + [0.0, 0.0, 0.0], + [0.5, 0.0, 0.0], + [1.5, 0.0, 0.0], + ], + dtype=np.float32, + ) + euler = np.zeros((3, 3), dtype=np.float32) + + poses_abs = build_abs_pose_from_components(xyz, euler, "euler_xyz", translation_scale=2.0) + + np.testing.assert_allclose(poses_abs[:, :3, 3], xyz / 2.0, atol=1e-6) + + +@pytest.mark.L0 +def test_pose_abs_to_rel_rotation_formats_follow_centralized_conventions() -> None: + """Relative-pose conversion should emit the canonical rot6d and euler_xyz blocks.""" + poses_abs = np.tile(np.eye(4, dtype=np.float32), (3, 1, 1)) + euler = np.array( + [ + [0.0, 0.0, 0.0], + [0.0, np.pi / 4, np.pi / 2], + ], + dtype=np.float32, + ) + matrices_np = R.from_euler("xyz", euler).as_matrix().astype(np.float32) + poses_abs[1:, :3, :3] = matrices_np + + rel_6d = pose_abs_to_rel(poses_abs, rotation_format="rot6d", pose_convention="backward_anchored") + expected_rot6d = matrices_np[:, :, :2].transpose(0, 2, 1).reshape(2, 6) + np.testing.assert_allclose(rel_6d[:, 3:], expected_rot6d, atol=1e-6) + + rel_3d = pose_abs_to_rel(poses_abs, rotation_format="euler_xyz", pose_convention="backward_anchored") + expected_rot3d = R.from_matrix(matrices_np).as_euler("xyz", degrees=False) + np.testing.assert_allclose(rel_3d[:, 3:], expected_rot3d, atol=1e-6) + + +@pytest.mark.L0 +def test_convert_rotation_rot6d_to_matrix_uses_column_based_action_convention() -> None: + """rot6d roundtrip should preserve matrices under the centralized column-based convention.""" + euler = np.array( + [ + [0.0, 0.0, 0.0], + [0.0, np.pi / 4, np.pi / 2], + ], + dtype=np.float32, + ) + matrices_np = R.from_euler("xyz", euler).as_matrix().astype(np.float32) + rot6d = matrices_np[:, :, :2].transpose(0, 2, 1).reshape(2, 6) + + reconstructed = convert_rotation(rot6d, input_format="rot6d", output_format="matrix") + + np.testing.assert_allclose(reconstructed, matrices_np, atol=1e-6) + + +@pytest.mark.L0 +def test_normalize_rotation_matrices_batched_matches_reference_loop() -> None: + """Batched SVD normalization should match the previous per-matrix loop behavior.""" + rng = np.random.default_rng(42) + matrices = rng.normal(size=(32, 3, 3)).astype(np.float32) + + # New batched implementation. + actual = _normalize_rotation_matrices(matrices) + + # Reference: previous loop implementation. + expected_list: list[np.ndarray] = [] + for rot_mat in matrices: + U, _, Vt = np.linalg.svd(rot_mat) + normalized = U @ Vt + if np.linalg.det(normalized) < 0: + U[:, -1] *= -1 + normalized = U @ Vt + expected_list.append(normalized.astype(np.float32)) + expected = np.stack(expected_list, axis=0) + + np.testing.assert_allclose(actual, expected, atol=1e-6) + np.testing.assert_allclose(np.linalg.det(actual), np.ones(actual.shape[0], dtype=np.float32), atol=1e-5) + + +@pytest.mark.L0 +@pytest.mark.parametrize("rotation_format", ["rot9d", "rot6d", "quat_xyzw", "euler_xyz", "axisangle"]) +@pytest.mark.parametrize( + "pose_convention", + ["backward_anchored", "backward_framewise"], +) +def test_pose_abs_to_rel_roundtrips_through_pose_rel_to_abs( + rotation_format: str, + pose_convention: str, +) -> None: + """Relative pose encoding should invert back to the original absolute poses.""" + poses_abs = _make_example_poses_abs() + + poses_rel = pose_abs_to_rel( + poses_abs, + rotation_format=rotation_format, + pose_convention=pose_convention, + translation_scale=2.5, + ) + reconstructed = pose_rel_to_abs( + poses_rel, + rotation_format=rotation_format, + pose_convention=pose_convention, + initial_pose=poses_abs[0], + translation_scale=2.5, + ) + + np.testing.assert_allclose(reconstructed, poses_abs, atol=1e-5) diff --git a/cosmos_framework/data/vfm/action/pusht_dataset.py b/cosmos_framework/data/vfm/action/pusht_dataset.py new file mode 100644 index 0000000..04dc54f --- /dev/null +++ b/cosmos_framework/data/vfm/action/pusht_dataset.py @@ -0,0 +1,237 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Any + +import torch + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.cosmos3_action_lerobot import ( + BaseActionLeRobotDataset, +) +from cosmos_framework.data.vfm.action.viewpoint_utils import Viewpoint + + +def _overlay_cross_from_xy_0_512( + video_tchw: torch.Tensor, + xy: torch.Tensor, + *, + color_rgb: tuple[float, float, float] = (1.0, 0.0, 0.0), + radius: int = 4, + thickness: int = 1, + action_norm_range: float = 512.0, +) -> torch.Tensor: + """ + Overlay a colored 'X' cross on video frames at (x, y) positions from `xy`. + + Assumptions: + - `video_tchw` is (T, C, H, W) float in [0, 1] (or at least 3-channel RGB). + - `xy` is (T_xy, 2+) with raw coords in [0, 512]. + - We draw `xy[t]` directly onto `video_tchw[t]`. + """ + if video_tchw.dim() != 4: + raise ValueError(f"Expected video_tchw to be 4D (T,C,H,W), got {tuple(video_tchw.shape)}") + if video_tchw.shape[1] < 3: + # Not RGB; nothing sensible to do. + return video_tchw + if xy.numel() == 0: + return video_tchw + + T, _, H, W = video_tchw.shape + # scale raw [0, 512] to pixel indices [0, W-1]/[0, H-1] + sx = (W - 1) / action_norm_range + sy = (H - 1) / action_norm_range + + # Ensure we can index in python loop + xy_f = xy[:, :2].detach().to(dtype=torch.float32) + + # Draw in-place + t_max = min(int(xy_f.shape[0]), int(T)) + if t_max <= 0: + return video_tchw + + r_col, g_col, b_col = (float(color_rgb[0]), float(color_rgb[1]), float(color_rgb[2])) + + for t in range(t_max): + frame_t = t + xy_t = xy_f[t] + if not torch.isfinite(xy_t).all(): + continue + x = int(torch.round(xy_t[0] * sx).clamp(0, W - 1).item()) + y = int(torch.round(xy_t[1] * sy).clamp(0, H - 1).item()) + + # Draw an "X" (two diagonals) centered at (x, y) + r = int(radius) + th = int(thickness) + for d in range(-r, r + 1): + # Diagonal 1: (x+d, y+d) + xx1, yy1 = x + d, y + d + # Diagonal 2: (x+d, y-d) + xx2, yy2 = x + d, y - d + + for ox in range(-th, th + 1): + for oy in range(-th, th + 1): + # (x+d, y+d) + xxx1, yyy1 = xx1 + ox, yy1 + oy + if 0 <= xxx1 < W and 0 <= yyy1 < H: + video_tchw[frame_t, 0, yyy1, xxx1] = r_col + video_tchw[frame_t, 1, yyy1, xxx1] = g_col + video_tchw[frame_t, 2, yyy1, xxx1] = b_col + # (x+d, y-d) + xxx2, yyy2 = xx2 + ox, yy2 + oy + if 0 <= xxx2 < W and 0 <= yyy2 < H: + video_tchw[frame_t, 0, yyy2, xxx2] = r_col + video_tchw[frame_t, 1, yyy2, xxx2] = g_col + video_tchw[frame_t, 2, yyy2, xxx2] = b_col + + return video_tchw + + +class PushTDataset(BaseActionLeRobotDataset): + """PushT dataset with deferred source registration. + + Sources are registered by ``_register_sources()`` which is called by + ``ActionUnifiedIterableDataset.assign_worker()`` during training, or + explicitly for standalone/eval use. + """ + + def __init__( + self, + repo_id: str = "lerobot/pusht", + root: str | None = None, + chunk_length: int = 16, + fps: int = 10, + split: str = "train", + split_seed: int = 0, + split_val_ratio: float = 0.05, + mode: str = "policy", + tolerance_s: float = 1e-4, + embodiment_type: str = "pusht", + force_cache_sync: bool = False, + action_norm_range: float = 512.0, + action_space: str = "relative", + overlay_cross: bool = False, + overlay_cross_radius: int = 2, + overlay_cross_thickness: int = 0, + augment_prompt: bool = False, + viewpoint: Viewpoint = "third_person_view", + ) -> None: + super().__init__( + fps=fps, + chunk_length=chunk_length, + split_seed=split_seed, + split_val_ratio=split_val_ratio, + split=split, + mode=mode, + embodiment_type=embodiment_type, + viewpoint=viewpoint, + tolerance_s=tolerance_s, + ) + + self.action_norm_range = action_norm_range + self.overlay_cross = bool(overlay_cross) + self.overlay_cross_radius = int(overlay_cross_radius) + self.overlay_cross_thickness = int(overlay_cross_thickness) + self.action_space = action_space + self.augment_prompt = augment_prompt + + self._delta_timestamps = { + "observation.image": [i * self._dt for i in range(0, chunk_length + 1)], + "observation.state": [i * self._dt for i in range(0, chunk_length + 1)], + "action": [i * self._dt for i in range(0, chunk_length + 1)], + } + self._repo_id = repo_id + self._root = root + self._force_cache_sync = force_cache_sync + self._all_shard_roots = [root or repo_id] + + def _register_sources(self, indices: list[int] | None = None) -> None: + if indices is not None and 0 not in indices: + return + self._register_source( + repo_id=self._repo_id, + root=self._root, + force_cache_sync=self._force_cache_sync, + delta_timestamps=self._delta_timestamps, + tolerance_s=self._tolerance_s, + dataset_label=self._repo_id, + ) + + def __getitem__(self, idx: int) -> dict[str, Any]: + mode, _, base_idx, item = self._fetch_sample(idx) + + if self.augment_prompt: + prefix = "You are given a task to push the green T into the yellow T region." + prompt = f"Current prediction mode is {mode}." + post_fix = f"The video is {self._chunk_length / self._fps} seconds long and is of {self._fps} FPS." + prompt = f"{prefix} {prompt} {post_fix}" + else: + prompt = "PushT task" + + # Video: LeRobot returns float32 in [0, 1] with shape (T, C, H, W) + video_tchw: torch.Tensor = item["observation.image"] + + # Action (raw): typically absolute XY in [0, 512] for PushT. + action_raw: torch.Tensor = item["action"] + # State (raw): typically contains current agent state/position, also in [0, 512] for PushT. + # We use the -1/fps state (first element) as the "current" reference. + state_raw: torch.Tensor = item["observation.state"] + + # Optionally overlay the raw action XY on the video for debugging/visualization. + # We draw state/action[t] directly on frame[t] (all are indexed by the same delta timestamps). + if self.overlay_cross: + try: + # State in red + _overlay_cross_from_xy_0_512( + video_tchw, + state_raw, # (T+1, D) + color_rgb=(1.0, 0.0, 0.0), + radius=self.overlay_cross_radius, + thickness=self.overlay_cross_thickness, + action_norm_range=self.action_norm_range, + ) + # Action in purple + _overlay_cross_from_xy_0_512( + video_tchw, + action_raw, # (T+1, D) + color_rgb=(1.0, 0.0, 1.0), + radius=self.overlay_cross_radius, + thickness=self.overlay_cross_thickness, + action_norm_range=self.action_norm_range, + ) + except Exception as e: + log.warning(f"Failed to overlay action cross for idx={base_idx}: {e}") + + # Keep raw video in LeRobot layout; the base helper does the final conversion. + video = video_tchw # [T,C,H,W] + + # Action: (T+1, D) -> (T, D) + # Compute relative action w.r.t. the "current" state (delta=0). + # LeRobot returns state/action at deltas [0, 1, ..., chunk_length] (length T+1). + # We use state[0] as the reference and take actions[0:T] (dropping the last one). + # + # This matches: rel_action[t] = action[t] - state_current, for t in [0..T-1]. + action = action_raw[:-1] # [T,D_action] + state_current = state_raw[:1] # [1,D_state] + if self.action_space == "relative": + action = action - state_current + elif self.action_space == "absolute": + action = action + else: + raise ValueError(f"Unsupported action space: {self.action_space}") + # Normalize action to [-1, 1] + action = action / self.action_norm_range + if action.max() > 1.0 or action.min() < -1.0: + log.warning(f"Action out of range: {action.max()}, {action.min()}") + + key = torch.tensor([base_idx], dtype=torch.long) + + return self._build_result( + mode=mode, + video=video, + action=action, + ai_caption=prompt, + action_space=self.action_space, + state=state_raw, + __key__=key, + ) diff --git a/cosmos_framework/data/vfm/action/robomind_dataset_config.py b/cosmos_framework/data/vfm/action/robomind_dataset_config.py new file mode 100644 index 0000000..d791a41 --- /dev/null +++ b/cosmos_framework/data/vfm/action/robomind_dataset_config.py @@ -0,0 +1,336 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +FRANKA_LEROBOT_ROOTS = [ + "benchmark1_0_release/franka_3rgb/241021_close_trash_bin_1", + "benchmark1_0_release/franka_3rgb/241021_insert_marker_1", + "benchmark1_0_release/franka_3rgb/241021_open_trash_bin_1", + "benchmark1_0_release/franka_3rgb/241021_remove_marker_1", + "benchmark1_0_release/franka_3rgb/241022_lamp_off_1", + "benchmark1_0_release/franka_3rgb/241022_lamp_on_1", + "benchmark1_0_release/franka_3rgb/241022_side_pull_close_drawer_1", + "benchmark1_0_release/franka_3rgb/241022_side_pull_open_drawer_1", + "benchmark1_0_release/franka_3rgb/241023_pick_pear_from_bowl_1", + "benchmark1_0_release/franka_3rgb/241023_pick_pear_from_bowl_2", + "benchmark1_0_release/franka_3rgb/241023_place_pear_in_bowl_1", + "benchmark1_0_release/franka_3rgb/241023_place_pear_in_bowl_2", + "benchmark1_0_release/franka_3rgb/blue_cub_on_pink", + "benchmark1_0_release/franka_3rgb/cap_close_dustbin", + "benchmark1_0_release/franka_3rgb/close_cap_lid", + "benchmark1_0_release/franka_3rgb/close_cap_tool_box", + "benchmark1_0_release/franka_3rgb/close_cap_trash_can", + "benchmark1_0_release/franka_3rgb/close_cap_trash_can_2", + "benchmark1_0_release/franka_3rgb/close_drawer", + "benchmark1_0_release/franka_3rgb/close_trash", + "benchmark1_0_release/franka_3rgb/open_cap_lid", + "benchmark1_0_release/franka_3rgb/open_cap_tool_box", + "benchmark1_0_release/franka_3rgb/open_cap_trash_can_1", + "benchmark1_0_release/franka_3rgb/open_cap_trash_can_2", + "benchmark1_0_release/franka_3rgb/open_drawer", + "benchmark1_0_release/franka_3rgb/open_the_drawer", + "benchmark1_0_release/franka_3rgb/open_trash", + "benchmark1_0_release/franka_3rgb/pick_apple_into_chest", + "benchmark1_0_release/franka_3rgb/pick_bread_desk", + "benchmark1_0_release/franka_3rgb/pick_bread_into_plate", + "benchmark1_0_release/franka_3rgb/pick_drawer_tool", + "benchmark1_0_release/franka_3rgb/pick_plate_from_plate_rack", + "benchmark1_0_release/franka_3rgb/pick_up_strawberry_from_bowl", + "benchmark1_0_release/franka_3rgb/pick_up_strawberry_in_bowl", + "benchmark1_0_release/franka_3rgb/piled_on_stack_blue_block_on_pink_block", + "benchmark1_0_release/franka_3rgb/piled_on_yellow_block_on_purple_block", + "benchmark1_0_release/franka_3rgb/place_in_block_1", + "benchmark1_0_release/franka_3rgb/place_in_block_in_plate_1", + "benchmark1_0_release/franka_3rgb/place_in_block_on_table", + "benchmark1_0_release/franka_3rgb/place_in_block_tennis_ball", + "benchmark1_0_release/franka_3rgb/place_in_bread_in_basket", + "benchmark1_0_release/franka_3rgb/place_in_bread_in_basket_1", + "benchmark1_0_release/franka_3rgb/place_in_bread_in_bread_machine", + "benchmark1_0_release/franka_3rgb/place_in_bread_in_plate", + "benchmark1_0_release/franka_3rgb/place_in_bread_on_plate_1", + "benchmark1_0_release/franka_3rgb/place_in_bread_on_plate_2", + "benchmark1_0_release/franka_3rgb/place_in_bread_on_table", + "benchmark1_0_release/franka_3rgb/place_in_bread_on_table_1", + "benchmark1_0_release/franka_3rgb/place_in_bread_on_table_2", + "benchmark1_0_release/franka_3rgb/place_in_cylinder", + "benchmark1_0_release/franka_3rgb/place_in_fruit", + "benchmark1_0_release/franka_3rgb/place_in_fruit_bread", + "benchmark1_0_release/franka_3rgb/place_in_fruit_in_basket", + "benchmark1_0_release/franka_3rgb/place_in_fruit_in_fruit_basket", + "benchmark1_0_release/franka_3rgb/place_in_fruit_on_table", + "benchmark1_0_release/franka_3rgb/place_in_pick_up_and_throw_away_1", + "benchmark1_0_release/franka_3rgb/place_in_purple_block_in_plate", + "benchmark1_0_release/franka_3rgb/place_in_purple_block_on_table", + "benchmark1_0_release/franka_3rgb/place_in_rectangular_prism", + "benchmark1_0_release/franka_3rgb/place_in_shape", + "benchmark1_0_release/franka_3rgb/place_in_take_bread_and_put_in_plate", + "benchmark1_0_release/franka_3rgb/place_in_toy", + "benchmark1_0_release/franka_3rgb/place_in_trash", + "benchmark1_0_release/franka_3rgb/place_in_yellow_block_on_table", + "benchmark1_0_release/franka_3rgb/place_plate_in_plate_rack", + "benchmark1_0_release/franka_3rgb/place_trash", + "benchmark1_0_release/franka_3rgb/rotate_close_cabinet", + "benchmark1_0_release/franka_3rgb/rotate_open_cabinet", + "benchmark1_0_release/franka_3rgb/slide_close_cabinet", + "benchmark1_0_release/franka_3rgb/slide_close_drawer", + "benchmark1_0_release/franka_3rgb/slide_close_drawer_1", + "benchmark1_0_release/franka_3rgb/slide_close_drawer_1_1", + "benchmark1_0_release/franka_3rgb/slide_open_cabinet", + "benchmark1_0_release/franka_3rgb/slide_open_drawer", + "benchmark1_0_release/franka_3rgb/slide_open_drawer_1", + "benchmark1_0_release/franka_3rgb/stick_target_blue_on_the_pink_obejct", + "benchmark1_0_release/franka_3rgb/twist_knob_start_bread_machine", + "benchmark1_1_release/franka_3rgb/apples_placed_on_a_ceramic_plate", + "benchmark1_1_release/franka_3rgb/bananas_placed_on_a_ceramic_plate", + "benchmark1_1_release/franka_3rgb/bread_is_placed_on_a_ceramic_plate", + "benchmark1_1_release/franka_3rgb/chili_peppers_are_placed_on_a_ceramic_plate", + "benchmark1_1_release/franka_3rgb/chili_peppers_are_placed_on_the_right_side_of_a_plastic_tray", + "benchmark1_1_release/franka_3rgb/close_garbage_bin", + "benchmark1_1_release/franka_3rgb/close_lid", + "benchmark1_1_release/franka_3rgb/cucumber_placed_on_a_ceramic_plate", + "benchmark1_1_release/franka_3rgb/flip_marco_cup", + "benchmark1_1_release/franka_3rgb/mobile_marco_cup", + "benchmark1_1_release/franka_3rgb/open_lid", + "benchmark1_1_release/franka_3rgb/place_marker", + "benchmark1_1_release/franka_3rgb/put_potatoes_on_a_ceramic_plate", + "benchmark1_1_release/franka_3rgb/put_the_bread_on_the_table", + "benchmark1_1_release/franka_3rgb/put_the_cucumber_on_the_left_side_of_the_bowl", + "benchmark1_1_release/franka_3rgb/put_the_red_apple_in_the_bowl", + "benchmark1_1_release/franka_3rgb/put_the_steamed_buns_on_a_ceramic_plate", + "benchmark1_1_release/franka_3rgb/red_apple_placed_in_the_center_of_the_desktop", + "benchmark1_1_release/franka_3rgb/side_opening_drawer", + "benchmark1_1_release/franka_3rgb/side_pull_drawer", + "benchmark1_1_release/franka_3rgb/strawberries_on_a_ceramic_plate", + "benchmark1_1_release/franka_3rgb/take_marker_pen", + "benchmark1_1_release/franka_3rgb/turn_off_lamp", + "benchmark1_1_release/franka_3rgb/turn_on_desk_lamp", + "benchmark1_1_release/franka_3rgb/yellow_bananas_placed_on_a_plastic_tray", + "benchmark1_1_release/franka_3rgb/yellow_square_placed_on_ceramic_plate", + "benchmark1_2_release/franka_3rgb/241223_upright_cup", +] + +FRANKA_DUAL_LEROBOT_ROOTS = [ + "benchmark1_1_release/franka_fr3_dual/both_pour_water", + "benchmark1_1_release/franka_fr3_dual/left_place_gray_plate_from_low_rack_on_left_of_table", + "benchmark1_1_release/franka_fr3_dual/left_place_gray_plate_on_lower_rack", + "benchmark1_1_release/franka_fr3_dual/left_push_plate_to_right_of_table", + "benchmark1_1_release/franka_fr3_dual/right_slide_close_drawer", + "benchmark1_1_release/franka_fr3_dual/right_slide_open_drawer", +] + +UR_LEROBOT_ROOTS = [ + "benchmark1_0_release/ur_1rgb/bread_in_basket_1", + "benchmark1_0_release/ur_1rgb/bread_in_basket_old", + "benchmark1_0_release/ur_1rgb/bread_on_table", + "benchmark1_0_release/ur_1rgb/close_top_drawer", + "benchmark1_0_release/ur_1rgb/close_top_white_drawer", + "benchmark1_0_release/ur_1rgb/close_trash_can", + "benchmark1_0_release/ur_1rgb/cover_pot_lid", + "benchmark1_0_release/ur_1rgb/green_pepper_in_basket", + "benchmark1_0_release/ur_1rgb/green_pepper_in_basket_1", + "benchmark1_0_release/ur_1rgb/green_pepper_on_table", + "benchmark1_0_release/ur_1rgb/open_pot_lid", + "benchmark1_0_release/ur_1rgb/open_top_drawer", + "benchmark1_0_release/ur_1rgb/open_top_white_drawer", + "benchmark1_0_release/ur_1rgb/open_trash_can", + "benchmark1_0_release/ur_1rgb/pick_up_banana", + "benchmark1_0_release/ur_1rgb/pick_up_bread", + "benchmark1_0_release/ur_1rgb/pick_up_bread_slice", + "benchmark1_0_release/ur_1rgb/pick_up_can", + "benchmark1_0_release/ur_1rgb/pick_up_donut", + "benchmark1_0_release/ur_1rgb/pick_up_egg", + "benchmark1_0_release/ur_1rgb/pick_up_green_onion", + "benchmark1_0_release/ur_1rgb/pick_up_green_pepper", + "benchmark1_0_release/ur_1rgb/pick_up_long_bread", + "benchmark1_0_release/ur_1rgb/pick_up_mangosteen", + "benchmark1_0_release/ur_1rgb/pick_up_paper_ball", + "benchmark1_0_release/ur_1rgb/pick_up_pear", + "benchmark1_0_release/ur_1rgb/pick_up_plastic_bottle", + "benchmark1_0_release/ur_1rgb/pick_up_pot_lid", + "benchmark1_0_release/ur_1rgb/pick_up_red_pepper", + "benchmark1_0_release/ur_1rgb/pick_up_round_bread", + "benchmark1_0_release/ur_1rgb/pick_up_round_bread_1", + "benchmark1_0_release/ur_1rgb/pick_up_square_bread", + "benchmark1_0_release/ur_1rgb/pick_up_toast", + "benchmark1_0_release/ur_1rgb/pick_up_triangle_bread", + "benchmark1_0_release/ur_1rgb/pick_up_yellow_pepper", + "benchmark1_0_release/ur_1rgb/put_banana_in_top_drawer", + "benchmark1_0_release/ur_1rgb/put_bread_in_pot", + "benchmark1_0_release/ur_1rgb/put_bread_slice_in_pot", + "benchmark1_0_release/ur_1rgb/put_bread_slice_in_top_white_drawer", + "benchmark1_0_release/ur_1rgb/put_can_in_trash_can", + "benchmark1_0_release/ur_1rgb/put_donut_in_pot", + "benchmark1_0_release/ur_1rgb/put_donut_in_top_white_drawer", + "benchmark1_0_release/ur_1rgb/put_egg_in_pot", + "benchmark1_0_release/ur_1rgb/put_egg_in_top_white_drawer", + "benchmark1_0_release/ur_1rgb/put_green_onion_in_pot", + "benchmark1_0_release/ur_1rgb/put_green_pepper_in_pot", + "benchmark1_0_release/ur_1rgb/put_green_pepper_in_top_white_drawer", + "benchmark1_0_release/ur_1rgb/put_long_bread_in_pot", + "benchmark1_0_release/ur_1rgb/put_long_bread_in_top_white_drawer", + "benchmark1_0_release/ur_1rgb/put_long_bread_in_trash_can", + "benchmark1_0_release/ur_1rgb/put_mangosteen_in_top_white_drawer", + "benchmark1_0_release/ur_1rgb/put_paper_ball_in_trash_can", + "benchmark1_0_release/ur_1rgb/put_pear_in_top_white_drawer", + "benchmark1_0_release/ur_1rgb/put_plastic_bottle_in_trash_can", + "benchmark1_0_release/ur_1rgb/put_pot_lid_on_table", + "benchmark1_0_release/ur_1rgb/put_red_pepper_in_pot", + "benchmark1_0_release/ur_1rgb/put_red_pepper_in_top_white_drawer", + "benchmark1_0_release/ur_1rgb/put_round_bread_in_pot", + "benchmark1_0_release/ur_1rgb/put_round_bread_in_top_white_drawer", + "benchmark1_0_release/ur_1rgb/put_round_bread_in_trash_can", + "benchmark1_0_release/ur_1rgb/put_square_bread_in_top_white_drawer", + "benchmark1_0_release/ur_1rgb/put_square_bread_in_trash_can", + "benchmark1_0_release/ur_1rgb/put_toast_in_pot", + "benchmark1_0_release/ur_1rgb/put_toast_in_top_white_drawer", + "benchmark1_0_release/ur_1rgb/put_triangle_bread_in_pot", + "benchmark1_0_release/ur_1rgb/put_triangle_bread_in_top_white_drawer", + "benchmark1_0_release/ur_1rgb/put_yellow_pepper_in_pot", + "benchmark1_0_release/ur_1rgb/put_yellow_pepper_in_top_drawer", + "benchmark1_0_release/ur_1rgb/put_yellow_pepper_in_top_white_drawer", + "benchmark1_0_release/ur_1rgb/red_pepper_in_basket", + "benchmark1_0_release/ur_1rgb/red_pepper_on_table", + "benchmark1_0_release/ur_1rgb/triangle_bread_in_basket", + "benchmark1_0_release/ur_1rgb/triangle_bread_in_basket_1", + "benchmark1_0_release/ur_1rgb/triangle_bread_on_table", + "benchmark1_0_release/ur_1rgb/yellow_pepper_in_basket", + "benchmark1_0_release/ur_1rgb/yellow_pepper_in_basket_1", + "benchmark1_0_release/ur_1rgb/yellow_pepper_on_table", + "benchmark1_1_release/ur_1rgb/green_pepper_on_plate", + "benchmark1_1_release/ur_1rgb/green_pepper_on_the_table", + "benchmark1_1_release/ur_1rgb/insert_the_flowers_from_the_vase_1025", + "benchmark1_1_release/ur_1rgb/insert_the_flowers_from_the_vase_1028", + "benchmark1_1_release/ur_1rgb/insert_the_flowers_from_the_vase_1029", + "benchmark1_1_release/ur_1rgb/insert_the_flowers_from_the_vase_1030", + "benchmark1_1_release/ur_1rgb/insert_the_flowers_from_the_vase_1031", + "benchmark1_1_release/ur_1rgb/insert_the_flowers_from_the_vase_1101", + "benchmark1_1_release/ur_1rgb/insert_the_flowers_from_the_vase_1126", + "benchmark1_1_release/ur_1rgb/pick_up_green_pepper_from_plate", + "benchmark1_1_release/ur_1rgb/pick_up_green_pepper_from_table", + "benchmark1_1_release/ur_1rgb/pick_up_red_pepper_from_plate", + "benchmark1_1_release/ur_1rgb/pick_up_red_pepper_from_table", + "benchmark1_1_release/ur_1rgb/pick_up_the_cucumber_from_the_basket_1125", + "benchmark1_1_release/ur_1rgb/pick_up_the_cucumber_from_the_table_1125", + "benchmark1_1_release/ur_1rgb/pick_up_the_flowers_from_the_table_1025", + "benchmark1_1_release/ur_1rgb/pick_up_the_flowers_from_the_table_1028", + "benchmark1_1_release/ur_1rgb/pick_up_the_flowers_from_the_table_1029", + "benchmark1_1_release/ur_1rgb/pick_up_the_flowers_from_the_table_1030", + "benchmark1_1_release/ur_1rgb/pick_up_the_flowers_from_the_table_1031", + "benchmark1_1_release/ur_1rgb/pick_up_the_flowers_from_the_table_1101", + "benchmark1_1_release/ur_1rgb/pick_up_the_flowers_from_the_table_1126", + "benchmark1_1_release/ur_1rgb/pick_up_the_flowers_from_the_vase_1025", + "benchmark1_1_release/ur_1rgb/pick_up_the_flowers_from_the_vase_1028", + "benchmark1_1_release/ur_1rgb/pick_up_the_flowers_from_the_vase_1029", + "benchmark1_1_release/ur_1rgb/pick_up_the_flowers_from_the_vase_1030", + "benchmark1_1_release/ur_1rgb/pick_up_the_flowers_from_the_vase_1031", + "benchmark1_1_release/ur_1rgb/pick_up_the_flowers_from_the_vase_1101", + "benchmark1_1_release/ur_1rgb/pick_up_the_flowers_from_the_vase_1126", + "benchmark1_1_release/ur_1rgb/pick_up_the_green_vegetable_from_the_basket_1121", + "benchmark1_1_release/ur_1rgb/pick_up_the_green_vegetable_from_the_table_1121", + "benchmark1_1_release/ur_1rgb/pick_up_the_mango_from_the_table_1104", + "benchmark1_1_release/ur_1rgb/pick_up_the_mango_from_the_table_1120", + "benchmark1_1_release/ur_1rgb/pick_up_the_mangoes_from_the_basket_1104", + "benchmark1_1_release/ur_1rgb/pick_up_the_mangoes_from_the_basket_1120", + "benchmark1_1_release/ur_1rgb/pick_up_the_oranges_from_the_basket_1119", + "benchmark1_1_release/ur_1rgb/pick_up_the_oranges_from_the_table_1119", + "benchmark1_1_release/ur_1rgb/pick_up_the_sweet_potato_from_the_basket_1122", + "benchmark1_1_release/ur_1rgb/pick_up_the_sweet_potato_from_the_table_1122", + "benchmark1_1_release/ur_1rgb/pick_up_the_yellow_peppers_from_the_basket_1101", + "benchmark1_1_release/ur_1rgb/pick_up_the_yellow_peppers_from_the_table_1101", + "benchmark1_1_release/ur_1rgb/pick_up_yellow_pepper_from_plate", + "benchmark1_1_release/ur_1rgb/pick_up_yellow_pepper_from_plate_copy_1734079773826", + "benchmark1_1_release/ur_1rgb/pick_up_yellow_pepper_from_table", + "benchmark1_1_release/ur_1rgb/pick_up_yellow_pepper_from_table_copy_1734079574938", + "benchmark1_1_release/ur_1rgb/pickupthebananafromtheplate", + "benchmark1_1_release/ur_1rgb/pickupthebananafromthetable", + "benchmark1_1_release/ur_1rgb/place_the_cucumber_on_the_table_1125", + "benchmark1_1_release/ur_1rgb/place_the_flowers_on_the_table_1025", + "benchmark1_1_release/ur_1rgb/place_the_flowers_on_the_table_1028", + "benchmark1_1_release/ur_1rgb/place_the_flowers_on_the_table_1029", + "benchmark1_1_release/ur_1rgb/place_the_flowers_on_the_table_1030", + "benchmark1_1_release/ur_1rgb/place_the_flowers_on_the_table_1031", + "benchmark1_1_release/ur_1rgb/place_the_flowers_on_the_table_1101", + "benchmark1_1_release/ur_1rgb/place_the_flowers_on_the_table_1126", + "benchmark1_1_release/ur_1rgb/place_the_green_vegetable_on_the_table_1121", + "benchmark1_1_release/ur_1rgb/place_the_mango_on_the_table_1104", + "benchmark1_1_release/ur_1rgb/place_the_mango_on_the_table_1120", + "benchmark1_1_release/ur_1rgb/place_the_oranges_on_the_table_1119", + "benchmark1_1_release/ur_1rgb/place_the_sweet_potato_on_the_table_1122", + "benchmark1_1_release/ur_1rgb/placebananaonaplate", + "benchmark1_1_release/ur_1rgb/put_the_cucumber_in_the_basket_1125", + "benchmark1_1_release/ur_1rgb/put_the_garbage_in_the_trash_can_1105", + "benchmark1_1_release/ur_1rgb/put_the_garbage_in_the_trash_can_1106", + "benchmark1_1_release/ur_1rgb/put_the_garbage_in_the_trash_can_1107", + "benchmark1_1_release/ur_1rgb/put_the_garbage_in_the_trash_can_1111", + "benchmark1_1_release/ur_1rgb/put_the_garbage_in_the_trash_can_1112", + "benchmark1_1_release/ur_1rgb/put_the_garbage_in_the_trash_can_1113", + "benchmark1_1_release/ur_1rgb/put_the_garbage_in_the_trash_can_1114", + "benchmark1_1_release/ur_1rgb/put_the_garbage_in_the_trash_can_1118", + "benchmark1_1_release/ur_1rgb/put_the_green_vegetable_in_the_basket_1121", + "benchmark1_1_release/ur_1rgb/put_the_mango_in_the_basket_1104", + "benchmark1_1_release/ur_1rgb/put_the_mango_in_the_basket_1120", + "benchmark1_1_release/ur_1rgb/put_the_oranges_in_the_basket_1119", + "benchmark1_1_release/ur_1rgb/put_the_sweet_potato_in_the_basket_1122", + "benchmark1_1_release/ur_1rgb/putthebananaonthetable", + "benchmark1_1_release/ur_1rgb/red_pepper_in_table", + "benchmark1_1_release/ur_1rgb/red_pepper_on_plate", + "benchmark1_1_release/ur_1rgb/uncap_open_the_trash_can_1104", + "benchmark1_1_release/ur_1rgb/uncap_open_the_trash_can_1105", + "benchmark1_1_release/ur_1rgb/uncap_open_the_trash_can_1106", + "benchmark1_1_release/ur_1rgb/uncap_open_the_trash_can_1107", + "benchmark1_1_release/ur_1rgb/uncap_open_the_trash_can_1108", + "benchmark1_1_release/ur_1rgb/uncap_open_the_trash_can_1111", + "benchmark1_1_release/ur_1rgb/uncap_open_the_trash_can_1112", + "benchmark1_1_release/ur_1rgb/uncap_open_the_trash_can_1113", + "benchmark1_1_release/ur_1rgb/uncap_open_the_trash_can_1114", + "benchmark1_1_release/ur_1rgb/uncap_open_the_trash_can_1118", + "benchmark1_1_release/ur_1rgb/yellow_pepper_on_plate", + "benchmark1_1_release/ur_1rgb/yellow_pepper_on_plate_copy_1734079761061", +] + +LEROBOT_ROOTS = { + "franka": FRANKA_LEROBOT_ROOTS, + "franka-dual": FRANKA_DUAL_LEROBOT_ROOTS, + "ur": UR_LEROBOT_ROOTS, +} + +FRANKA_OBSERVATION_FEATURES = ["observation.images.camera_top", "observation.states.end_effector"] + +FRANKA_DUAL_OBSERVATION_FEATURES = ["observation.images.camera_front", "observation.states.end_effector"] + +UR_OBSERVATION_FEATURES = ["observation.images.camera_top"] + +OBSERVATION_FEATURES = { + "franka": FRANKA_OBSERVATION_FEATURES, + "franka-dual": FRANKA_DUAL_OBSERVATION_FEATURES, + "ur": UR_OBSERVATION_FEATURES, +} + +# All available camera keys per embodiment (used by concat_view mode). +FRANKA_ALL_CAMERA_KEYS = [ + "observation.images.camera_top", + "observation.images.camera_left", + "observation.images.camera_right", +] + +FRANKA_DUAL_ALL_CAMERA_KEYS = [ + "observation.images.camera_front", + "observation.images.camera_left", + "observation.images.camera_right", +] + +ALL_CAMERA_KEYS = { + "franka": FRANKA_ALL_CAMERA_KEYS, + "franka-dual": FRANKA_DUAL_ALL_CAMERA_KEYS, +} + +FRANKA_ACTION_FEATURES = ["actions.joint_position"] + +FRANKA_DUAL_ACTION_FEATURES = ["actions.joint_position"] + +UR_ACTION_FEATURES = ["actions.joint_position"] + +ACTION_FEATURES = { + "franka": FRANKA_ACTION_FEATURES, + "franka-dual": FRANKA_DUAL_ACTION_FEATURES, + "ur": UR_ACTION_FEATURES, +} diff --git a/cosmos_framework/data/vfm/action/robomind_franka_dataset.py b/cosmos_framework/data/vfm/action/robomind_franka_dataset.py new file mode 100644 index 0000000..21ecb0b --- /dev/null +++ b/cosmos_framework/data/vfm/action/robomind_franka_dataset.py @@ -0,0 +1,278 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""RoboMIND Franka datasets for single-arm and dual-arm embodiments.""" + +from __future__ import annotations + +import math +import os +from typing import Any, cast + +import numpy as np +import torch +import torch.nn.functional as F + +from cosmos_framework.data.vfm.action.cosmos3_action_lerobot import ( + ActionNormalization, + ActionSpec, + BaseActionLeRobotDataset, + Gripper, + Pos, + Rot, + build_action_spec, +) +from cosmos_framework.data.vfm.action.pose_utils import ( + PoseConvention, + build_abs_pose_from_components, + pose_abs_to_rel, +) +from cosmos_framework.data.vfm.action.robomind_dataset_config import ( + ACTION_FEATURES, + ALL_CAMERA_KEYS, + LEROBOT_ROOTS, + OBSERVATION_FEATURES, +) +from cosmos_framework.data.vfm.action.viewpoint_utils import Viewpoint + +_ROBOMIND_FRANKA_TO_OPENCV: np.ndarray = np.array( + [ + [0.0, -1.0, 0.0, 0.0], + [1.0, 0.0, 0.0, 0.0], + [0.0, 0.0, 1.0, 0.0], + [0.0, 0.0, 0.0, 1.0], + ], + dtype=np.float32, +) + +FrankaInitialPose = torch.Tensor | tuple[torch.Tensor, torch.Tensor] + + +class RoboMINDFrankaDataset(BaseActionLeRobotDataset): + """RoboMIND dataset for Franka single-arm and dual-arm embodiments.""" + + SUPPORTED_EMBODIMENTS: tuple[str, str] = ("robomind-franka", "robomind-franka-dual") + + # RoboMIND-Franka has ~3x faster motion than the typical teleoperation + # datasets (bridge / DROID / fractal). Empirically (see + # ``debug/idle_test/recommend_thresholds_norm.txt``) the slowest 1 % of + # motion sits at ~22 mm/s for single-arm and ~15 mm/s for dual-arm. + # + # **Dual-arm caveat**: the dual-arm dataset frequently has one arm's + # state recorded as a near-zero stutter throughout a chunk (data quality + # issue — only one arm is actually being teleoperated). Because the POS + # branch uses the combined L2 across both arms, the threshold then + # effectively becomes a per-arm threshold for whichever arm is active. + # We compensate by tightening dual-arm to the global default (5 mm/s, + # 1.5°/s) so a single arm doing a slow approach (~1mm/f at 10 Hz) is no + # longer classified as idle. + # + # Class defaults below match single-arm. Dual-arm overrides at instance + # construction (see ``__init__``). + _IDLE_EPS_T_SINGLE: float = 22e-3 + _IDLE_EPS_R_SINGLE: float = math.radians(3.0) + _IDLE_EPS_T_DUAL: float = 5e-3 # = base default; tight enough + _IDLE_EPS_R_DUAL: float = math.radians(1.5) # for "single-arm-slow" cases + idle_eps_t_per_sec: float = _IDLE_EPS_T_SINGLE + idle_eps_r_per_sec: float = _IDLE_EPS_R_SINGLE + + def __init__( + self, + root: str = "", + fps: float = 10.0, + chunk_length: int = 16, + split_seed: int = 42, + split_val_ratio: float = 0.05, + split: str = "train", + mode: str = "policy", + embodiment_type: str = "robomind-franka", + pose_convention: str = "backward_framewise", + action_normalization: ActionNormalization | None = None, + viewpoint: Viewpoint = "concat_view", + enable_fast_init: bool = False, + ) -> None: + if embodiment_type not in self.SUPPORTED_EMBODIMENTS: + raise ValueError( + f"RoboMINDFrankaDataset only supports {self.SUPPORTED_EMBODIMENTS}, " + f"got embodiment_type={embodiment_type!r}" + ) + + super().__init__( + fps=fps, + chunk_length=chunk_length, + split_seed=split_seed, + split_val_ratio=split_val_ratio, + split=split, + mode=mode, + embodiment_type=embodiment_type, + viewpoint=viewpoint, + pose_convention=pose_convention, + rotation_format="rot6d", + action_normalization=action_normalization, + tolerance_s=1e-4, + enable_fast_init=enable_fast_init, + ) + + self._to_opencv: np.ndarray = _ROBOMIND_FRANKA_TO_OPENCV[:3, :3] + self._is_concat_view: bool = viewpoint == "concat_view" + + # Per-embodiment idle thresholds (instance-level override of the + # class default which matches single-arm). Dual-arm tightens both + # eps_t and eps_r to reflect its smaller per-frame motion tail. + if embodiment_type == "robomind-franka-dual": + self.idle_eps_t_per_sec = self._IDLE_EPS_T_DUAL + self.idle_eps_r_per_sec = self._IDLE_EPS_R_DUAL + + embodiment_key = embodiment_type.removeprefix("robomind-") + lerobot_roots = LEROBOT_ROOTS[embodiment_key] + observation_features = list(OBSERVATION_FEATURES[embodiment_key]) + action_features = ACTION_FEATURES[embodiment_key] + + if self._is_concat_view and embodiment_key in ALL_CAMERA_KEYS: + for cam_key in ALL_CAMERA_KEYS[embodiment_key]: + if cam_key not in observation_features: + observation_features.append(cam_key) + + self._all_shard_roots = [os.path.join(root, shard_root) for shard_root in lerobot_roots] + self._delta_timestamps = { + **{key: [i * self._dt for i in range(0, self._chunk_length + 1)] for key in observation_features}, + **{key: [i * self._dt for i in range(0, self._chunk_length)] for key in action_features}, + } + + def _build_relative_poses( + self, + positions: torch.Tensor | np.ndarray, + euler_xyz: torch.Tensor | np.ndarray, + ) -> tuple[np.ndarray, torch.Tensor]: + poses_abs = build_abs_pose_from_components(positions, euler_xyz, "euler_xyz") + poses_abs[:, :3, :3] = poses_abs[:, :3, :3] @ self._to_opencv + initial_pose = torch.from_numpy(poses_abs[0].copy()).float() + pose_convention = cast(PoseConvention, self._pose_convention) + poses_rel = cast( + np.ndarray, pose_abs_to_rel(poses_abs, rotation_format="rot6d", pose_convention=pose_convention) + ) + return poses_rel, initial_pose + + def _build_action(self, sample: dict[str, Any]) -> tuple[torch.Tensor, FrankaInitialPose]: + state = sample["observation.states.end_effector"] + gripper = sample["actions.joint_position"] + + if self._embodiment_type == "robomind-franka": + poses_rel, initial_pose = self._build_relative_poses(state[:, 0:3], state[:, 3:6]) + action = torch.cat( + [ + torch.from_numpy(poses_rel).float(), + 1.0 - gripper[:, [7]], + ], + dim=-1, + ) # [T, 10] + return action, initial_pose + + poses_rel_left, initial_pose_left = self._build_relative_poses(state[:, 0:3], state[:, 3:6]) + poses_rel_right, initial_pose_right = self._build_relative_poses(state[:, 6:9], state[:, 9:12]) + action = torch.cat( + [ + torch.from_numpy(poses_rel_left).float(), + 1.0 - gripper[:, [7]], + torch.from_numpy(poses_rel_right).float(), + 1.0 - gripper[:, [15]], + ], + dim=-1, + ) # [T, 20] + return action, (initial_pose_left, initial_pose_right) + + def _compose_multi_view_franka(self, sample: dict[str, Any]) -> torch.Tensor: # returns [T,C,H',W'] + top_or_front_key = ( + "observation.images.camera_top" + if self._embodiment_type == "robomind-franka" + else "observation.images.camera_front" + ) + top_or_front = sample[top_or_front_key] # [T,C,H,W] + left = sample["observation.images.camera_left"] # [T,C,H,W] + right = sample["observation.images.camera_right"] # [T,C,H,W] + + _, _, height_ref, width_ref = top_or_front.shape + half_height, half_width = height_ref // 2, width_ref // 2 + + left = F.interpolate( + left, size=(half_height, half_width), mode="bilinear", align_corners=False + ) # [T,C,H/2,W/2] + right = F.interpolate( + right, size=(half_height, half_width), mode="bilinear", align_corners=False + ) # [T,C,H/2,W/2] + bottom = torch.cat([left, right], dim=-1) # [T,C,H/2,W] + + composite = torch.cat([top_or_front, bottom], dim=-2) # [T,C,3H/2,W] + return composite # [T,C,3H/2,W] + + def _build_action_spec(self) -> ActionSpec: + """RoboMIND Franka: 10D single-arm or 20D dual-arm. + + Single (``robomind-franka``): + ``[Pos, Rot6d, Gripper]`` (10D) + + Dual (``robomind-franka-dual``): + ``[L_Pos, L_Rot6d, L_Gripper, R_Pos, R_Rot6d, R_Gripper]`` (20D) + """ + if self._embodiment_type == "robomind-franka": + return build_action_spec(Pos(), Rot("rot6d"), Gripper()) + # dual arm + return build_action_spec( + Pos(prefix="left"), + Rot("rot6d", prefix="left"), + Gripper(prefix="left"), + Pos(prefix="right"), + Rot("rot6d", prefix="right"), + Gripper(prefix="right"), + ) + + def __getitem__(self, idx: int) -> dict[str, Any]: + mode, _, _, sample = self._fetch_sample(idx) + ai_caption = sample["task"] + + if self._skip_video_loading: + video = None + additional_view_description = None + elif self._is_concat_view: + video = self._compose_multi_view_franka(sample) + additional_view_description = ( + "The top row shows third-person perspective view looking towards the robot from the front. " + "The bottom-left video shows the third-person perspective view looking at the scene from the left side. " + "The bottom-right video shows the third-person perspective view looking at the scene from the right side." + ) + elif self._embodiment_type == "robomind-franka": + video = sample["observation.images.camera_top"] # [T,C,H,W] + additional_view_description = None + elif self._embodiment_type == "robomind-franka-dual": + video = sample["observation.images.camera_front"] # [T,C,H,W] + additional_view_description = None + else: + raise ValueError(f"Unknown embodiment: {self._embodiment_type}") + + action, initial_pose = self._build_action(sample) + + extras: dict[str, Any] = {} + if isinstance(initial_pose, tuple): + extras["initial_pose"] = initial_pose[0] + extras["initial_pose_right"] = initial_pose[1] + else: + extras["initial_pose"] = initial_pose + + if additional_view_description is not None: + extras["additional_view_description"] = additional_view_description + + return self._build_result( + mode=mode, + video=video, + action=action, + ai_caption=ai_caption, + **extras, + ) + + @property + def action_dim(self) -> int: + if self._embodiment_type == "robomind-franka": + return 10 + if self._embodiment_type == "robomind-franka-dual": + return 20 + raise ValueError(f"Unknown embodiment: {self._embodiment_type}") diff --git a/cosmos_framework/data/vfm/action/robomind_ur_dataset.py b/cosmos_framework/data/vfm/action/robomind_ur_dataset.py new file mode 100644 index 0000000..b9b52ba --- /dev/null +++ b/cosmos_framework/data/vfm/action/robomind_ur_dataset.py @@ -0,0 +1,253 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""RoboMIND UR dataset for single-arm UR5e embodiment.""" + +from __future__ import annotations + +import os +from typing import Any, cast + +import numpy as np +import torch + +from cosmos_framework.data.vfm.action.cosmos3_action_lerobot import ( + ActionNormalization, + ActionSpec, + BaseActionLeRobotDataset, + Gripper, + Pos, + Rot, + build_action_spec, +) +from cosmos_framework.data.vfm.action.pose_utils import ( + PoseConvention, + pose_abs_to_rel, +) +from cosmos_framework.data.vfm.action.robomind_dataset_config import ( + ACTION_FEATURES, + LEROBOT_ROOTS, + OBSERVATION_FEATURES, +) +from cosmos_framework.data.vfm.action.viewpoint_utils import Viewpoint + +# UR EE frame → OpenCV convention rotation (3×3, post-multiplied). +# Identity: attachment_site (quat="-1 1 0 0" in ur5e_robotiq_2f85.xml) already +# satisfies OpenCV convention (z = approach) +_ROBOMIND_UR_TO_OPENCV: np.ndarray = np.eye(3, dtype=np.float32) + +_UR5E_ARM_JOINTS = 6 # shoulder_pan … wrist_3 +_UR5E_EE_SITE = "attachment_site" # flange site in ur5e_robotiq_2f85.xml + + +class RoboMINDURDataset(BaseActionLeRobotDataset): + """RoboMIND dataset for UR embodiment. + + Franka variants live in ``robomind_franka_dataset.py``. + + Action format: 10D ``[pos_delta(3) | rot6d_delta(6) | gripper(1)]`` + derived from MuJoCo FK of ``actions.joint_position`` (6 arm joints → + ``attachment_site`` SE(3) pose). ``observation.states.end_effector`` is + NOT used because it is recorded incorrectly (constant) in ~89 % of UR + episodes; ``actions.joint_position`` is valid for 100 % of episodes. + + The sample also contains ``joint_configs`` — absolute joint angles + ``(T, 7)`` from ``actions.joint_position[1:T+1]`` — for FK-based robot + mesh animation in the viewer. + """ + + SUPPORTED_EMBODIMENTS: tuple[str] = ("robomind-ur",) + + def __init__( + self, + root: str = "", + fps: float = 10.0, + chunk_length: int = 16, + split_seed: int = 42, + split_val_ratio: float = 0.05, + split: str = "train", + mode: str = "policy", + embodiment_type: str = "robomind-ur", + pose_convention: str = "backward_framewise", + action_normalization: ActionNormalization | None = None, + viewpoint: Viewpoint = "third_person_view", + enable_fast_init: bool = False, + ) -> None: + if embodiment_type not in self.SUPPORTED_EMBODIMENTS: + raise ValueError( + f"RoboMINDURDataset only supports {self.SUPPORTED_EMBODIMENTS}; " + "use RoboMINDFrankaDataset for Franka variants. " + f"Got embodiment_type={embodiment_type!r}." + ) + + super().__init__( + fps=fps, + chunk_length=chunk_length, + split_seed=split_seed, + split_val_ratio=split_val_ratio, + split=split, + mode=mode, + embodiment_type=embodiment_type, + viewpoint=viewpoint, + pose_convention=pose_convention, + rotation_format="rot6d", + action_normalization=action_normalization, + tolerance_s=1e-4, + enable_fast_init=enable_fast_init, + ) + + self._to_opencv: np.ndarray = _ROBOMIND_UR_TO_OPENCV + + embodiment_key = embodiment_type.removeprefix("robomind-") + lerobot_roots = LEROBOT_ROOTS[embodiment_key] + observation_features = list(OBSERVATION_FEATURES[embodiment_key]) + action_features = ACTION_FEATURES[embodiment_key] + + self._all_shard_roots = [os.path.join(root, x) for x in lerobot_roots] + + # T+1 joint positions: frame 0 is the initial state; frames 1..T are + # the targets after each action step (used for both FK EE poses and mesh). + _extended = frozenset({"actions.joint_position"}) + self._delta_timestamps = { + **{k: [i * self._dt for i in range(0, self._chunk_length + 1)] for k in observation_features}, + **{ + k: [i * self._dt for i in range(0, self._chunk_length + (1 if k in _extended else 0))] + for k in action_features + }, + } + + # MuJoCo model for FK — loaded once per dataset instance. + # We derive EE poses from FK on actions.joint_position rather than + # observation.states.end_effector because the latter is recorded + # incorrectly (frozen constant) in ~89 % of UR episodes across all 77 + # task shards (6,474 / 7,251 episodes). actions.joint_position is + # valid for 100 % of episodes and is the only reliable EE source. + self._mj_model, self._mj_data, self._ee_site_id = self._init_mujoco() + + @staticmethod + def _init_mujoco(): + """Load UR5e+Robotiq MuJoCo model (kinematics-only) and locate the EE site. + + Strips all geoms and mesh/texture/material assets from the MJCF via + ``MjSpec`` before compile, so the model loads without any mesh files + on disk. FK only needs the kinematic tree (bodies, joints, sites, + inertials), so ``mj_forward`` + ``site_xpos``/``site_xmat`` still + produce identical EE poses. Uses the committed XML directly to skip + the Menagerie mesh download in ``get_mjcf_path``. + """ + from pathlib import Path + + import mujoco + + mjcf_path = str(Path(__file__).parent / "urdf_visualizer" / "ur5e_robotiq_2f85.xml") + spec = mujoco.MjSpec.from_file(mjcf_path) + + # Drop all geoms recursively — FK never touches them. + def _strip_geoms(body): + for g in list(body.geoms): + spec.delete(g) + for child in body.bodies: + _strip_geoms(child) + + _strip_geoms(spec.worldbody) + + # Drop asset entries that reference external files. + for m in list(spec.meshes): + spec.delete(m) + for t in list(spec.textures): + spec.delete(t) + for mat in list(spec.materials): + spec.delete(mat) + + mj_model = spec.compile() + mj_data = mujoco.MjData(mj_model) + ee_site_id = mujoco.mj_name2id(mj_model, mujoco.mjtObj.mjOBJ_SITE, _UR5E_EE_SITE) + if ee_site_id < 0: + raise RuntimeError(f"EE site '{_UR5E_EE_SITE}' not found in {mjcf_path}") + return mj_model, mj_data, ee_site_id + + def _fk_ee_poses(self, arm_q: np.ndarray) -> tuple[np.ndarray, np.ndarray]: + """Run MuJoCo FK for T+1 arm configs → EE site positions and rotations. + + Args: + arm_q: ``(T+1, 6)`` arm joint angles in radians. + + Returns: + ``(positions (T+1, 3), rotations (T+1, 3, 3))`` in MuJoCo world frame. + """ + import mujoco + + T1 = len(arm_q) + positions = np.empty((T1, 3), dtype=np.float32) + rotations = np.empty((T1, 3, 3), dtype=np.float32) + for t in range(T1): + self._mj_data.qpos[:_UR5E_ARM_JOINTS] = arm_q[t] + mujoco.mj_forward(self._mj_model, self._mj_data) + positions[t] = self._mj_data.site_xpos[self._ee_site_id] + rotations[t] = self._mj_data.site_xmat[self._ee_site_id].reshape(3, 3) + return positions, rotations + + def _build_action(self, sample: dict[str, Any]) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """Build 10D UR action from FK EE poses derived from joint positions. + + Returns ``(action_10d, initial_pose, joint_configs)`` where: + - ``action_10d`` is ``(T, 10)``: ``[pos_delta(3) | rot6d_delta(6) | gripper(1)]`` + - ``initial_pose`` is ``(4, 4)`` float32 tensor (FK EE pose at frame 0) + - ``joint_configs`` is ``(T, 7)`` float32 — frames 1..T of joint positions + (arm joints + raw gripper) for FK mesh animation in the viewer. + """ + q = sample["actions.joint_position"] # [T+1, 7]: 6 arm joints + 1 gripper + q_np = q.numpy().astype(np.float32) if isinstance(q, torch.Tensor) else np.asarray(q, dtype=np.float32) + T = len(q_np) - 1 + + # FK EE trajectory: T+1 absolute poses from arm joints via MuJoCo + fk_pos, fk_rot = self._fk_ee_poses(q_np[:, :_UR5E_ARM_JOINTS]) + + poses_abs = np.tile(np.eye(4, dtype=np.float32), (T + 1, 1, 1)) + poses_abs[:, :3, 3] = fk_pos + poses_abs[:, :3, :3] = fk_rot @ self._to_opencv + + initial_pose = torch.from_numpy(poses_abs[0].copy()).float() + pose_convention = cast(PoseConvention, self._pose_convention) + poses_rel = cast( + np.ndarray, pose_abs_to_rel(poses_abs, rotation_format="rot6d", pose_convention=pose_convention) + ) + + # Raw UR gripper: 0=open, 1=closed (maps directly to Robotiq ctrl: raw*255, + # where 0=open, 255=closed). Invert so action uses 0=closed, 1=open + # joint_configs keeps the raw value; FK mesh uses raw * 255 → Robotiq ctrl. + gripper = torch.from_numpy(1.0 - q_np[:T, 6:7]) + action = torch.cat([torch.from_numpy(poses_rel).float(), gripper.float()], dim=-1) # [T, 10] + + # Mesh animation: frames 1..T of joint position (post-action states) + joint_configs = q_np[1 : 1 + T].copy() # [T, 7] + + return action, initial_pose, torch.from_numpy(joint_configs) + + def _build_action_spec(self) -> ActionSpec: + """RoboMIND UR: 10D = ``[Pos, Rot6d, Gripper]``.""" + return build_action_spec(Pos(), Rot("rot6d"), Gripper()) + + def __getitem__(self, idx: int) -> dict[str, Any]: + mode, _, _, sample = self._fetch_sample(idx) + + ai_caption = sample["task"] + if self._skip_video_loading: + video = None + else: + video = sample["observation.images.camera_top"] # [T,C,H,W] + + action, initial_pose, joint_configs = self._build_action(sample) + + return self._build_result( + mode=mode, + video=video, + action=action, + ai_caption=ai_caption, + initial_pose=initial_pose, + joint_configs=joint_configs, + ) + + @property + def action_dim(self) -> int: + return 10 # 9D SE(3) EE deltas + 1 gripper diff --git a/cosmos_framework/data/vfm/action/transforms.py b/cosmos_framework/data/vfm/action/transforms.py new file mode 100644 index 0000000..1941645 --- /dev/null +++ b/cosmos_framework/data/vfm/action/transforms.py @@ -0,0 +1,666 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Dataset transform wrappers for the Action project. + +This module provides the ``ActionTransformPipeline`` and spatial padding utilities. + +The reflection padding snaps each sample to the closest predefined resolution from +``VIDEO_RES_SIZE_INFO`` (matching VFM's approach), guaranteeing a bounded set of +output shapes that are all multiples of 16. + +See :func:`~.unified_dataset.wrap_dataset` for the convenience factory that +combines datasets with transforms, and :class:`~.unified_dataset.MapToIterableAdapter` +for the map-to-iterable wrapper. +""" + +from __future__ import annotations + +import torch +import torchvision.transforms.functional as transforms_F + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.json_formatter import ActionPromptJsonFormatter +from cosmos_framework.data.vfm.action.viewpoint_utils import ViewpointTextInfo +from cosmos_framework.data.vfm.augmentors.duration_fps_text_timestamps import DurationFPSTextTimeStamps +from cosmos_framework.data.vfm.augmentors.idle_frames_text_info import IdleFramesTextInfo +from cosmos_framework.data.vfm.augmentors.resolution_text_info import ResolutionTextInfo +from cosmos_framework.data.vfm.augmentors.text_tokenizer import TextTokenizerTransform +from cosmos_framework.data.vfm.sequence_packing import SequencePlan +from cosmos_framework.data.vfm.utils import VIDEO_RES_SIZE_INFO +from cosmos_framework.utils.vfm.data_utils import get_vision_data_resolution + + +def _should_append_idle_frame_info(mode: object) -> bool: + """Return whether idle-frame prompt metadata should be surfaced.""" + return mode != "inverse_dynamics" + + +def pad_action_to_max_dim(action: torch.Tensor, max_action_dim: int) -> torch.Tensor: + """Pad action tensor to max_action_dim along the last dimension. + + Args: + action: Action tensor of shape (T, D) where D is the current action dimension. + max_action_dim: Target action dimension to pad to. + + Returns: + Padded action tensor of shape (T, max_action_dim). + """ + if action.shape[-1] > max_action_dim: + raise ValueError(f"Action dimension {action.shape[-1]} is greater than max_action_dim {max_action_dim}") + elif action.shape[-1] == max_action_dim: + return action + else: + padding_size = max_action_dim - action.shape[-1] + zero_padding = torch.zeros( + *action.shape[:-1], padding_size, dtype=action.dtype, device=action.device + ) # [T,padding_size] + return torch.cat([action, zero_padding], dim=-1) # [T,max_action_dim] + + +def find_closest_target_size(h: int, w: int, resolution: str | int) -> tuple[int, int]: + """Find the closest predefined target size for a given input resolution. + + Looks up ``VIDEO_RES_SIZE_INFO[resolution]`` and selects the aspect ratio + whose ``H/W`` ratio is closest to the input ``h/w``. + + Args: + h: Input height in pixels. + w: Input width in pixels. + resolution: Resolution tier key (e.g. ``"256"``, ``"480"``, ``"720"``). + + Returns: + ``(target_w, target_h)`` from the predefined table. + + Raises: + ValueError: If *resolution* is not a key in ``VIDEO_RES_SIZE_INFO``. + """ + if isinstance(resolution, int): + resolution = str(resolution) + if resolution not in VIDEO_RES_SIZE_INFO: + raise ValueError( + f"Resolution '{resolution}' not found in VIDEO_RES_SIZE_INFO. Available: {list(VIDEO_RES_SIZE_INFO.keys())}" + ) + + candidates = VIDEO_RES_SIZE_INFO[resolution] + input_ratio = h / w + + best_key: str | None = None + best_diff = float("inf") + for aspect_key, (cand_w, cand_h) in candidates.items(): + cand_ratio = cand_h / cand_w + diff = abs(input_ratio - cand_ratio) + if diff < best_diff: + best_diff = diff + best_key = aspect_key + + assert best_key is not None + target_w, target_h = candidates[best_key] + return target_w, target_h + + +def reflection_pad_to_target( + data_dict: dict, + keys: list[str], + keep_aspect_ratio: bool, + target_w: int, + target_h: int, +) -> dict: + """Resize (aspect-preserving) and reflection-pad tensors to exact target size. + + For each key in *keys*, the tensor is: + + 1. Resized so its spatial dimensions fit within ``(target_h, target_w)`` + while preserving the aspect ratio (matching VFM's + ``ResizeLargestSideAspectPreserving``). + 2. Reflection-padded (or edge-padded when the padding exceeds the spatial + dimension) to reach exactly ``(target_h, target_w)`` (matching VFM's + ``ReflectionPadding``). + + After processing, the following entries are added to *data_dict*: + + - ``"image_size"``: ``torch.Tensor`` of shape ``(4,)`` containing + ``[target_h, target_w, orig_h_resized, orig_w_resized]`` where + ``target_h/w`` is the padded canvas size and ``orig_h/w_resized`` + is the original spatial size after aspect-preserving resize (i.e. + the content region before padding). After ``default_collate`` + this becomes ``(B, 4)``; the ``IterativeJointDataLoader`` then + splits it into per-sample ``(1, 4)`` tensors so the model can + index as ``data_batch["image_size"][i][0][0]``. + + Args: + data_dict: The sample dictionary (mutated in-place). + keys: Data-dict keys whose tensors should be resized and padded. + Tensors must have shape ``(C, H, W)`` or ``(C, T, H, W)``. + keep_aspect_ratio: Whether to keep the aspect ratio of the input tensor. + target_w: Target width in pixels. + target_h: Target height in pixels. + + Returns: + The mutated *data_dict*. + """ + orig_h_resized: int = 0 + orig_w_resized: int = 0 + + for key in keys: + if key not in data_dict: + continue + tensor = data_dict[key] + if not isinstance(tensor, torch.Tensor): + continue + + # Extract spatial dims + if tensor.ndim == 3: + orig_h, orig_w = tensor.shape[-2:] + elif tensor.ndim == 4: + orig_h, orig_w = tensor.shape[-2:] + else: + raise ValueError(f"Unexpected tensor ndim={tensor.ndim} for key '{key}', expected 3 or 4") + + # Step 1: aspect-preserving resize to fit within (target_h, target_w) + if keep_aspect_ratio: + # Prevent upscaling the video by setting the upper bound of scaling_ratio to 1.0. + scaling_ratio = min(target_w / orig_w, target_h / orig_h, 1.0) + orig_h_resized = int(scaling_ratio * orig_h + 0.5) + orig_w_resized = int(scaling_ratio * orig_w + 0.5) + assert orig_h_resized <= target_h and orig_w_resized <= target_w, ( + f"Resize error: orig ({orig_h}, {orig_w}) target ({target_h}, {target_w}) " + f"computed ({orig_h_resized}, {orig_w_resized})" + ) + else: + orig_h_resized = target_h + orig_w_resized = target_w + + if orig_h_resized != orig_h or orig_w_resized != orig_w: + tensor = transforms_F.resize( + tensor, + size=[orig_h_resized, orig_w_resized], + interpolation=transforms_F.InterpolationMode.BICUBIC, + antialias=True, + ) + + # Step 2: padding to exact target size (bottom and right only) + if orig_w_resized != target_w or orig_h_resized != target_h: + padding_right = target_w - orig_w_resized + padding_bottom = target_h - orig_h_resized + padding = [0, 0, padding_right, padding_bottom] + + if padding_right >= orig_w_resized or padding_bottom >= orig_h_resized: + tensor = transforms_F.pad(tensor, padding, padding_mode="edge") + else: + tensor = transforms_F.pad(tensor, padding, padding_mode="reflect") + + data_dict[key] = tensor + + # image_size: shape (4,) — [target_h, target_w, orig_h_resized, orig_w_resized]. + # Matches VFM's item_dataset convention. default_collate stacks to (B, 4); + # IterativeJointDataLoader._get_next_sample slices to (1, 4) per sample so + # the model can index [i][0][0]. + data_dict["image_size"] = torch.tensor( + [target_h, target_w, orig_h_resized, orig_w_resized], dtype=torch.float + ) # [4] + + return data_dict + + +def remove_reflection_padding( + tensor: torch.Tensor, + image_size: torch.Tensor, +) -> torch.Tensor: + """Remove reflection padding added by :func:`reflection_pad_to_target`. + + Content is at top-left; crops to ``(orig_h_resized, orig_w_resized)``. + + Args: + tensor: Tensor whose last two dimensions are the padded spatial dims. + Supports any leading dimensions, e.g. ``(C, T, H, W)`` or + ``(C, H, W)``. + image_size: 1-D tensor of shape ``(4,)`` containing + ``[target_h, target_w, orig_h_resized, orig_w_resized]`` where + ``orig_h/w_resized`` is the original spatial size after + aspect-preserving resize (i.e. the content region before + padding) — the same convention stored by + :func:`reflection_pad_to_target` and VFM's + ``ReflectionPadding``. + + Returns: + Cropped tensor of shape ``(..., orig_h_resized, orig_w_resized)``. + """ + target_h = int(image_size[0].item()) + target_w = int(image_size[1].item()) + orig_h_resized = int(image_size[2].item()) + orig_w_resized = int(image_size[3].item()) + + if orig_h_resized == target_h and orig_w_resized == target_w: + return tensor + + return tensor[..., :orig_h_resized, :orig_w_resized].contiguous() + + +def build_sequence_plan_from_mode( + mode: str, + video_length: int, + action_length: int, + has_text: bool = True, + video_temporal_downsample: int = 4, + num_history_actions: int = 0, +) -> SequencePlan: + """Build a SequencePlan based on the training mode. + + This function determines whether action should be included and computes the + appropriate condition frame indexes for vision and action based on the mode. + + Args: + mode: Training mode. One of: + - "image2video": Image-to-video generation (no action) + - "forward_dynamics": Predict video given first frame and all actions + - "inverse_dynamics": Predict actions given all video frames + - "policy": Predict both actions and video given first frame + video_length: Number of video frames (including the conditioning frame). + action_length: Number of action steps (typically video_length - 1). + has_text: Whether text conditioning is available. Defaults to True. + video_temporal_downsample: Temporal downsampling factor of the video + tokenizer. Used to compute condition frame indexes for inverse + dynamics mode. Defaults to 4. + + Returns: + SequencePlan instance with appropriate settings. + Use ``sequence_plan.has_action`` to check if action should be included. + + Raises: + ValueError: If mode is not one of the supported modes. + + Example: + >>> sequence_plan = build_sequence_plan_from_mode( + ... mode="policy", + ... video_length=5, + ... action_length=4, + ... ) + >>> sequence_plan.has_action + True + >>> sequence_plan.as_dict() + {'has_text': True, 'has_vision': True, 'has_action': True, + 'condition_frame_indexes_vision': [0], 'condition_frame_indexes_action': []} + """ + valid_modes = ["image2video", "forward_dynamics", "inverse_dynamics", "policy"] + if mode not in valid_modes: + raise ValueError(f"Invalid mode: {mode!r}. Must be one of {valid_modes}") + + # Determine if action should be included based on mode + # image2video mode: no action (pure image-to-video generation) + # forward_dynamics, inverse_dynamics, policy: action is needed + has_action = mode != "image2video" + + # Determine condition frame indexes based on mode + # image2video/forward_dynamics/policy: first frame is clean (conditioning) + # inverse_dynamics: all frames are provided as context + if mode in ["image2video", "forward_dynamics", "policy"]: + condition_frame_indexes_vision = [0] + elif mode == "inverse_dynamics": + # All frames are observed for inverse dynamics + condition_frame_indexes_vision = list(range(0, (video_length - 1) // video_temporal_downsample + 1)) + else: + condition_frame_indexes_vision = [] + + # For action conditioning indexes: + # forward_dynamics: all action steps are clean (conditioning) + # inverse_dynamics/policy: action is supervised (predicted) + # History frames (prepended) are always conditioning. + base_action_length = action_length - num_history_actions + if mode == "forward_dynamics": + condition_frame_indexes_action = list(range(action_length)) + + # This currently assumes that the action length is the same as the video length - 1 + # and if action length is the same as the video length, then the first action is the conditioning action + elif base_action_length == video_length - 1: + condition_frame_indexes_action = list(range(num_history_actions)) + elif base_action_length == video_length: + condition_frame_indexes_action = list(range(num_history_actions + 1)) + + if base_action_length == video_length - 1: + action_start_frame_offset = 1 - num_history_actions + if base_action_length == video_length: + action_start_frame_offset = -num_history_actions + + return SequencePlan( + has_text=has_text, + has_vision=True, + has_action=has_action, + condition_frame_indexes_vision=condition_frame_indexes_vision, + condition_frame_indexes_action=condition_frame_indexes_action, + action_start_frame_offset=action_start_frame_offset, + ) + + +class VideoResize: + """Resize and reflection-pad video-aligned tensors for a single sample. + + Resolution is supplied at call time. When ``resolution`` is ``None``, the + tier is auto-detected from the sample's ``"video"`` spatial dimensions. + + Args: + pad_keys: Data-dict keys whose values should be resized and padded. + Pass an empty list to disable padding entirely. Defaults to + ``["video"]``. + keep_aspect_ratio: Whether to resize aspect-preservingly to the closest + predefined target size before padding. Defaults to ``True``. + log_prefix: Prefix used in debug logging. + """ + + def __init__( + self, + pad_keys: list[str] | None = None, + keep_aspect_ratio: bool = True, + log_prefix: str = "VideoResize", + ) -> None: + self.pad_keys = pad_keys if pad_keys is not None else ["video"] + self.keep_aspect_ratio = keep_aspect_ratio + self.log_prefix = log_prefix + + def __call__(self, data_dict: dict, resolution: str | int | None) -> dict: + """Resize and pad a sample in-place. + + Args: + data_dict: Sample dictionary containing a ``"video"`` entry. + resolution: Resolution tier key (e.g. ``"256"``, ``"480"``, + ``"720"``). When ``None``, auto-detected from video dimensions. + + Returns: + The same dictionary, mutated in-place with padded tensors and an + ``"image_size"`` entry. + """ + video = data_dict.get("video") + assert isinstance(video, torch.Tensor), "video is required for reflection padding" + h, w = video.shape[-2:] + + if resolution is None: + resolution = get_vision_data_resolution((h, w)) + + if self.keep_aspect_ratio: + target_w, target_h = find_closest_target_size(h, w, resolution) + else: + target_w = int(resolution) + target_h = int(resolution) + reflection_pad_to_target(data_dict, self.pad_keys, self.keep_aspect_ratio, target_w, target_h) + + return data_dict + + def _log_shapes(self, data_dict: dict, when: str) -> None: + """Log tensor shapes for the configured pad keys.""" + for key in self.pad_keys: + val = data_dict.get(key) + if isinstance(val, torch.Tensor): + log.debug(f"{self.log_prefix}: {when} padding '{key}' shape = {tuple(val.shape)}") + + +class ActionTransformPipeline: + """A composable transform pipeline that chains ``VideoResize``, text + tokenization, and automatic sequence plan construction. + + Reflection padding snaps each sample to the closest predefined aspect + ratio from ``VIDEO_RES_SIZE_INFO[resolution]``, resizes + (aspect-preserving) to fit within the target, then reflection-pads to + the exact target size. This guarantees a bounded set of output shapes + (5 per resolution tier), all multiples of 16. Resolution is supplied + at call time via the required ``resolution`` argument to ``__call__``; + when ``resolution`` is ``None``, the tier is auto-detected from the + video's spatial dimensions via ``get_vision_data_resolution``. + + Text tokenization is enabled when ``tokenizer_config`` is provided. + + When the data dictionary contains a ``"mode"`` key, the pipeline automatically + builds a ``SequencePlan`` via :func:`build_sequence_plan_from_mode` and attaches + it as ``data_dict["sequence_plan"]``. For modes where action is not needed + (e.g. ``"image2video"``), the ``"action"`` and ``"domain_id"`` keys are set to + ``None``. + + Args: + pad_keys: Data-dict keys whose values should be resized and padded. Pass + an empty list to disable padding entirely. Defaults to ``["video"]``. + tokenizer_config: A lazy-instantiable config dict for the VLM tokenizer. When + ``None``, text tokenization is skipped. Defaults to ``None``. + cfg_dropout_rate: Probability of replacing the caption with an empty string for + classifier-free guidance. Only used when text tokenization is enabled. + Defaults to ``0.0``. + caption_key: The data-dict key that contains the input caption string. + Defaults to ``"ai_caption"``. + text_token_key: The data-dict key where tokenized text IDs will be stored. + Defaults to ``"text_token_ids"``. + video_temporal_downsample: Temporal downsampling factor of the video tokenizer. + Used when building a ``SequencePlan`` for ``"inverse_dynamics"`` mode. + Defaults to 4. + max_action_dim: Target action dimension to pad to. The ``"action"`` tensor + in every sample is padded along its last dimension via + :func:`pad_action_to_max_dim`. Defaults to 32. + action_channel_masking: When ``True`` (default), the original action + dimension is stored in ``"raw_action_dim"`` so that the model masks + loss/noise/velocity on zero-padded action channels. When ``False``, + ``"raw_action_dim"`` is set to ``None`` and the model treats all + ``max_action_dim`` channels equally (original main-branch behavior). + append_viewpoint_info: Whether to append viewpoint type metadata to the + caption (via ``ViewpointTextInfo`` augmentor). Requires that + samples contain a ``"viewpoint"`` key. Defaults to ``True``. + append_duration_fps_timestamps: Whether to append duration and FPS metadata to the + caption (matching VFM's ``DurationFPSTextTimeStamps`` augmentor). + Defaults to ``True``. + append_resolution_info: Whether to append resolution metadata to the + caption (matching VFM's ``ResolutionTextInfo`` augmentor). + Defaults to ``True``. + append_idle_frames: Whether to append the idle-frame count out of the + total action frames to the caption (Pi0.7-style metadata, via + ``IdleFramesTextInfo`` augmentor). The dataset is responsible for + populating ``data_dict["idle_frames"]``; samples without it are + silently skipped. Idle-frame text is skipped only for + ``"inverse_dynamics"`` mode. Defaults to ``False`` so existing + experiments are unaffected. + idle_frames_dropout: Per-field dropout rate for the idle-frame segment. + With this probability the augmentor leaves the caption unchanged + (matching Pi0.7's ~5% per-component dropout). Independent of the + global ``cfg_dropout_rate``, which empties the whole caption. + Defaults to 0.05. + format_prompt_as_json: Whether to replace the plain text prompt with a + structured JSON-compatible dictionary before tokenization. When + enabled, legacy string metadata appenders are skipped and the JSON + formatter owns viewpoint, action, resolution, duration, FPS, and + idle-frame fields. Defaults to ``False``. + """ + + def __init__( + self, + pad_keys: list[str] | None = None, + keep_aspect_ratio: bool = True, + tokenizer_config: dict | None = None, + cfg_dropout_rate: float = 0.0, + caption_key: str = "ai_caption", + text_token_key: str = "text_token_ids", + video_temporal_downsample: int = 4, + max_action_dim: int = 32, + action_channel_masking: bool = True, + append_viewpoint_info: bool = True, + append_duration_fps_timestamps: bool = True, + append_resolution_info: bool = True, + append_idle_frames: bool = False, + idle_frames_dropout: float = 0.05, + format_prompt_as_json: bool = False, + ) -> None: + self.caption_key: str = caption_key + self.video_temporal_downsample: int = video_temporal_downsample + self.max_action_dim: int = max_action_dim + self.action_channel_masking: bool = action_channel_masking + + # --- Spatial resize/padding stage (resolution supplied at call time) --- + self.video_resize: VideoResize = VideoResize( + pad_keys=pad_keys, + keep_aspect_ratio=keep_aspect_ratio, + log_prefix="ActionTransformPipeline", + ) + self.pad_keys: list[str] = self.video_resize.pad_keys + self.keep_aspect_ratio: bool = self.video_resize.keep_aspect_ratio + + self.prompt_json_formatter: ActionPromptJsonFormatter | None = None + if format_prompt_as_json: + self.prompt_json_formatter = ActionPromptJsonFormatter(caption_key=caption_key) + + # --- Viewpoint text augmentor (runs after ai_caption, before duration/FPS) --- + self.viewpoint_augmentor: ViewpointTextInfo | None = None + if append_viewpoint_info and self.prompt_json_formatter is None: + self.viewpoint_augmentor = ViewpointTextInfo( + input_keys=[caption_key, "viewpoint"], + output_keys=[caption_key], + args={"caption_key": caption_key, "viewpoint_key": "viewpoint", "enabled": True}, + ) + + # --- Duration/FPS text augmentor (runs before tokenization) --- + self.duration_fps_augmentor: DurationFPSTextTimeStamps | None = None + if append_duration_fps_timestamps and self.prompt_json_formatter is None: + self.duration_fps_augmentor = DurationFPSTextTimeStamps( + input_keys=[caption_key, "video", "conditioning_fps"], + output_keys=[caption_key], + args={"caption_key": caption_key, "video_key": "video", "fps_key": "conditioning_fps"}, + ) + + # --- Resolution text augmentor (runs before tokenization) --- + self.resolution_info_augmentor: ResolutionTextInfo | None = None + if append_resolution_info and self.prompt_json_formatter is None: + self.resolution_info_augmentor = ResolutionTextInfo( + input_keys=[caption_key, "video", "image_size"], + output_keys=[caption_key], + args={"caption_key": caption_key, "video_key": "video", "enabled": True}, + ) + + # --- IdleFrames text augmentor (Pi0.7-style episode metadata) --- + # Runs after resolution info, before tokenization. Per-field dropout is + # independent from the tokenizer's global cfg_dropout_rate. + self.idle_frames_augmentor: IdleFramesTextInfo | None = None + if append_idle_frames and self.prompt_json_formatter is None: + self.idle_frames_augmentor = IdleFramesTextInfo( + input_keys=[caption_key, "idle_frames", "action"], + output_keys=[caption_key], + args={ + "caption_key": caption_key, + "idle_frames_key": "idle_frames", + "action_key": "action", + "dropout_rate": idle_frames_dropout, + "enabled": True, + }, + ) + + # --- Text tokenizer augmentor --- + self.text_tokenizer: TextTokenizerTransform | None = None + if tokenizer_config is not None: + self.text_tokenizer = TextTokenizerTransform( + input_keys=[caption_key], + output_keys=[text_token_key], + args={ + "tokenizer_config": tokenizer_config, + "cfg_dropout_rate": cfg_dropout_rate, + }, + ) + + def __call__(self, data_dict: dict, resolution: str | None) -> dict: + """Apply the transform pipeline to a single data dictionary. + + Resolution is required at call time and is the only source of truth + for this sample. When ``resolution`` is ``None``, the tier is + auto-detected from the video's spatial dimensions. + + The pipeline runs in order: + + 1. Resize + reflection-pad spatial dimensions to the closest + predefined target from ``VIDEO_RES_SIZE_INFO[resolution]``. + 2. Format the caption as a structured JSON prompt (if enabled). + 3. Otherwise, append viewpoint type metadata to caption (if enabled). + 4. Append duration/FPS metadata to caption (if enabled). + 5. Append resolution metadata to caption (if enabled). + 6. Append idle-frame metadata (Pi0.7-style) to caption unless the + sample is in inverse dynamics mode (if enabled). + 7. Tokenize caption text (if enabled). + 8. Build a ``SequencePlan`` from the ``"mode"`` key (if present). + 9. If action is needed by the plan, pad ``"action"`` to ``max_action_dim``. + 10. Otherwise, nullify ``"action"`` and ``"domain_id"`` (e.g. in + ``"image2video"`` mode). + + Args: + data_dict: A sample dictionary as returned by a Action dataset. + resolution: Resolution tier key (e.g. ``"256"``, ``"480"``, ``"720"``) + for this sample. When ``None``, auto-detected from video dimensions. + + Returns: + The same dictionary, mutated in-place with padded tensors, + ``image_size``, tokenized text IDs, and a + ``"sequence_plan"`` entry added. + """ + mode = data_dict.get("mode") + assert mode is not None, "mode is required" + + # 1. Resize + reflection-pad spatial dimensions to the closest predefined target from ``VIDEO_RES_SIZE_INFO[resolution]``. + data_dict = self.video_resize(data_dict, resolution) + + # 2. Format the caption as structured JSON when requested; otherwise run the legacy string appenders. + if self.prompt_json_formatter is not None: + data_dict = self.prompt_json_formatter(data_dict) + else: + # 3. Append viewpoint type metadata to caption (if enabled). + if self.viewpoint_augmentor is not None: + result = self.viewpoint_augmentor(data_dict) + if result is not None: + data_dict = result + + # 4. Append duration/FPS metadata to caption (if enabled). + if self.duration_fps_augmentor is not None: + result = self.duration_fps_augmentor(data_dict) + if result is not None: + data_dict = result + + # 5. Append resolution metadata to caption (if enabled). + if self.resolution_info_augmentor is not None: + result = self.resolution_info_augmentor(data_dict) + if result is not None: + data_dict = result + + # 6. Append idle-frame metadata to caption (if enabled for this mode). + if self.idle_frames_augmentor is not None and _should_append_idle_frame_info(mode): + result = self.idle_frames_augmentor(data_dict) + if result is not None: + data_dict = result + + # 7. Tokenize caption text (if enabled). + if self.text_tokenizer is not None: + data_dict = self.text_tokenizer(data_dict) + + # 8. Build a ``SequencePlan`` from the ``"mode"`` key (if present). + video = data_dict.get("video") + action = data_dict.get("action") + assert video is not None, "video is required" + video_length = video.shape[1] # [C,T,H,W] -> T + action_length = action.shape[0] if isinstance(action, torch.Tensor) else max(video_length - 1, 0) + + # Prepend history action frames (ground-truth conditioning) if present. + history_action = data_dict.pop("history_action", None) + num_history_actions = 0 + if history_action is not None and isinstance(action, torch.Tensor): + num_history_actions = history_action.shape[0] + action = torch.cat([history_action, action], dim=0) + action_length += num_history_actions + + sequence_plan = build_sequence_plan_from_mode( + mode=mode, + video_length=video_length, + action_length=action_length, + video_temporal_downsample=self.video_temporal_downsample, + num_history_actions=num_history_actions, + ) + data_dict["sequence_plan"] = sequence_plan + + if sequence_plan.has_action: + assert isinstance(action, torch.Tensor), "action tensor is required when sequence plan has action" + data_dict["raw_action_dim"] = torch.tensor(action.shape[1]) if self.action_channel_masking else None + data_dict["action"] = pad_action_to_max_dim(action, self.max_action_dim) + else: + # Nullify action-related fields when action is not needed so the + # collate function can simply stack all non-None actions. + data_dict["raw_action_dim"] = None + data_dict["action"] = None + data_dict["domain_id"] = None + + return data_dict diff --git a/cosmos_framework/data/vfm/action/transforms_test.py b/cosmos_framework/data/vfm/action/transforms_test.py new file mode 100644 index 0000000..5f024fb --- /dev/null +++ b/cosmos_framework/data/vfm/action/transforms_test.py @@ -0,0 +1,325 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest +import torch + +from cosmos_framework.data.vfm.action.json_formatter import ActionPromptJsonFormatter +from cosmos_framework.data.vfm.action.transforms import ActionTransformPipeline +from cosmos_framework.data.vfm.augmentors.duration_fps_text_timestamps import DurationFPSTextTimeStamps +from cosmos_framework.data.vfm.augmentors.resolution_text_info import ResolutionTextInfo + + +@pytest.mark.L0 +def test_action_prompt_json_formatter_builds_requested_structure() -> None: + formatter = ActionPromptJsonFormatter() + video = torch.zeros(3, 12, 480, 640) # [C,T,H,W] + action = torch.zeros(11, 7) # [T,D] + image_size = torch.tensor([480, 640, 480, 640]) # [4] + fps = torch.tensor(24) # [] + idle_frames = torch.tensor(2) # [] + data_dict = { + "ai_caption": "Pick up the cup", + "video": video, + "action": action, + "conditioning_fps": fps, + "image_size": image_size, + "viewpoint": "concat_view", + "additional_view_description": "The top row is the wrist camera and the bottom row is the scene camera.", + "idle_frames": idle_frames, + } + + result = formatter(data_dict) + + prompt = result["ai_caption"] + assert list(prompt.keys()) == ["cinematography", "actions", "duration", "fps", "resolution", "aspect_ratio"] + assert list(prompt["actions"][0].keys()) == ["time", "description", "idle_frame"] + assert prompt == { + "cinematography": { + "framing": ( + "This video contains concatenated views from multiple camera perspectives. " + "The top row is the wrist camera and the bottom row is the scene camera." + ) + }, + "actions": [ + { + "time": "0:00-0:00", + "description": "Pick up the cup.", + "idle_frame": "2 out of 11.", + } + ], + "duration": "0s", + "fps": 24.0, + "resolution": {"H": 480, "W": 640}, + "aspect_ratio": "4,3", + } + assert "additional_view_description" not in result + + +@pytest.mark.L0 +def test_action_prompt_json_formatter_drops_empty_fields() -> None: + formatter = ActionPromptJsonFormatter() + video = torch.zeros(3, 12, 480, 640) # [C,T,H,W] + action = torch.zeros(11, 7) # [T,D] + image_size = torch.tensor([480, 640, 480, 640]) # [4] + fps = torch.tensor(24) # [] + data_dict = { + "ai_caption": "Pick up the cup.", + "video": video, + "action": action, + "conditioning_fps": fps, + "image_size": image_size, + "viewpoint": "third_person_view", + } + + result = formatter(data_dict) + + assert result["ai_caption"]["actions"] == [ + { + "time": "0:00-0:00", + "description": "Pick up the cup.", + } + ] + + +@pytest.mark.L0 +def test_action_prompt_json_formatter_drops_empty_viewpoint() -> None: + formatter = ActionPromptJsonFormatter() + video = torch.zeros(3, 12, 480, 640) # [C,T,H,W] + action = torch.zeros(11, 7) # [T,D] + image_size = torch.tensor([480, 640, 480, 640]) # [4] + fps = torch.tensor(24) # [] + data_dict = { + "ai_caption": "Pick up the cup.", + "video": video, + "action": action, + "conditioning_fps": fps, + "image_size": image_size, + } + + result = formatter(data_dict) + + assert "cinematography" not in result["ai_caption"] + + +@pytest.mark.L0 +def test_action_transform_pipeline_json_prompt_toggle() -> None: + pipeline = ActionTransformPipeline( + tokenizer_config=None, + max_action_dim=4, + format_prompt_as_json=True, + ) + video = torch.zeros(3, 17, 192, 320) # [C,T,H,W] + action = torch.zeros(16, 2) # [T,D] + data_dict = { + "ai_caption": "Open the drawer.", + "video": video, + "action": action, + "conditioning_fps": torch.tensor(8), # [] + "mode": "policy", + "domain_id": torch.tensor(0), # [] + "viewpoint": "third_person_view", + "idle_frames": torch.tensor(3), # [] + } + + result = pipeline(data_dict, resolution="256") + + prompt = result["ai_caption"] + assert isinstance(prompt, dict) + assert list(prompt.keys()) == ["cinematography", "actions", "duration", "fps", "resolution", "aspect_ratio"] + assert list(prompt["actions"][0].keys()) == ["time", "description", "idle_frame"] + assert prompt["cinematography"] == { + "framing": "This video is captured from a third-person perspective looking towards the agent from the front." + } + assert prompt["actions"] == [ + { + "time": "0:00-0:02", + "description": "Open the drawer.", + "idle_frame": "3 out of 16.", + } + ] + assert prompt["duration"] == "2s" + assert prompt["fps"] == 8.0 + assert prompt["resolution"] == {"H": 192, "W": 320} + assert prompt["aspect_ratio"] == "16,9" + assert result["action"].shape == (16, 4) + + +@pytest.mark.L0 +def test_action_transform_pipeline_keeps_ai_caption_string_path() -> None: + pipeline = ActionTransformPipeline( + tokenizer_config=None, + max_action_dim=4, + append_idle_frames=True, + idle_frames_dropout=0.0, + ) + video = torch.zeros(3, 17, 256, 256) # [C,T,H,W] + action = torch.zeros(16, 2) # [T,D] + data_dict = { + "ai_caption": "Open the drawer.", + "video": video, + "action": action, + "conditioning_fps": torch.tensor(8), # [] + "mode": "policy", + "domain_id": torch.tensor(0), # [] + "viewpoint": "third_person_view", + "idle_frames": torch.tensor(3), # [] + } + + result = pipeline(data_dict, resolution="256") + + assert result["ai_caption"] == ( + "Open the drawer. " + "This video is captured from a third-person perspective looking towards the agent from the front. " + "The video is 2.0 seconds long and is of 8 FPS. " + "This video is of 256x256 resolution. " + "IdleFrames: 3 out of 16." + ) + assert result["action"].shape == (16, 4) + + +@pytest.mark.L0 +def test_action_transform_pipeline_keeps_idle_frames_for_forward_dynamics() -> None: + pipeline = ActionTransformPipeline( + tokenizer_config=None, + max_action_dim=4, + append_idle_frames=True, + idle_frames_dropout=0.0, + ) + video = torch.zeros(3, 17, 256, 256) # [C,T,H,W] + action = torch.zeros(16, 2) # [T,D] + data_dict = { + "ai_caption": "Open the drawer.", + "video": video, + "action": action, + "conditioning_fps": torch.tensor(8), # [] + "mode": "forward_dynamics", + "domain_id": torch.tensor(0), # [] + "viewpoint": "third_person_view", + "idle_frames": torch.tensor(3), # [] + } + + result = pipeline(data_dict, resolution="256") + + assert "IdleFrames: 3 out of 16." in result["ai_caption"] + assert result["action"].shape == (16, 4) + + +@pytest.mark.L0 +def test_action_transform_pipeline_skips_idle_frames_for_inverse_dynamics_string_path() -> None: + pipeline = ActionTransformPipeline( + tokenizer_config=None, + max_action_dim=4, + append_idle_frames=True, + idle_frames_dropout=0.0, + ) + video = torch.zeros(3, 17, 256, 256) # [C,T,H,W] + action = torch.zeros(16, 2) # [T,D] + data_dict = { + "ai_caption": "Open the drawer.", + "video": video, + "action": action, + "conditioning_fps": torch.tensor(8), # [] + "mode": "inverse_dynamics", + "domain_id": torch.tensor(0), # [] + "viewpoint": "third_person_view", + "idle_frames": torch.tensor(3), # [] + } + + result = pipeline(data_dict, resolution="256") + + assert "IdleFrames" not in result["ai_caption"] + assert result["action"].shape == (16, 4) + + +@pytest.mark.L0 +def test_action_transform_pipeline_skips_idle_frames_for_inverse_dynamics_json_prompt() -> None: + pipeline = ActionTransformPipeline( + tokenizer_config=None, + max_action_dim=4, + format_prompt_as_json=True, + ) + video = torch.zeros(3, 17, 256, 256) # [C,T,H,W] + action = torch.zeros(16, 2) # [T,D] + data_dict = { + "ai_caption": "Open the drawer.", + "video": video, + "action": action, + "conditioning_fps": torch.tensor(8), # [] + "mode": "inverse_dynamics", + "domain_id": torch.tensor(0), # [] + "viewpoint": "third_person_view", + "idle_frames": torch.tensor(3), # [] + } + + result = pipeline(data_dict, resolution="256") + + prompt = result["ai_caption"] + assert isinstance(prompt, dict) + assert prompt["actions"] == [ + { + "time": "0:00-0:02", + "description": "Open the drawer.", + } + ] + assert result["action"].shape == (16, 4) + + +@pytest.mark.L0 +def test_action_prompt_json_formatter_matches_video_json_common_metadata() -> None: + formatter = ActionPromptJsonFormatter() + video = torch.zeros(3, 23, 192, 320) # [C,T,H,W] + image_size = torch.tensor([192, 320, 192, 320]) # [4] + fps = torch.tensor(8.0) # [] + action_data_dict = { + "ai_caption": "Open the drawer.", + "video": video, + "action": torch.zeros(22, 2), # [T,D] + "conditioning_fps": fps, + "image_size": image_size, + "viewpoint": "third_person_view", + "idle_frames": torch.tensor(3), # [] + } + + action_prompt = formatter(action_data_dict)["ai_caption"] + + video_data_dict = { + "ai_caption": { + "cinematography": { + "framing": "This video is captured from a third-person perspective looking towards the agent from the front." + }, + "actions": [ + { + "time": "0:00-0:03", + "description": "Open the drawer.", + } + ], + }, + "video": video, + "conditioning_fps": fps, + "image_size": image_size, + "__url__": SimpleNamespace(meta=SimpleNamespace(opts={"aspect_ratio": "16,9"})), + } + duration_augmentor = DurationFPSTextTimeStamps( + input_keys=["ai_caption", "video", "conditioning_fps"], + args={"caption_key": "ai_caption", "video_key": "video", "fps_key": "conditioning_fps"}, + ) + resolution_augmentor = ResolutionTextInfo( + input_keys=["ai_caption", "video", "image_size"], + args={"caption_key": "ai_caption", "video_key": "video"}, + ) + duration_augmentor(video_data_dict) + resolution_augmentor(video_data_dict) + video_prompt = video_data_dict["ai_caption"] + + common_top_level_keys = ["cinematography", "duration", "fps", "resolution", "aspect_ratio"] + assert {key: action_prompt[key] for key in common_top_level_keys} == { + key: video_prompt[key] for key in common_top_level_keys + } + assert action_prompt["actions"][0]["time"] == video_prompt["actions"][0]["time"] + assert action_prompt["actions"][0]["description"] == video_prompt["actions"][0]["description"] + assert action_prompt["actions"][0]["idle_frame"] == "3 out of 22." diff --git a/cosmos_framework/data/vfm/action/umi/data_classes.py b/cosmos_framework/data/vfm/action/umi/data_classes.py new file mode 100644 index 0000000..081639a --- /dev/null +++ b/cosmos_framework/data/vfm/action/umi/data_classes.py @@ -0,0 +1,97 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from dataclasses import dataclass +from typing import Any, Union, cast + +from omegaconf import DictConfig, OmegaConf + + +@dataclass +class SourceDataMeta: + name: str + """The data name from the source dataset.""" + shape: tuple[int, ...] + """The shape of a single time step of the data.""" + include_indices: list[int] + """Indices of the data to include in the dataset relative to the current step (0). Negative indices means the data is from the past.""" + + def __post_init__(self): + if isinstance(self.shape, list): + self.shape = tuple(self.shape) + if len(self.include_indices) == 0: + raise ValueError(f"include_indices must be a non-empty list in {self.name}.") + for i, index in enumerate(self.include_indices): + if i < len(self.include_indices) - 1: + if index > self.include_indices[i + 1]: + raise ValueError( + f"include_indices must be monotonically increasing, but got {self.include_indices} in {self.name}." + ) + if len(self.shape) == 0: + raise ValueError(f"shape must be a non-empty list in {self.name}.") + + +@dataclass +class DataMeta: + name: str + """The output name to be used for training.""" + shape: tuple[int, ...] + """The shape of a single time step of the data.""" + data_type: str + """low_dim or image""" + length: int + """The length of the data.""" + normalizer: str + """identity, range, normal. range: normalize to [-1, 1]; normal: normalize to mean=0, std=1""" + augmentation: list[dict[str, Any]] + """The augmentation to apply to the data.""" + source_entry_names: list[str] + """The source entry names to use for the data.""" + + def __post_init__(self): + if isinstance(self.shape, list): + self.shape = tuple(self.shape) + + if self.data_type not in ["low_dim", "image"]: + raise ValueError(f"data_type must be one of ['low_dim', 'image'] in {self.name}.") + + if len(self.source_entry_names) == 0: + raise ValueError(f"source_entry_names must be a non-empty list in {self.name}.") + + if self.length <= 0: + raise ValueError(f"length must be greater than 0 in {self.name}.") + + if len(self.shape) == 0: + raise ValueError(f"shape must be a non-empty list in {self.name}.") + + if self.normalizer not in ["identity", "range", "normal", "clamped_range"]: + raise ValueError( + f"normalizer must be one of ['identity', 'range', 'normal', 'clamped_range'] in {self.name}." + ) + + +def construct_data_meta( + data_meta: Union[dict[str, dict[str, Any]], DictConfig], +) -> dict[str, DataMeta]: + if isinstance(data_meta, DictConfig): + data_meta = cast(dict[str, dict[str, Any]], OmegaConf.to_container(data_meta, resolve=True)) + data_meta_dict = {} + for name, entry_meta_dict in data_meta.items(): + entry_meta_dict.update({"name": name}) + data_meta_dict[name] = DataMeta(**entry_meta_dict) + return data_meta_dict + + +def construct_source_data_meta( + source_data_meta: Union[dict[str, dict[str, Any]], DictConfig], +) -> dict[str, SourceDataMeta]: + if isinstance(source_data_meta, DictConfig): + source_data_meta = cast( + dict[str, dict[str, Any]], + OmegaConf.to_container(source_data_meta, resolve=True), + ) + source_data_meta_dict = {} + for name, entry_meta_dict in source_data_meta.items(): + entry_meta_dict.update({"name": name}) + source_data_meta_dict[name] = SourceDataMeta(**entry_meta_dict) + return source_data_meta_dict diff --git a/cosmos_framework/data/vfm/action/umi/data_utils.py b/cosmos_framework/data/vfm/action/umi/data_utils.py new file mode 100644 index 0000000..937fe83 --- /dev/null +++ b/cosmos_framework/data/vfm/action/umi/data_utils.py @@ -0,0 +1,38 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Any, Callable + +import numpy as np +import torch + + +def aggregate_batch(batch: list[Any], aggregate_fn: Callable[[list[Any]], Any], merge_none: bool = True) -> Any: + """ + Custom collate function to concatenate nested tensors/ndarray/float along a specified axis. + If merge_none is True, the field that has None values will be merged into a single None value. Otherwise will return a list of None values. + Popular choices of aggregate_fn: + - partial(torch.cat, dim=existing_dim), if you want to concatenate along an existing dimension + - partial(torch.stack, dim=new_dim), if you want to stack to a new dimension + + Args: + batch (List[Any]): A list of samples from the dataset. + aggregate_fn (Callable[[list[Any]], Any]): The function to aggregate the tensors/ndarray/float. + + Returns: + Any: The concatenated batch. + """ + if len(batch) == 0: + return batch + elem = batch[0] + if isinstance(elem, torch.Tensor) or isinstance(elem, np.ndarray) or isinstance(elem, float): + return aggregate_fn(batch) + elif isinstance(elem, dict): + return {key: aggregate_batch([d[key] for d in batch], aggregate_fn) for key in elem.keys()} + elif isinstance(elem, list): + return [aggregate_batch(samples, aggregate_fn) for samples in zip(*batch)] + elif elem is None: + if merge_none: + return None + else: + return batch diff --git a/cosmos_framework/data/vfm/action/umi/imagecodecs_numcodecs.py b/cosmos_framework/data/vfm/action/umi/imagecodecs_numcodecs.py new file mode 100644 index 0000000..5fb2f81 --- /dev/null +++ b/cosmos_framework/data/vfm/action/umi/imagecodecs_numcodecs.py @@ -0,0 +1,1344 @@ +# imagecodecs/numcodecs.py + +# Copyright (c) 2021-2022, Christoph Gohlke +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE +# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +"""Additional numcodecs implemented using imagecodecs.""" + +__version__ = "2022.9.26" + +__all__ = ("register_codecs",) + +import imagecodecs +import numpy +from numcodecs.abc import Codec +from numcodecs.registry import get_codec, register_codec + + +def protective_squeeze(x: numpy.ndarray): + """ + Squeeze dim only if it's not the last dim. + Image dim expected to be *, H, W, C + """ + img_shape = x.shape[-3:] + if len(x.shape) > 3: + n_imgs = numpy.prod(x.shape[:-3]) + if n_imgs > 1: + img_shape = (-1,) + img_shape + return x.reshape(img_shape) + + +def get_default_image_compressor(**kwargs): + if imagecodecs.JPEGXL: + # has JPEGXL + this_kwargs = { + "effort": 3, + "distance": 0.3, + # bug in libjxl, invalid codestream for non-lossless + # when decoding speed > 1 + "decodingspeed": 1, + } + this_kwargs.update(kwargs) + return JpegXl(**this_kwargs) + else: + this_kwargs = {"level": 50} + this_kwargs.update(kwargs) + return Jpeg2k(**this_kwargs) + + +class Aec(Codec): + """AEC codec for numcodecs.""" + + codec_id = "imagecodecs_aec" + + def __init__(self, bitspersample=None, flags=None, blocksize=None, rsi=None): + self.bitspersample = bitspersample + self.flags = flags + self.blocksize = blocksize + self.rsi = rsi + + def encode(self, buf): + return imagecodecs.aec_encode( + buf, + bitspersample=self.bitspersample, + flags=self.flags, + blocksize=self.blocksize, + rsi=self.rsi, + ) + + def decode(self, buf, out=None): + return imagecodecs.aec_decode( + buf, + bitspersample=self.bitspersample, + flags=self.flags, + blocksize=self.blocksize, + rsi=self.rsi, + out=_flat(out), + ) + + +class Apng(Codec): + """APNG codec for numcodecs.""" + + codec_id = "imagecodecs_apng" + + def __init__(self, level=None, photometric=None, delay=None): + self.level = level + self.photometric = photometric + self.delay = delay + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.apng_encode( + buf, + level=self.level, + photometric=self.photometric, + delay=self.delay, + ) + + def decode(self, buf, out=None): + return imagecodecs.apng_decode(buf, out=out) + + +class Avif(Codec): + """AVIF codec for numcodecs.""" + + codec_id = "imagecodecs_avif" + + def __init__( + self, + level=None, + speed=None, + tilelog2=None, + bitspersample=None, + pixelformat=None, + numthreads=None, + index=None, + ): + self.level = level + self.speed = speed + self.tilelog2 = tilelog2 + self.bitspersample = bitspersample + self.pixelformat = pixelformat + self.numthreads = numthreads + self.index = index + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.avif_encode( + buf, + level=self.level, + speed=self.speed, + tilelog2=self.tilelog2, + bitspersample=self.bitspersample, + pixelformat=self.pixelformat, + numthreads=self.numthreads, + ) + + def decode(self, buf, out=None): + return imagecodecs.avif_decode(buf, index=self.index, numthreads=self.numthreads, out=out) + + +class Bitorder(Codec): + """Bitorder codec for numcodecs.""" + + codec_id = "imagecodecs_bitorder" + + def encode(self, buf): + return imagecodecs.bitorder_encode(buf) + + def decode(self, buf, out=None): + return imagecodecs.bitorder_decode(buf, out=_flat(out)) + + +class Bitshuffle(Codec): + """Bitshuffle codec for numcodecs.""" + + codec_id = "imagecodecs_bitshuffle" + + def __init__(self, itemsize=1, blocksize=0): + self.itemsize = itemsize + self.blocksize = blocksize + + def encode(self, buf): + return imagecodecs.bitshuffle_encode(buf, itemsize=self.itemsize, blocksize=self.blocksize).tobytes() + + def decode(self, buf, out=None): + return imagecodecs.bitshuffle_decode( + buf, + itemsize=self.itemsize, + blocksize=self.blocksize, + out=_flat(out), + ) + + +class Blosc(Codec): + """Blosc codec for numcodecs.""" + + codec_id = "imagecodecs_blosc" + + def __init__( + self, + level=None, + compressor=None, + typesize=None, + blocksize=None, + shuffle=None, + numthreads=None, + ): + self.level = level + self.compressor = compressor + self.typesize = typesize + self.blocksize = blocksize + self.shuffle = shuffle + self.numthreads = numthreads + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.blosc_encode( + buf, + level=self.level, + compressor=self.compressor, + typesize=self.typesize, + blocksize=self.blocksize, + shuffle=self.shuffle, + numthreads=self.numthreads, + ) + + def decode(self, buf, out=None): + return imagecodecs.blosc_decode(buf, numthreads=self.numthreads, out=_flat(out)) + + +class Blosc2(Codec): + """Blosc2 codec for numcodecs.""" + + codec_id = "imagecodecs_blosc2" + + def __init__( + self, + level=None, + compressor=None, + typesize=None, + blocksize=None, + shuffle=None, + numthreads=None, + ): + self.level = level + self.compressor = compressor + self.typesize = typesize + self.blocksize = blocksize + self.shuffle = shuffle + self.numthreads = numthreads + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.blosc2_encode( + buf, + level=self.level, + compressor=self.compressor, + typesize=self.typesize, + blocksize=self.blocksize, + shuffle=self.shuffle, + numthreads=self.numthreads, + ) + + def decode(self, buf, out=None): + return imagecodecs.blosc2_decode(buf, numthreads=self.numthreads, out=_flat(out)) + + +class Brotli(Codec): + """Brotli codec for numcodecs.""" + + codec_id = "imagecodecs_brotli" + + def __init__(self, level=None, mode=None, lgwin=None): + self.level = level + self.mode = mode + self.lgwin = lgwin + + def encode(self, buf): + return imagecodecs.brotli_encode(buf, level=self.level, mode=self.mode, lgwin=self.lgwin) + + def decode(self, buf, out=None): + return imagecodecs.brotli_decode(buf, out=_flat(out)) + + +class ByteShuffle(Codec): + """ByteShuffle codec for numcodecs.""" + + codec_id = "imagecodecs_byteshuffle" + + def __init__(self, shape, dtype, axis=-1, dist=1, delta=False, reorder=False): + self.shape = tuple(shape) + self.dtype = numpy.dtype(dtype).str + self.axis = axis + self.dist = dist + self.delta = bool(delta) + self.reorder = bool(reorder) + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + assert buf.shape == self.shape + assert buf.dtype == self.dtype + return imagecodecs.byteshuffle_encode( + buf, + axis=self.axis, + dist=self.dist, + delta=self.delta, + reorder=self.reorder, + ).tobytes() + + def decode(self, buf, out=None): + if not isinstance(buf, numpy.ndarray): + buf = numpy.frombuffer(buf, dtype=self.dtype).reshape(*self.shape) + return imagecodecs.byteshuffle_decode( + buf, + axis=self.axis, + dist=self.dist, + delta=self.delta, + reorder=self.reorder, + out=out, + ) + + +class Bz2(Codec): + """Bz2 codec for numcodecs.""" + + codec_id = "imagecodecs_bz2" + + def __init__(self, level=None): + self.level = level + + def encode(self, buf): + return imagecodecs.bz2_encode(buf, level=self.level) + + def decode(self, buf, out=None): + return imagecodecs.bz2_decode(buf, out=_flat(out)) + + +class Cms(Codec): + """CMS codec for numcodecs.""" + + codec_id = "imagecodecs_cms" + + def __init__(self, *args, **kwargs): + pass + + def encode(self, buf, out=None): + # return imagecodecs.cms_transform(buf) + raise NotImplementedError + + def decode(self, buf, out=None): + # return imagecodecs.cms_transform(buf) + raise NotImplementedError + + +class Deflate(Codec): + """Deflate codec for numcodecs.""" + + codec_id = "imagecodecs_deflate" + + def __init__(self, level=None, raw=False): + self.level = level + self.raw = bool(raw) + + def encode(self, buf): + return imagecodecs.deflate_encode(buf, level=self.level, raw=self.raw) + + def decode(self, buf, out=None): + return imagecodecs.deflate_decode(buf, out=_flat(out), raw=self.raw) + + +class Delta(Codec): + """Delta codec for numcodecs.""" + + codec_id = "imagecodecs_delta" + + def __init__(self, shape=None, dtype=None, axis=-1, dist=1): + self.shape = None if shape is None else tuple(shape) + self.dtype = None if dtype is None else numpy.dtype(dtype).str + self.axis = axis + self.dist = dist + + def encode(self, buf): + if self.shape is not None or self.dtype is not None: + buf = protective_squeeze(numpy.asarray(buf)) + assert buf.shape == self.shape + assert buf.dtype == self.dtype + return imagecodecs.delta_encode(buf, axis=self.axis, dist=self.dist).tobytes() + + def decode(self, buf, out=None): + if self.shape is not None or self.dtype is not None: + buf = numpy.frombuffer(buf, dtype=self.dtype).reshape(*self.shape) + return imagecodecs.delta_decode(buf, axis=self.axis, dist=self.dist, out=out) + + +class Float24(Codec): + """Float24 codec for numcodecs.""" + + codec_id = "imagecodecs_float24" + + def __init__(self, byteorder=None, rounding=None): + self.byteorder = byteorder + self.rounding = rounding + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.float24_encode(buf, byteorder=self.byteorder, rounding=self.rounding) + + def decode(self, buf, out=None): + return imagecodecs.float24_decode(buf, byteorder=self.byteorder, out=out) + + +class FloatPred(Codec): + """Floating Point Predictor codec for numcodecs.""" + + codec_id = "imagecodecs_floatpred" + + def __init__(self, shape, dtype, axis=-1, dist=1): + self.shape = tuple(shape) + self.dtype = numpy.dtype(dtype).str + self.axis = axis + self.dist = dist + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + assert buf.shape == self.shape + assert buf.dtype == self.dtype + return imagecodecs.floatpred_encode(buf, axis=self.axis, dist=self.dist).tobytes() + + def decode(self, buf, out=None): + if not isinstance(buf, numpy.ndarray): + buf = numpy.frombuffer(buf, dtype=self.dtype).reshape(*self.shape) + return imagecodecs.floatpred_decode(buf, axis=self.axis, dist=self.dist, out=out) + + +class Gif(Codec): + """GIF codec for numcodecs.""" + + codec_id = "imagecodecs_gif" + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.gif_encode(buf) + + def decode(self, buf, out=None): + return imagecodecs.gif_decode(buf, asrgb=False, out=out) + + +class Heif(Codec): + """HEIF codec for numcodecs.""" + + codec_id = "imagecodecs_heif" + + def __init__( + self, + level=None, + bitspersample=None, + photometric=None, + compression=None, + numthreads=None, + index=None, + ): + self.level = level + self.bitspersample = bitspersample + self.photometric = photometric + self.compression = compression + self.numthreads = numthreads + self.index = index + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.heif_encode( + buf, + level=self.level, + bitspersample=self.bitspersample, + photometric=self.photometric, + compression=self.compression, + numthreads=self.numthreads, + ) + + def decode(self, buf, out=None): + return imagecodecs.heif_decode( + buf, + index=self.index, + photometric=self.photometric, + numthreads=self.numthreads, + out=out, + ) + + +class Jetraw(Codec): + """Jetraw codec for numcodecs.""" + + codec_id = "imagecodecs_jetraw" + + def __init__( + self, + shape, + identifier, + parameters=None, + verbosity=None, + errorbound=None, + ): + self.shape = shape + self.identifier = identifier + self.errorbound = errorbound + imagecodecs.jetraw_init(parameters, verbosity) + + def encode(self, buf): + return imagecodecs.jetraw_encode(buf, identifier=self.identifier, errorbound=self.errorbound) + + def decode(self, buf, out=None): + if out is None: + out = numpy.empty(self.shape, numpy.uint16) + return imagecodecs.jetraw_decode(buf, out=out) + + +class Jpeg(Codec): + """JPEG codec for numcodecs.""" + + codec_id = "imagecodecs_jpeg" + + def __init__( + self, + bitspersample=None, + tables=None, + header=None, + colorspace_data=None, + colorspace_jpeg=None, + level=None, + subsampling=None, + optimize=None, + smoothing=None, + ): + self.tables = tables + self.header = header + self.bitspersample = bitspersample + self.colorspace_data = colorspace_data + self.colorspace_jpeg = colorspace_jpeg + self.level = level + self.subsampling = subsampling + self.optimize = optimize + self.smoothing = smoothing + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.jpeg_encode( + buf, + level=self.level, + colorspace=self.colorspace_data, + outcolorspace=self.colorspace_jpeg, + subsampling=self.subsampling, + optimize=self.optimize, + smoothing=self.smoothing, + ) + + def decode(self, buf, out=None): + out_shape = None + if out is not None: + out_shape = out.shape + out = protective_squeeze(out) + img = imagecodecs.jpeg_decode( + buf, + bitspersample=self.bitspersample, + tables=self.tables, + header=self.header, + colorspace=self.colorspace_jpeg, + outcolorspace=self.colorspace_data, + out=out, + ) + if out_shape is not None: + img = img.reshape(out_shape) + return img + + def get_config(self): + """Return dictionary holding configuration parameters.""" + config = dict(id=self.codec_id) + for key in self.__dict__: + if not key.startswith("_"): + value = getattr(self, key) + if value is not None and key in ("header", "tables"): + import base64 + + value = base64.b64encode(value).decode() + config[key] = value + return config + + @classmethod + def from_config(cls, config): + """Instantiate codec from configuration object.""" + for key in ("header", "tables"): + value = config.get(key, None) + if value is not None and isinstance(value, str): + import base64 + + config[key] = base64.b64decode(value.encode()) + return cls(**config) + + +class Jpeg2k(Codec): + """JPEG 2000 codec for numcodecs.""" + + codec_id = "imagecodecs_jpeg2k" + + def __init__( + self, + level=None, + codecformat=None, + colorspace=None, + tile=None, + reversible=None, + bitspersample=None, + resolutions=None, + numthreads=None, + verbose=0, + ): + self.level = level + self.codecformat = codecformat + self.colorspace = colorspace + self.tile = None if tile is None else tuple(tile) + self.reversible = reversible + self.bitspersample = bitspersample + self.resolutions = resolutions + self.numthreads = numthreads + self.verbose = verbose + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.jpeg2k_encode( + buf, + level=self.level, + codecformat=self.codecformat, + colorspace=self.colorspace, + tile=self.tile, + reversible=self.reversible, + bitspersample=self.bitspersample, + resolutions=self.resolutions, + numthreads=self.numthreads, + verbose=self.verbose, + ) + + def decode(self, buf, out=None): + return imagecodecs.jpeg2k_decode(buf, verbose=self.verbose, numthreads=self.numthreads, out=out) + + +class JpegLs(Codec): + """JPEG LS codec for numcodecs.""" + + codec_id = "imagecodecs_jpegls" + + def __init__(self, level=None): + self.level = level + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.jpegls_encode(buf, level=self.level) + + def decode(self, buf, out=None): + return imagecodecs.jpegls_decode(buf, out=out) + + +class JpegXl(Codec): + """JPEG XL codec for numcodecs.""" + + codec_id = "imagecodecs_jpegxl" + + def __init__( + self, + # encode + level=None, + effort=None, + distance=None, + lossless=None, + decodingspeed=None, + photometric=None, + planar=None, + usecontainer=None, + # decode + index=None, + keeporientation=None, + # both + numthreads=None, + ): + """ + Return JPEG XL image from numpy array. + Float must be in nominal range 0..1. + + Currently L, LA, RGB, RGBA images are supported in contig mode. + Extra channels are only supported for grayscale images in planar mode. + + Parameters + ---------- + level : Default to None, i.e. not overwriting lossess and decodingspeed options. + When < 0: Use lossless compression + When in [0,1,2,3,4]: Sets the decoding speed tier for the provided options. + Minimum is 0 (slowest to decode, best quality/density), and maximum + is 4 (fastest to decode, at the cost of some quality/density). + effort : Default to 3. + Sets encoder effort/speed level without affecting decoding speed. + Valid values are, from faster to slower speed: 1:lightning 2:thunder + 3:falcon 4:cheetah 5:hare 6:wombat 7:squirrel 8:kitten 9:tortoise. + Speed: lightning, thunder, falcon, cheetah, hare, wombat, squirrel, kitten, tortoise + control the encoder effort in ascending order. + This also affects memory usage: using lower effort will typically reduce memory + consumption during encoding. + lightning and thunder are fast modes useful for lossless mode (modular). + falcon disables all of the following tools. + cheetah enables coefficient reordering, context clustering, and heuristics for selecting DCT sizes and quantization steps. + hare enables Gaborish filtering, chroma from luma, and an initial estimate of quantization steps. + wombat enables error diffusion quantization and full DCT size selection heuristics. + squirrel (default) enables dots, patches, and spline detection, and full context clustering. + kitten optimizes the adaptive quantization for a psychovisual metric. + tortoise enables a more thorough adaptive quantization search. + distance : Default to 1.0 + Sets the distance level for lossy compression: target max butteraugli distance, + lower = higher quality. Range: 0 .. 15. 0.0 = mathematically lossless + (however, use JxlEncoderSetFrameLossless instead to use true lossless, + as setting distance to 0 alone is not the only requirement). + 1.0 = visually lossless. Recommended range: 0.5 .. 3.0. + lossess : Default to False. + Use lossess encoding. + decodingspeed : Default to 0. + Duplicate to level. [0,4] + photometric : Return JxlColorSpace value. + Default logic is quite complicated but works most of the time. + Accepted value: + int: [-1,3] + str: ['RGB', + 'WHITEISZERO', 'MINISWHITE', + 'BLACKISZERO', 'MINISBLACK', 'GRAY', + 'XYB', 'KNOWN'] + planar : Enable multi-channel mode. + Default to false. + usecontainer : + Forces the encoder to use the box-based container format (BMFF) + even when not necessary. + When using JxlEncoderUseBoxes, JxlEncoderStoreJPEGMetadata or + JxlEncoderSetCodestreamLevel with level 10, the encoder will + automatically also use the container format, it is not necessary + to use JxlEncoderUseContainer for those use cases. + By default this setting is disabled. + index : Selectively decode frames for animation. + Default to 0, decode all frames. + When set to > 0, decode that frame index only. + keeporientation : + Enables or disables preserving of as-in-bitstream pixeldata orientation. + Some images are encoded with an Orientation tag indicating that the + decoder must perform a rotation and/or mirroring to the encoded image data. + + If skip_reorientation is JXL_FALSE (the default): the decoder will apply + the transformation from the orientation setting, hence rendering the image + according to its specified intent. When producing a JxlBasicInfo, the decoder + will always set the orientation field to JXL_ORIENT_IDENTITY (matching the + returned pixel data) and also align xsize and ysize so that they correspond + to the width and the height of the returned pixel data. + + If skip_reorientation is JXL_TRUE: the decoder will skip applying the + transformation from the orientation setting, returning the image in + the as-in-bitstream pixeldata orientation. This may be faster to decode + since the decoder doesnt have to apply the transformation, but can + cause wrong display of the image if the orientation tag is not correctly + taken into account by the user. + + By default, this option is disabled, and the returned pixel data is + re-oriented according to the images Orientation setting. + threads : Default to 1. + If <= 0, use all cores. + If > 32, clipped to 32. + """ + + self.level = level + self.effort = effort + self.distance = distance + self.lossless = bool(lossless) + self.decodingspeed = decodingspeed + self.photometric = photometric + self.planar = planar + self.usecontainer = usecontainer + self.index = index + self.keeporientation = keeporientation + self.numthreads = numthreads + + def encode(self, buf): + + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.jpegxl_encode( + buf, + level=self.level, + effort=self.effort, + distance=self.distance, + lossless=self.lossless, + decodingspeed=self.decodingspeed, + photometric=self.photometric, + planar=self.planar, + usecontainer=self.usecontainer, + numthreads=self.numthreads, + ) + + def decode(self, buf, out=None): + return imagecodecs.jpegxl_decode( + buf, + index=self.index, + keeporientation=self.keeporientation, + numthreads=self.numthreads, + out=out, + ) + + +class JpegXr(Codec): + """JPEG XR codec for numcodecs.""" + + codec_id = "imagecodecs_jpegxr" + + def __init__( + self, + level=None, + photometric=None, + hasalpha=None, + resolution=None, + fp2int=None, + ): + self.level = level + self.photometric = photometric + self.hasalpha = hasalpha + self.resolution = resolution + self.fp2int = fp2int + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.jpegxr_encode( + buf, + level=self.level, + photometric=self.photometric, + hasalpha=self.hasalpha, + resolution=self.resolution, + ) + + def decode(self, buf, out=None): + return imagecodecs.jpegxr_decode(buf, fp2int=self.fp2int, out=out) + + +class Lerc(Codec): + """LERC codec for numcodecs.""" + + codec_id = "imagecodecs_lerc" + + def __init__(self, level=None, version=None, planar=None): + self.level = level + self.version = version + self.planar = bool(planar) + + # self.mask = None + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.lerc_encode( + buf, + level=self.level, + version=self.version, + planar=self.planar, + ) + + def decode(self, buf, out=None): + return imagecodecs.lerc_decode(buf, out=out) + + +class Ljpeg(Codec): + """LJPEG codec for numcodecs.""" + + codec_id = "imagecodecs_ljpeg" + + def __init__(self, bitspersample=None): + self.bitspersample = bitspersample + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.ljpeg_encode(buf, bitspersample=self.bitspersample) + + def decode(self, buf, out=None): + return imagecodecs.ljpeg_decode(buf, out=out) + + +class Lz4(Codec): + """LZ4 codec for numcodecs.""" + + codec_id = "imagecodecs_lz4" + + def __init__(self, level=None, hc=False, header=True): + self.level = level + self.hc = hc + self.header = bool(header) + + def encode(self, buf): + return imagecodecs.lz4_encode(buf, level=self.level, hc=self.hc, header=self.header) + + def decode(self, buf, out=None): + return imagecodecs.lz4_decode(buf, header=self.header, out=_flat(out)) + + +class Lz4f(Codec): + """LZ4F codec for numcodecs.""" + + codec_id = "imagecodecs_lz4f" + + def __init__( + self, + level=None, + blocksizeid=False, + contentchecksum=None, + blockchecksum=None, + ): + self.level = level + self.blocksizeid = blocksizeid + self.contentchecksum = contentchecksum + self.blockchecksum = blockchecksum + + def encode(self, buf): + return imagecodecs.lz4f_encode( + buf, + level=self.level, + blocksizeid=self.blocksizeid, + contentchecksum=self.contentchecksum, + blockchecksum=self.blockchecksum, + ) + + def decode(self, buf, out=None): + return imagecodecs.lz4f_decode(buf, out=_flat(out)) + + +class Lzf(Codec): + """LZF codec for numcodecs.""" + + codec_id = "imagecodecs_lzf" + + def __init__(self, header=True): + self.header = bool(header) + + def encode(self, buf): + return imagecodecs.lzf_encode(buf, header=self.header) + + def decode(self, buf, out=None): + return imagecodecs.lzf_decode(buf, header=self.header, out=_flat(out)) + + +class Lzma(Codec): + """LZMA codec for numcodecs.""" + + codec_id = "imagecodecs_lzma" + + def __init__(self, level=None): + self.level = level + + def encode(self, buf): + return imagecodecs.lzma_encode(buf, level=self.level) + + def decode(self, buf, out=None): + return imagecodecs.lzma_decode(buf, out=_flat(out)) + + +class Lzw(Codec): + """LZW codec for numcodecs.""" + + codec_id = "imagecodecs_lzw" + + def encode(self, buf): + return imagecodecs.lzw_encode(buf) + + def decode(self, buf, out=None): + return imagecodecs.lzw_decode(buf, out=_flat(out)) + + +class PackBits(Codec): + """PackBits codec for numcodecs.""" + + codec_id = "imagecodecs_packbits" + + def __init__(self, axis=None): + self.axis = axis + + def encode(self, buf): + if not isinstance(buf, (bytes, bytearray)): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.packbits_encode(buf, axis=self.axis) + + def decode(self, buf, out=None): + return imagecodecs.packbits_decode(buf, out=_flat(out)) + + +class Pglz(Codec): + """PGLZ codec for numcodecs.""" + + codec_id = "imagecodecs_pglz" + + def __init__(self, header=True, strategy=None): + self.header = bool(header) + self.strategy = strategy + + def encode(self, buf): + return imagecodecs.pglz_encode(buf, strategy=self.strategy, header=self.header) + + def decode(self, buf, out=None): + return imagecodecs.pglz_decode(buf, header=self.header, out=_flat(out)) + + +class Png(Codec): + """PNG codec for numcodecs.""" + + codec_id = "imagecodecs_png" + + def __init__(self, level=None): + self.level = level + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.png_encode(buf, level=self.level) + + def decode(self, buf, out=None): + return imagecodecs.png_decode(buf, out=out) + + +class Qoi(Codec): + """QOI codec for numcodecs.""" + + codec_id = "imagecodecs_qoi" + + def __init__(self): + pass + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.qoi_encode(buf) + + def decode(self, buf, out=None): + return imagecodecs.qoi_decode(buf, out=out) + + +class Rgbe(Codec): + """RGBE codec for numcodecs.""" + + codec_id = "imagecodecs_rgbe" + + def __init__(self, header=False, shape=None, rle=None): + if not header and shape is None: + raise ValueError("must specify data shape if no header") + if shape and shape[-1] != 3: + raise ValueError("invalid shape") + self.shape = shape + self.header = bool(header) + self.rle = None if rle is None else bool(rle) + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.rgbe_encode(buf, header=self.header, rle=self.rle) + + def decode(self, buf, out=None): + if out is None and not self.header: + out = numpy.empty(self.shape, numpy.float32) + return imagecodecs.rgbe_decode(buf, header=self.header, rle=self.rle, out=out) + + +class Rcomp(Codec): + """Rcomp codec for numcodecs.""" + + codec_id = "imagecodecs_rcomp" + + def __init__(self, shape, dtype, nblock=None): + self.shape = tuple(shape) + self.dtype = numpy.dtype(dtype).str + self.nblock = nblock + + def encode(self, buf): + return imagecodecs.rcomp_encode(buf, nblock=self.nblock) + + def decode(self, buf, out=None): + return imagecodecs.rcomp_decode( + buf, + shape=self.shape, + dtype=self.dtype, + nblock=self.nblock, + out=out, + ) + + +class Snappy(Codec): + """Snappy codec for numcodecs.""" + + codec_id = "imagecodecs_snappy" + + def encode(self, buf): + return imagecodecs.snappy_encode(buf) + + def decode(self, buf, out=None): + return imagecodecs.snappy_decode(buf, out=_flat(out)) + + +class Spng(Codec): + """SPNG codec for numcodecs.""" + + codec_id = "imagecodecs_spng" + + def __init__(self, level=None): + self.level = level + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.spng_encode(buf, level=self.level) + + def decode(self, buf, out=None): + return imagecodecs.spng_decode(buf, out=out) + + +class Tiff(Codec): + """TIFF codec for numcodecs.""" + + codec_id = "imagecodecs_tiff" + + def __init__(self, index=None, asrgb=None, verbose=0): + self.index = index + self.asrgb = bool(asrgb) + self.verbose = verbose + + def encode(self, buf): + + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.tiff_encode(buf) + + def decode(self, buf, out=None): + return imagecodecs.tiff_decode( + buf, + index=self.index, + asrgb=self.asrgb, + verbose=self.verbose, + out=out, + ) + + +class Webp(Codec): + """WebP codec for numcodecs.""" + + codec_id = "imagecodecs_webp" + + def __init__(self, level=None, lossless=None, method=None, hasalpha=None): + self.level = level + self.hasalpha = bool(hasalpha) + self.method = method + self.lossless = lossless + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + return imagecodecs.webp_encode(buf, level=self.level, lossless=self.lossless, method=self.method) + + def decode(self, buf, out=None): + return imagecodecs.webp_decode(buf, hasalpha=self.hasalpha, out=out) + + +class Xor(Codec): + """XOR codec for numcodecs.""" + + codec_id = "imagecodecs_xor" + + def __init__(self, shape=None, dtype=None, axis=-1): + self.shape = None if shape is None else tuple(shape) + self.dtype = None if dtype is None else numpy.dtype(dtype).str + self.axis = axis + + def encode(self, buf): + if self.shape is not None or self.dtype is not None: + buf = protective_squeeze(numpy.asarray(buf)) + assert buf.shape == self.shape + assert buf.dtype == self.dtype + return imagecodecs.xor_encode(buf, axis=self.axis).tobytes() + + def decode(self, buf, out=None): + if self.shape is not None or self.dtype is not None: + buf = numpy.frombuffer(buf, dtype=self.dtype).reshape(*self.shape) + return imagecodecs.xor_decode(buf, axis=self.axis, out=_flat(out)) + + +class Zfp(Codec): + """ZFP codec for numcodecs.""" + + codec_id = "imagecodecs_zfp" + + def __init__( + self, + shape=None, + dtype=None, + strides=None, + level=None, + mode=None, + execution=None, + numthreads=None, + chunksize=None, + header=True, + ): + if header: + self.shape = None + self.dtype = None + self.strides = None + elif shape is None or dtype is None: + raise ValueError("invalid shape or dtype") + else: + self.shape = tuple(shape) + self.dtype = numpy.dtype(dtype).str + self.strides = None if strides is None else tuple(strides) + self.level = level + self.mode = mode + self.execution = execution + self.numthreads = numthreads + self.chunksize = chunksize + self.header = bool(header) + + def encode(self, buf): + buf = protective_squeeze(numpy.asarray(buf)) + if not self.header: + assert buf.shape == self.shape + assert buf.dtype == self.dtype + return imagecodecs.zfp_encode( + buf, + level=self.level, + mode=self.mode, + execution=self.execution, + header=self.header, + numthreads=self.numthreads, + chunksize=self.chunksize, + ) + + def decode(self, buf, out=None): + if self.header: + return imagecodecs.zfp_decode(buf, out=out) + return imagecodecs.zfp_decode( + buf, + shape=self.shape, + dtype=numpy.dtype(self.dtype), + strides=self.strides, + numthreads=self.numthreads, + out=out, + ) + + +class Zlib(Codec): + """Zlib codec for numcodecs.""" + + codec_id = "imagecodecs_zlib" + + def __init__(self, level=None): + self.level = level + + def encode(self, buf): + return imagecodecs.zlib_encode(buf, level=self.level) + + def decode(self, buf, out=None): + return imagecodecs.zlib_decode(buf, out=_flat(out)) + + +class Zlibng(Codec): + """Zlibng codec for numcodecs.""" + + codec_id = "imagecodecs_zlibng" + + def __init__(self, level=None): + self.level = level + + def encode(self, buf): + return imagecodecs.zlibng_encode(buf, level=self.level) + + def decode(self, buf, out=None): + return imagecodecs.zlibng_decode(buf, out=_flat(out)) + + +class Zopfli(Codec): + """Zopfli codec for numcodecs.""" + + codec_id = "imagecodecs_zopfli" + + def encode(self, buf): + return imagecodecs.zopfli_encode(buf) + + def decode(self, buf, out=None): + return imagecodecs.zopfli_decode(buf, out=_flat(out)) + + +class Zstd(Codec): + """ZStandard codec for numcodecs.""" + + codec_id = "imagecodecs_zstd" + + def __init__(self, level=None): + self.level = level + + def encode(self, buf): + return imagecodecs.zstd_encode(buf, level=self.level) + + def decode(self, buf, out=None): + return imagecodecs.zstd_decode(buf, out=_flat(out)) + + +def _flat(out): + """Return numpy array as contiguous view of bytes if possible.""" + if out is None: + return None + view = memoryview(out) + if view.readonly or not view.contiguous: + return None + return view.cast("B") + + +def register_codecs(codecs=None, force=False, verbose=True): + """Register codecs in this module with numcodecs.""" + for name, cls in globals().items(): + if not hasattr(cls, "codec_id") or name == "Codec": + continue + if codecs is not None and cls.codec_id not in codecs: + continue + try: + try: + get_codec({"id": cls.codec_id}) + except TypeError: + # registered, but failed + pass + except ValueError: + # not registered yet + pass + else: + if not force: + if verbose: + log_warning(f"numcodec {cls.codec_id!r} already registered") + continue + if verbose: + log_warning(f"replacing registered numcodec {cls.codec_id!r}") + register_codec(cls) + + +def log_warning(msg, *args, **kwargs): + """Log message with level WARNING.""" + import logging + + logging.getLogger(__name__).warning(msg, *args, **kwargs) diff --git a/cosmos_framework/data/vfm/action/umi/normalizer.py b/cosmos_framework/data/vfm/action/umi/normalizer.py new file mode 100644 index 0000000..bf27181 --- /dev/null +++ b/cosmos_framework/data/vfm/action/umi/normalizer.py @@ -0,0 +1,298 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Any + +import numpy as np +import torch +from torch import nn + +from cosmos_framework.data.vfm.action.umi.data_classes import DataMeta + + +class SingleFieldLinearNormalizer(nn.Module): + def __init__(self, meta: DataMeta): + super().__init__() + self.meta: DataMeta = meta + self.scale: nn.Parameter + self.offset: nn.Parameter + self.normalizer_type: str = meta.normalizer + + def fit(self, x: torch.Tensor): + raise NotImplementedError() + + def from_dict( + self, + state_dict: dict[str, torch.Tensor], + ): + if self.normalizer_type == "identity": + return + + state_dict_tensor: dict[str, torch.Tensor] = {} + for key, val in state_dict.items(): + if key not in ["scale", "offset", "min", "max"]: + continue + + if isinstance(val, torch.Tensor): + state_dict_tensor[key] = val + elif isinstance(val, np.ndarray): + state_dict_tensor[key] = torch.from_numpy(val) + elif isinstance(val, list): + state_dict_tensor[key] = torch.Tensor(val) + else: + raise ValueError(f"Unknown type {type(val)} for {key}") + + if "min" in state_dict_tensor and "max" in state_dict_tensor: + # Map [min, max] to [-1, 1] + # normalize: (x-offset) / scale + state_dict_tensor["scale"] = (state_dict_tensor["max"] - state_dict_tensor["min"]) / 2 + 1e-7 + state_dict_tensor["offset"] = (state_dict_tensor["max"] + state_dict_tensor["min"]) / 2 + del state_dict_tensor["min"] + del state_dict_tensor["max"] + + # Set scale to 1 if the original scale is too small + if hasattr(self, "skip_threshold"): + scale_too_small_mask = state_dict_tensor["scale"] < self.skip_threshold + state_dict_tensor["scale"][scale_too_small_mask] = 1.0 + + keys = ["scale", "offset"] + + for key in keys: + val = state_dict_tensor[key] + assert key in state_dict_tensor, f"State dict must contain '{key}' key for {self.meta.name}" + + if self.normalizer_type in ["range", "normal", "clamped_range"]: + assert val.shape == self.meta.shape, ( + f"{key} must have the same shape as the data {self.meta.shape} for range normalizer {self.meta.name}" + ) + else: + raise ValueError( + f"Unknown normalizer {self.normalizer_type} for {self.meta.name}. Valid normalizers are 'identity', 'range', 'normal', 'clamped_range'." + ) + + setattr(self, key, nn.Parameter(val)) + + def normalize(self, x: torch.Tensor) -> torch.Tensor: + raise NotImplementedError() + + def unnormalize(self, x: torch.Tensor) -> torch.Tensor: + raise NotImplementedError() + + def as_dict(self, data_class: str) -> dict[str, Any]: + assert self.scale is not None and self.offset is not None, ( + f"Normalizer for {self.meta.name} is not initialized." + ) + if data_class == "numpy": + return { + "type": self.normalizer_type, + "scale": self.scale.detach().cpu().numpy(), + "offset": self.offset.detach().cpu().numpy(), + } + elif data_class == "torch": + return { + "type": self.normalizer_type, + "scale": self.scale.detach().cpu(), + "offset": self.offset.detach().cpu(), + } + elif data_class == "list": + return { + "type": self.normalizer_type, + "scale": self.scale.detach().cpu().tolist(), + "offset": self.offset.detach().cpu().tolist(), + } + else: + raise ValueError( + f"Unknown data type {data_class} for normalizer {self.meta.name}. Valid types are 'numpy', 'torch', and 'list'." + ) + + def _check_input_shape(self, x: torch.Tensor): + data_dim = len(self.meta.shape) + assert x.shape[-data_dim:] == self.meta.shape, ( + f"The last {data_dim} dimensions of {self.meta.name} (shape {x.shape}) must match {self.meta.shape} from meta data" + ) + + +class IdentityNormalizer(SingleFieldLinearNormalizer): + def __init__(self, meta: DataMeta): + super().__init__(meta) + self.scale = nn.Parameter(torch.tensor(1.0)) + self.offset = nn.Parameter(torch.tensor(0.0)) + + def fit(self, x: torch.Tensor): + pass + + def load(self, state_dict: dict[str, torch.Tensor]): + pass + + def normalize(self, x: torch.Tensor) -> torch.Tensor: + return x + + def unnormalize(self, x: torch.Tensor) -> torch.Tensor: + return x + + +class RangeNormalizer(SingleFieldLinearNormalizer): + """ + Normalize data to be between -1 and 1. + """ + + def __init__(self, meta: DataMeta, skip_threshold: float = 1e-2): + super().__init__(meta) + self.scale = nn.Parameter(torch.nan * torch.ones(meta.shape)) + self.offset = nn.Parameter(torch.nan * torch.ones(meta.shape)) + self.skip_threshold = skip_threshold + + def fit(self, x: torch.Tensor): + """ + x: (traj_num, *shape) + """ + self._check_input_shape(x) + x = x.clone().detach().reshape(-1, *self.meta.shape) + min_val = x.min(dim=0).values + max_val = x.max(dim=0).values + scale = nn.Parameter((max_val - min_val) / 2 + 1e-7) + self.scale[:] = 1.0 + max_abs_val = torch.max(torch.abs(x), dim=0).values + ratio = scale / max_abs_val + normalize_mask = (max_abs_val > self.skip_threshold) & (ratio > self.skip_threshold) + print(f"{self.meta.name}: Normalize mask {normalize_mask}") + self.scale[normalize_mask] = scale[normalize_mask] + self.offset = nn.Parameter((max_val + min_val) / 2) + assert tuple(self.scale.shape) == tuple(self.offset.shape) == self.meta.shape + + def normalize(self, x: torch.Tensor) -> torch.Tensor: + assert not self.scale.isnan().any() and not self.offset.isnan().any(), ( + f"Normalizer for {self.meta.name} is not initialized" + ) + self._check_input_shape(x) + return (x - self.offset) / self.scale + + def unnormalize(self, x: torch.Tensor) -> torch.Tensor: + assert not self.scale.isnan().any() and not self.offset.isnan().any(), ( + f"Normalizer for {self.meta.name} is not initialized" + ) + self._check_input_shape(x) + return x * self.scale + self.offset + + +class ClampedRangeNormalizer(SingleFieldLinearNormalizer): + """ + Normalize data to be between -1 and 1, but clip the data if the normalized value is out of range. + """ + + def __init__(self, meta: DataMeta): + super().__init__(meta) + self.scale = nn.Parameter(torch.nan * torch.ones(meta.shape)) + self.offset = nn.Parameter(torch.nan * torch.ones(meta.shape)) + + def fit(self, x: torch.Tensor): + raise NotImplementedError("Please manually assign the scale and offset parameters after calculating the stats") + + def normalize(self, x: torch.Tensor) -> torch.Tensor: + assert not self.scale.isnan().any() and not self.offset.isnan().any(), ( + f"Normalizer for {self.meta.name} is not initialized" + ) + self._check_input_shape(x) + return torch.clamp((x - self.offset) / self.scale, -1, 1) + + def unnormalize(self, x: torch.Tensor) -> torch.Tensor: + assert not self.scale.isnan().any() and not self.offset.isnan().any(), ( + f"Normalizer for {self.meta.name} is not initialized" + ) + self._check_input_shape(x) + return x * self.scale + self.offset + + +class NormalNormalizer(RangeNormalizer): + """ + Normalize data distribution to be N(0, 1). + """ + + def fit(self, x: torch.Tensor): + print(f"Fitting normalizer for {self.meta.name}") + self._check_input_shape(x) + x = x.clone().detach().reshape(-1, *self.meta.shape) + mean = x.mean(dim=0) + std = x.std(dim=0) + self.scale = nn.Parameter(std) + self.offset = nn.Parameter(mean) + + +class FixedNormalizer(nn.Module): + """ + Normalizer that is fixed after fitting. Not trainable. + """ + + @torch.no_grad() + def __init__( + self, + data_meta: dict[str, DataMeta], + ): + super().__init__() + self.data_meta: dict[str, DataMeta] = data_meta + self.normalizers: nn.ModuleDict = nn.ModuleDict() + + for meta in self.data_meta.values(): + if meta.normalizer == "identity": + self.normalizers[meta.name] = IdentityNormalizer(meta) + elif meta.normalizer == "range": + self.normalizers[meta.name] = RangeNormalizer(meta) + elif meta.normalizer == "normal": + self.normalizers[meta.name] = NormalNormalizer(meta) + elif meta.normalizer == "clamped_range": + self.normalizers[meta.name] = ClampedRangeNormalizer(meta) + else: + raise ValueError(f"Unknown normalizer {meta.normalizer} for {meta.name}") + + @torch.no_grad() + def fit_normalizer(self, data_dict: dict[str, torch.Tensor]): + for meta in self.data_meta.values(): + if meta.normalizer not in ["range", "normal", "clamped_range"]: + continue + normalizer = self.normalizers[meta.name] + assert isinstance(normalizer, SingleFieldLinearNormalizer) + normalizer.fit(data_dict[meta.name]) + + @torch.no_grad() + def from_dict( + self, + state_dict: dict[str, dict[str, torch.Tensor]], + ): + for meta in self.data_meta.values(): + if meta.normalizer == "identity": + continue + assert meta.name in state_dict, f"State dict for {meta.name} not found when loading normalizer" + normalizer = self.normalizers[meta.name] + assert isinstance(normalizer, SingleFieldLinearNormalizer) + normalizer.from_dict(state_dict[meta.name]) + + @torch.no_grad() + def normalize(self, data_dict: dict[str, torch.Tensor]) -> dict[str, torch.Tensor]: + for name, data in data_dict.items(): + if name not in self.normalizers: + continue + normalizer = self.normalizers[name] + assert isinstance(normalizer, SingleFieldLinearNormalizer) + data_dict[name] = normalizer.normalize(data) + + return data_dict + + @torch.no_grad() + def unnormalize(self, data_dict: dict[str, torch.Tensor]) -> dict[str, torch.Tensor]: + for name, data in data_dict.items(): + if name not in self.normalizers: + continue + normalizer = self.normalizers[name] + assert isinstance(normalizer, SingleFieldLinearNormalizer) + data_dict[name] = normalizer.unnormalize(data) + return data_dict + + @torch.no_grad() + def as_dict(self, data_class: str) -> dict[str, dict[str, Any]]: + state_dict = {} + for name, normalizer in self.normalizers.items(): + if name not in state_dict: + state_dict[name] = {} + assert isinstance(normalizer, SingleFieldLinearNormalizer) + state_dict[name] = normalizer.as_dict(data_class) + return state_dict diff --git a/cosmos_framework/data/vfm/action/umi/pose_utils.py b/cosmos_framework/data/vfm/action/umi/pose_utils.py new file mode 100644 index 0000000..d244220 --- /dev/null +++ b/cosmos_framework/data/vfm/action/umi/pose_utils.py @@ -0,0 +1,286 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Any + +import numpy as np +import numpy.typing as npt +from numba import njit + + +@njit(cache=True) +def qmult(q1: npt.NDArray[Any], q2: npt.NDArray[Any]) -> npt.NDArray[Any]: + q = np.array( + [ + q1[0] * q2[0] - q1[1] * q2[1] - q1[2] * q2[2] - q1[3] * q2[3], + q1[0] * q2[1] + q1[1] * q2[0] + q1[2] * q2[3] - q1[3] * q2[2], + q1[0] * q2[2] - q1[1] * q2[3] + q1[2] * q2[0] + q1[3] * q2[1], + q1[0] * q2[3] + q1[1] * q2[2] - q1[2] * q2[1] + q1[3] * q2[0], + ] + ) + + return q + + +@njit(cache=True) +def qconjugate(q: npt.NDArray[Any]) -> npt.NDArray[Any]: + return np.array([q[0], -q[1], -q[2], -q[3]]) + + +@njit(cache=True) +def get_absolute_pose( + init_pose_xyz_wxyz: npt.NDArray[Any], + relative_pose_xyz_wxyz: npt.NDArray[Any], +): + """The new pose is in the same frame of reference as the initial pose""" + new_pose_xyz_wxyz = np.zeros(7, init_pose_xyz_wxyz.dtype) + relative_pos_in_init_frame_as_quat_wxyz = np.zeros(4, init_pose_xyz_wxyz.dtype) + relative_pos_in_init_frame_as_quat_wxyz[1:] = relative_pose_xyz_wxyz[:3] + init_rot_qinv = qconjugate(init_pose_xyz_wxyz[3:]) + relative_pos_in_world_frame_as_quat_wxyz = qmult( + qmult(init_pose_xyz_wxyz[3:], relative_pos_in_init_frame_as_quat_wxyz), + init_rot_qinv, + ) + new_pose_xyz_wxyz[:3] = init_pose_xyz_wxyz[:3] + relative_pos_in_world_frame_as_quat_wxyz[1:] + quat = qmult(init_pose_xyz_wxyz[3:], relative_pose_xyz_wxyz[3:]) + if quat[0] < 0: + quat = -quat + new_pose_xyz_wxyz[3:] = quat + return new_pose_xyz_wxyz + + +@njit(cache=True) +def get_relative_pose( + new_pose_xyz_wxyz: npt.NDArray[Any], + init_pose_xyz_wxyz: npt.NDArray[Any], +): + """The two poses are in the same frame of reference""" + relative_pose_xyz_wxyz = np.zeros(7, new_pose_xyz_wxyz.dtype) + relative_pos_in_world_frame_as_quat_wxyz = np.zeros(4, new_pose_xyz_wxyz.dtype) + relative_pos_in_world_frame_as_quat_wxyz[1:] = new_pose_xyz_wxyz[:3] - init_pose_xyz_wxyz[:3] + init_rot_qinv = qconjugate(init_pose_xyz_wxyz[3:]) + relative_pose_xyz_wxyz[:3] = qmult( + qmult(init_rot_qinv, relative_pos_in_world_frame_as_quat_wxyz), + init_pose_xyz_wxyz[3:], + )[1:] + quat = qmult(init_rot_qinv, new_pose_xyz_wxyz[3:]) + if quat[0] < 0: + quat = -quat + relative_pose_xyz_wxyz[3:] = quat + return relative_pose_xyz_wxyz + + +@njit(cache=True) +def invert_pose(pose_xyz_wxyz: npt.NDArray[Any]) -> npt.NDArray[Any]: + qinv = qconjugate(pose_xyz_wxyz[3:]) + pos_quat_wxyz = np.zeros(4, pose_xyz_wxyz.dtype) + pos_quat_wxyz[1:] = pose_xyz_wxyz[:3] + rotated_pos = qmult( + qmult(qinv, pos_quat_wxyz), + pose_xyz_wxyz[3:], + ) + inverted_pose = np.zeros(7, pose_xyz_wxyz.dtype) + inverted_pose[:3] = -rotated_pos[1:] + if qinv[0] < 0: + qinv = -qinv + inverted_pose[3:] = qinv + return inverted_pose + + +@njit(cache=True) +def quat_wxyz_to_rot_6d(quat_wxyz: npt.NDArray[Any]) -> npt.NDArray[Any]: + """ + Convert a quaternion to a 6D representation: the first two rows of the corresponding rotation matrix. + https://arxiv.org/pdf/1812.07035 + quat_wxyz: (4, ) + return: (6, ) + """ + assert quat_wxyz.shape == (4,) + w, x, y, z = quat_wxyz[0], quat_wxyz[1], quat_wxyz[2], quat_wxyz[3] + + R = np.array( + [ + [1 - 2 * y * y - 2 * z * z, 2 * x * y - 2 * w * z, 2 * x * z + 2 * w * y], + [2 * x * y + 2 * w * z, 1 - 2 * x * x - 2 * z * z, 2 * y * z - 2 * w * x], + [2 * x * z - 2 * w * y, 2 * y * z + 2 * w * x, 1 - 2 * x * x - 2 * y * y], + ] + ) + + rot_6d = np.zeros(6) + rot_6d[:3] = R[0, :] + rot_6d[3:] = R[1, :] + + return rot_6d + + +@njit(cache=True) +def rot_6d_to_quat_wxyz(rot_6d: npt.NDArray[Any]) -> npt.NDArray[Any]: + """ + Convert a 6D representation to a quaternion. + https://arxiv.org/pdf/1812.07035 + rot_6d: (6, ) + return: (4, ) + """ + + assert rot_6d.shape == (6,) + a1, a2 = rot_6d[:3], rot_6d[3:] + b1 = a1 / np.linalg.norm(a1) + b2 = a2 - np.dot(b1, a2) * b1 + b2 = b2 / np.linalg.norm(b2) + b3 = np.cross(b1, b2) + + m = np.zeros((3, 3)) + m[0, :] = b1 + m[1, :] = b2 + m[2, :] = b3 + + trace = np.trace(m) + + if trace > 0: + s = 0.5 / np.sqrt(trace + 1.0) + w = 0.25 / s + x = (m[2, 1] - m[1, 2]) * s + y = (m[0, 2] - m[2, 0]) * s + z = (m[1, 0] - m[0, 1]) * s + elif m[0, 0] > m[1, 1] and m[0, 0] > m[2, 2]: + s = 2.0 * np.sqrt(1.0 + m[0, 0] - m[1, 1] - m[2, 2]) + w = (m[2, 1] - m[1, 2]) / s + x = 0.25 * s + y = (m[0, 1] + m[1, 0]) / s + z = (m[0, 2] + m[2, 0]) / s + elif m[1, 1] > m[2, 2]: + s = 2.0 * np.sqrt(1.0 + m[1, 1] - m[0, 0] - m[2, 2]) + w = (m[0, 2] - m[2, 0]) / s + x = (m[0, 1] + m[1, 0]) / s + y = 0.25 * s + z = (m[1, 2] + m[2, 1]) / s + else: + s = 2.0 * np.sqrt(1.0 + m[2, 2] - m[0, 0] - m[1, 1]) + w = (m[1, 0] - m[0, 1]) / s + x = (m[0, 2] + m[2, 0]) / s + y = (m[1, 2] + m[2, 1]) / s + z = 0.25 * s + + return np.array([w, x, y, z]) + + +# @njit(cache=True) +def quat_wxyz_to_rot_6d_batch(quat_wxyz: npt.NDArray[Any]) -> npt.NDArray[Any]: + """ + input (..., 4) + output (..., 6) + """ + assert quat_wxyz.shape[-1] == 4 + input_shape = quat_wxyz.shape[:-1] + quat_wxyz = quat_wxyz.copy().reshape(-1, 4) + rot_6d = np.zeros((quat_wxyz.shape[0], 6)) + for i in range(quat_wxyz.shape[0]): + rot_6d[i] = quat_wxyz_to_rot_6d(quat_wxyz[i]) + return rot_6d.reshape(*input_shape, 6) + + +# @njit(cache=True) +def rot_6d_to_quat_wxyz_batch(rot_6d: npt.NDArray[Any]) -> npt.NDArray[Any]: + """ + input (..., 6) + output (..., 4) + """ + assert rot_6d.shape[-1] == 6, f"rot_6d.shape: {rot_6d.shape}" + input_shape = rot_6d.shape[:-1] + rot_6d = rot_6d.copy().reshape(-1, 6) + quat_wxyz = np.zeros((rot_6d.shape[0], 4)) + for i in range(rot_6d.shape[0]): + quat_wxyz[i] = rot_6d_to_quat_wxyz(rot_6d[i]) + return quat_wxyz.reshape(*input_shape, 4) + + +def convert_batch_to_10d(eef_xyz_wxyz: npt.NDArray[np.float32], gripper_width: npt.NDArray[np.float32]): + """ + eef_xyz_wxyz: (batch_size, obs_history_len, 7) + gripper_width: (batch_size, obs_history_len, 1) + + return: + robot0_10d: (batch_size, obs_history_len, 10) + """ + + assert eef_xyz_wxyz.shape[0:2] == gripper_width.shape[0:2] + + batch_size, obs_history_len = eef_xyz_wxyz.shape[0:2] + + pose10d = np.zeros((batch_size, obs_history_len, 10)) + pose10d[:, :, :3] = eef_xyz_wxyz[:, :, :3] + pose10d[:, :, 3:9] = quat_wxyz_to_rot_6d_batch(eef_xyz_wxyz[:, :, 3:]) + pose10d[:, :, 9] = gripper_width[:, :, 0] + + return pose10d + + +def convert_10d_to_batch(pose10d: npt.NDArray[np.float32]): + """ + pose10d: (batch_size, obs_history_len, 10) + + return: + eef_xyz_wxyz: (batch_size, obs_history_len, 7) + gripper_width: (batch_size, obs_history_len, 1) + """ + batch_size, obs_history_len = pose10d.shape[0:2] + eef_xyz_wxyz = np.zeros((batch_size, obs_history_len, 7), dtype=np.float32) + eef_xyz_wxyz[:, :, :3] = pose10d[:, :, :3] + eef_xyz_wxyz[:, :, 3:] = rot_6d_to_quat_wxyz_batch(pose10d[:, :, 3:9]) + gripper_width = pose10d[:, :, 9:] + + return eef_xyz_wxyz, gripper_width + + +# def pos_axang_to_mat(position: npt.NDArray[Any], axis_angle: npt.NDArray[Any]) -> npt.NDArray[Any]: +# """ +# Convert a position and axis-angle to a 4x4 matrix. +# position: (3, ) or (N, 3) +# axis_angle: (3, ) or (N, 3) +# return: (4, 4) or (N, 4, 4) +# """ +# pass + +# def mat_to_pose10d(mat: npt.NDArray[Any]) -> npt.NDArray[Any]: +# """ +# Convert a 4x4 matrix to a 10D pose. +# mat: (4, 4) or (N, 4, 4) +# return: (10, ) or (N, 10) +# """ +# if len(mat.shape) == 3: +# pose10d = np.zeros((mat.shape[0], 10)) +# pose10d[:, :3] = mat[:, :3, 3] +# pose10d[:, 3:] = quat_wxyz_to_rot_6d(mat[:, :3, :3]) +# else: +# pose10d = np.zeros(10) +# pose10d[:3] = mat[:3, 3] +# pose10d[3:] = quat_wxyz_to_rot_6d(mat[:3, :3]) + + +if __name__ == "__main__": + pose_1 = np.random.rand(7) + pose_1[3:] /= np.linalg.norm(pose_1[3:]) + pose_2 = np.random.rand(7) + pose_2[3:] /= np.linalg.norm(pose_2[3:]) + + pose_2_relative_to_pose_1 = get_relative_pose(pose_2, pose_1) + absolute_pose_2 = get_absolute_pose(pose_1, pose_2_relative_to_pose_1) + assert np.allclose(pose_2, absolute_pose_2) + pose_1_relative_to_pose_2 = get_relative_pose(pose_1, pose_2) + absolute_pose_1 = get_absolute_pose(pose_2, pose_1_relative_to_pose_2) + assert np.allclose(pose_1, absolute_pose_1) + inverted_pose_2_relative_to_pose_1 = invert_pose(pose_2_relative_to_pose_1) + assert np.allclose(inverted_pose_2_relative_to_pose_1, pose_1_relative_to_pose_2) + + assert np.allclose(pose_1, invert_pose(invert_pose(pose_1))) + assert np.allclose(pose_2, invert_pose(invert_pose(pose_2))) + + rot_6d = quat_wxyz_to_rot_6d(pose_1[3:]) + quat_wxyz = rot_6d_to_quat_wxyz(rot_6d) + assert np.allclose(quat_wxyz, pose_1[3:]) + + rot_6d = quat_wxyz_to_rot_6d(pose_2[3:]) + quat_wxyz = rot_6d_to_quat_wxyz(rot_6d) + assert np.allclose(quat_wxyz, pose_2[3:]) + + print("Test passed") diff --git a/cosmos_framework/data/vfm/action/umi/transforms.py b/cosmos_framework/data/vfm/action/umi/transforms.py new file mode 100644 index 0000000..f25a950 --- /dev/null +++ b/cosmos_framework/data/vfm/action/umi/transforms.py @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Any, Union + +import kornia.augmentation as K +import torch +import torch.nn as nn +from einops import rearrange + +from cosmos_framework.data.vfm.action.umi.data_classes import DataMeta + + +class BaseTransforms: + def __init__(self, data_meta: dict[str, DataMeta], apply_image_augmentation_in_cpu: bool): + self.transforms: dict[str, Union[K.VideoSequential, nn.Sequential]] = {} + self.data_meta: dict[str, DataMeta] = data_meta + self.apply_image_augmentation_in_cpu: bool = apply_image_augmentation_in_cpu + for entry_meta in data_meta.values(): + transforms_list = [] + for aug_cfg in entry_meta.augmentation: + aug_name = aug_cfg["name"] + aug_cfg.pop("name") + if entry_meta.data_type == "image": + if aug_name in K.__dict__: + transform_cls = K.__dict__[aug_name] + else: + raise ValueError(f"Augmentation {aug_name} not found in kornia.augmentation.") + elif entry_meta.data_type == "low_dim": + raise ValueError( + f"Augmentation {aug_name} not found in low dim transforms. Please implement your own augmentation method." + ) + + transforms_list.append(transform_cls(**aug_cfg)) + if len(transforms_list) > 0: + if entry_meta.data_type == "image": + self.transforms[entry_meta.name] = K.VideoSequential(*transforms_list) + else: + self.transforms[entry_meta.name] = nn.Sequential(*transforms_list) + + def to(self, device: Union[torch.device, str]): + for transform in self.transforms.values(): + transform.to(device) + + def apply(self, data_dict: dict[str, Any], consistent_on_batch: bool = False) -> dict[str, torch.Tensor]: + for name, data in data_dict.items(): + if not self.apply_image_augmentation_in_cpu and self.data_meta[name].data_type == "image": + continue + if isinstance(data, dict): + data_dict[name] = self.apply(data, consistent_on_batch) + elif isinstance(data, torch.Tensor): + if name in self.transforms: + batch_size, traj_len, *shape = data.shape + if consistent_on_batch: + data = data.reshape(1, batch_size * traj_len, *shape) + + data_dim_num = len(self.data_meta[name].shape) + new_data_dim_num = len(data.shape) + squeeze_data = False + if new_data_dim_num - data_dim_num == 1: + data = data.unsqueeze(0) + squeeze_data = True + elif new_data_dim_num - data_dim_num != 2: + raise ValueError( + f"Data {name} has more than 2 additional dimensions: {data.shape}. Currently only support (traj_len, *shape) or (batch_size, traj_len, *shape)." + ) + try: + data = self.transforms[name](data) + except Exception as e: + print(f"Error applying transform {name} to data {data.shape}: {e}") + raise e + if squeeze_data: + data = data.squeeze(0) + if consistent_on_batch: + data = rearrange(data, "1 (b t) ... -> b t ...", b=batch_size, t=traj_len) # [B,T,...] + data_dict[name] = data + + else: + raise ValueError(f"Unknown data type {type(data)} for {name}") + return data_dict diff --git a/cosmos_framework/data/vfm/action/umi_dataset.py b/cosmos_framework/data/vfm/action/umi_dataset.py new file mode 100644 index 0000000..9e05850 --- /dev/null +++ b/cosmos_framework/data/vfm/action/umi_dataset.py @@ -0,0 +1,1226 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import copy +import json +import multiprocessing +import os +import re +import sys +from typing import Any, Literal, Optional, Union, cast + +import numpy as np +import numpy.typing as npt +import torch +import tqdm +import zarr +from omegaconf import DictConfig, OmegaConf +from torch.utils.data import Dataset + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.umi.data_classes import ( + DataMeta, + SourceDataMeta, + construct_data_meta, + construct_source_data_meta, +) +from cosmos_framework.data.vfm.action.umi.data_utils import aggregate_batch +from cosmos_framework.data.vfm.action.umi.imagecodecs_numcodecs import register_codecs +from cosmos_framework.data.vfm.action.umi.normalizer import FixedNormalizer +from cosmos_framework.data.vfm.action.umi.transforms import BaseTransforms + +_ACTION_DIR = os.path.dirname(os.path.abspath(__file__)) +_BUNDLED_NORMALIZERS_DIR = os.path.join(_ACTION_DIR, "normalizers") + + +def _resolve_normalizer_path(normalizer_dir: str, dataset_name: str) -> str | None: + """Resolve normalizer JSON path from bundled files shipped with the repo.""" + if normalizer_dir.endswith(".json"): + filename = os.path.basename(normalizer_dir) + else: + filename = f"{dataset_name}_normalizer.json" + path = os.path.join(_BUNDLED_NORMALIZERS_DIR, filename) + return path if os.path.exists(path) else None + + +register_codecs() +import random + +import hydra + +from cosmos_framework.data.vfm.action.domain_utils import get_domain_id +from cosmos_framework.data.vfm.action.pose_utils import build_abs_pose_from_components, pose_abs_to_rel + + +class BaseDataset(Dataset[dict[str, torch.Tensor]]): + """ + Base class for all datasets. + """ + + def __init__( + self, + root: str, + # The folder that contains all the zarr stores + # data path should be: root/name/episode_data.zarr or root/name.zarr + name: str, + robot_num: int, + # compressed_dir: str, # The folder that contains the lz4 compressed data (compressed_dir/name.lz4) + # If dataset is not found in root, the program will extract the data from compressed_dir + include_episode_num: int, + include_episode_indices: list[int], + used_episode_ratio: float, + index_pool_size_per_episode: int, + history_padding_length: int, + future_padding_length: int, + seed: int, + source_data_meta: Union[dict[str, dict[str, Any]], DictConfig], + output_data_meta: Union[dict[str, dict[str, Any]], DictConfig], + starting_percentile_max: float, + starting_percentile_min: float, + apply_image_augmentation_in_cpu: bool, + use_relative_pose: bool, + relative_pose_mode: Literal["frame_wise", "anchored"], + use_relative_gripper_width: bool, + normalizer_sample_num: int, + normalizer_dir: str, + repeat_dataset_num: int, + mode: str, + image_size: int, + use_grasp_state_repr: bool, + eef_z_offset: float, + use_eef_pose: bool, + **unused_kwargs, + ): + log.info(f"BaseDataset unused_kwargs: {unused_kwargs}") + + if not use_grasp_state_repr: + output_data_meta.pop("grasp_states") + + if "eef_poses" not in output_data_meta: + raise ValueError("eef_poses is not in output_data_meta") + + self.use_eef_pose: bool = use_eef_pose + + if name.endswith(".zarr"): + name = name.replace(".zarr", "") + + if "/" in name: + # HACK: For re-organized UMI datasets + assert len(name.split("/")) == 2, f"name {name} should be in the format of 'project_name/dataset_name'" + data_project_name = name.split("/")[0] + name = name.split("/")[1] + root = os.path.join(root, data_project_name) + # compressed_dir = os.path.join(compressed_dir, data_project_name) + assert normalizer_dir == "" or normalizer_dir.endswith(".json"), ( + f"normalizer_dir {normalizer_dir} should be a json file or empty" + ) + + # os.makedirs(root, exist_ok=True) + + assert isinstance(robot_num, int) and robot_num >= 1, ( + f"robot_num must be an integer greater than 0, but got {robot_num}." + ) + self.robot_num: int = robot_num + + # if not torch.cuda.is_available() or torch.cuda.current_device() == 0: + # if os.path.exists(os.path.join(root, name, "episode_data.zarr")): + # zarr_path = os.path.join(root, name, "episode_data.zarr") + # elif os.path.exists(os.path.join(root, name + ".zarr")): + # zarr_path = os.path.join(root, name + ".zarr") + # elif os.path.exists(os.path.join(root, name)): + # zarr_path = os.path.join(root, name) + # else: + # print(f"Dataset {name} not found in {root}") + # print( + # f"Checked paths: {os.path.join(root, name, 'episode_data.zarr')}, {os.path.join(root, name + '.zarr')}, {os.path.join(root, name)}" + # ) + + # lz4_path = os.path.join(compressed_dir, name + ".tar.lz4") + # lz4_zarr_path = os.path.join(compressed_dir, name + ".zarr.tar.lz4") + # if os.path.exists(lz4_path): + # print(f"Extracting dataset {name} from {compressed_dir} to {root}") + # subprocess.run( + # [f"lz4 -d -c {lz4_path} | tar xf - -C {root}"], + # cwd=root, + # shell=True, + # check=True, + # ) + # elif os.path.exists(lz4_zarr_path): + # print(f"Extracting dataset {name} from {compressed_dir} to {root}") + # subprocess.run(f"mkdir -p {root}/{name}.zarr", shell=True) + # subprocess.run( + # [f"lz4 -d -c {lz4_zarr_path} | tar xf - -C {root}/{name}.zarr --strip-components=1"], + # cwd=root, + # shell=True, + # check=True, + # ) + # else: + # # For raw UMI datasets + # zip_path = os.path.join(compressed_dir, name + ".zarr.zip") + # if os.path.exists(zip_path): + # print(f"Extracting dataset {name} from {compressed_dir} to {root}") + # subprocess.run( + # [f"unzip {zip_path} -d {root}"], + # cwd=root, + # shell=True, + # check=True, + # ) + # print(f"Dataset {name} extracted to {root}") + # else: + # raise FileNotFoundError(f"Dataset {name} not found in {root} or {compressed_dir}") + # if dist.is_available() and dist.is_initialized(): + # dist.barrier() + if os.path.exists(os.path.join(root, name, "episode_data.zarr")): + zarr_path = os.path.join(root, name, "episode_data.zarr") + elif os.path.exists(os.path.join(root, name + ".zarr")): + zarr_path = os.path.join(root, name + ".zarr") + elif os.path.exists(os.path.join(root, name)): + zarr_path = os.path.join(root, name) + else: + raise FileNotFoundError(f"Dataset {name} not found in {root}") + + self.zarr_path: str = zarr_path + self.name: str = name + log.info(f"Dataset name: {self.name}") + + self.include_episode_num: int = include_episode_num + self.include_episode_indices: list[int] = include_episode_indices + self.used_episode_ratio: float = used_episode_ratio + self.index_pool_size_per_episode: int = index_pool_size_per_episode + self.history_padding_length: int = history_padding_length + self.future_padding_length: int = future_padding_length + self.seed: int = seed + self.rng: np.random.Generator = np.random.default_rng(seed) + self.starting_percentile_max: float = starting_percentile_max + self.starting_percentile_min: float = starting_percentile_min + self.apply_image_augmentation_in_cpu: bool = apply_image_augmentation_in_cpu + self.use_relative_pose: bool = use_relative_pose + assert relative_pose_mode in ["frame_wise", "anchored"], ( + f"relative_pose_mode must be one of frame_wise or anchored, but got {relative_pose_mode}" + ) + self.relative_pose_mode: Literal["frame_wise", "anchored"] = relative_pose_mode + self.use_relative_gripper_width: bool = use_relative_gripper_width + self.mode: str = mode + self.image_size: int = image_size + self.domain_id: int = get_domain_id("umi") + + if os.path.exists(f"{self.zarr_path}/../sim_config.yaml"): + log.info(f"sim_config.yaml found in {self.zarr_path}/../") + sim_config_dict = OmegaConf.load(f"{self.zarr_path}/../sim_config.yaml") + self.sim_config_str = OmegaConf.to_yaml(sim_config_dict, resolve=False) + else: + self.sim_config_str = "" + + log.info(f"{self.zarr_path=}") + zarr_store = zarr.open(self.zarr_path, mode="r") + assert isinstance(zarr_store, zarr.Group), f"zarr store {self.zarr_path} is not a group." + self.zarr_store: zarr.Group = zarr_store + + assert len(source_data_meta) > 0, "source_data_meta is empty." + self.source_data_meta: dict[str, SourceDataMeta] = construct_source_data_meta(source_data_meta) + + assert len(output_data_meta) > 0, "output_data_meta is empty." + self.output_data_meta: dict[str, DataMeta] = construct_data_meta(output_data_meta) + + self.max_history_length: int = max( + 0, + -min(entry_meta.include_indices[0] for entry_meta in self.source_data_meta.values()), + ) + self.max_future_length: int = max( + 0, + max(entry_meta.include_indices[-1] for entry_meta in self.source_data_meta.values()), + ) + if self.history_padding_length > self.max_history_length: + raise ValueError( + f"history_padding_length {self.history_padding_length} is larger than max_history_length {self.max_history_length}. This may cause ambiguity in the data." + ) + + self.normalizer: Optional[FixedNormalizer] = None + self.normalizer_dir: str = normalizer_dir + if normalizer_dir != "": + normalizer_path = _resolve_normalizer_path(normalizer_dir, self.name) + if normalizer_path is not None: + log.info(f"Loading normalizer from {normalizer_path}.") + self.normalizer = FixedNormalizer(self.output_data_meta) + with open(normalizer_path) as f: + self.normalizer.from_dict(json.load(f)) + self.normalizer.to(torch.device("cpu")) + log.info(f"Normalizer loaded from {normalizer_path}.") + else: + raise FileNotFoundError(f"No normalizer found for normalizer_dir={normalizer_dir}.") + else: + log.info(f"normalizer_dir is empty.") + self.normalizer_sample_num: int = normalizer_sample_num + + self.store_episode_num: int + self.used_episode_indices: list[int] + self.used_episode_num: int + + self.index_pool: list[tuple[int, int]] = [] + """ + index_pool has self.store_episode_num * self.used_episode_ratio * self.index_pool_size_per_episode items. + Each item contains a tuple of (episode_idx, index), where index means the 0 index of this trajectory in an episode. + """ + + self.episode_frame_nums: dict[int, int] = {} + self.episode_valid_indices_min: dict[int, int] = {} + self.episode_valid_indices_max: dict[int, int] = {} # Exclusive + + self.transforms: BaseTransforms = BaseTransforms(self.output_data_meta, self.apply_image_augmentation_in_cpu) + + self.repeat_dataset_num: float = repeat_dataset_num + self.use_grasp_state_repr: bool = use_grasp_state_repr + self.eef_z_offset: float = eef_z_offset + + def _check_data_validity(self): + raise NotImplementedError("This method should be implemented in subclasses.") + + def _get_single_traj_data(self, episode_idx: int, traj_idx: int, output_entry_names: list[str] | None = None): + raise NotImplementedError("This method should be implemented in subclasses.") + + def _create_index_pool(self): + self.index_pool = [] + for episode_idx in self.used_episode_indices: + valid_idx_min = self.episode_valid_indices_min[episode_idx] + valid_idx_max = self.episode_valid_indices_max[episode_idx] + # valid_idx_min <= sample_idx < valid_idx_max + + zero_idx_max = valid_idx_min + int( + (valid_idx_max - valid_idx_min) * self.starting_percentile_max + ) # Exclusive + zero_idx_min = valid_idx_min + int( + (valid_idx_max - valid_idx_min) * self.starting_percentile_min + ) # Inclusive + + if self.index_pool_size_per_episode == -1: + index_pool_size = zero_idx_max - zero_idx_min + else: + assert self.index_pool_size_per_episode > 0, ( + f"index_pool_size_per_episode must be positive or -1, but got {self.index_pool_size_per_episode}." + ) + index_pool_size = self.index_pool_size_per_episode + + if index_pool_size >= zero_idx_max - zero_idx_min: + indices = np.arange(zero_idx_min, zero_idx_max) + random_indices = self.rng.choice( + range(zero_idx_min, zero_idx_max), + size=index_pool_size - (zero_idx_max - zero_idx_min), + replace=True, + ) + indices = np.concatenate([indices, random_indices]) + else: + indices = self.rng.choice( + range(zero_idx_min, zero_idx_max), + size=index_pool_size, + replace=False, + ) + indices = np.sort(indices) + for index in indices: + self.index_pool.append((episode_idx, index)) + + def _update_episode_indices(self): + if len(self.include_episode_indices) > 0: + log.info(f"Dataset {self.name}: Using specified episode indices: {self.include_episode_indices}.") + self.include_episode_num = len(self.include_episode_indices) + for episode_idx in self.include_episode_indices: + assert episode_idx < self.store_episode_num, ( + f"episode_idx {episode_idx} is out of range. Max is {self.store_episode_num}." + ) + else: + if self.include_episode_num > 0: + assert self.include_episode_num <= self.store_episode_num, ( + f"include_episode_num {self.include_episode_num} is greater than the number of episodes {self.store_episode_num}." + ) + self.include_episode_indices = self.rng.choice( + self.store_episode_num, size=self.include_episode_num, replace=False + ).tolist() + log.info( + f"Dataset {self.name}: Using {self.include_episode_num} episodes from {self.store_episode_num} episodes: {self.include_episode_indices}" + ) + elif self.include_episode_num == -1: + self.include_episode_num = self.store_episode_num + self.include_episode_indices = list(range(self.include_episode_num)) + log.info( + f"Dataset {self.name}: Using all {self.include_episode_num} episodes from {self.store_episode_num}" + ) + else: + raise ValueError( + f"include_episode_num {self.include_episode_num} is invalid. Must be -1 or a positive integer." + ) + + self.include_episode_indices = sorted(self.include_episode_indices) + + self.used_episode_indices = cast( + list[int], + self.rng.choice( + self.include_episode_indices, + size=int(self.include_episode_num * self.used_episode_ratio), + replace=False, + ).tolist(), + ) + self.used_episode_indices = sorted(self.used_episode_indices) + self.used_episode_num = len(self.used_episode_indices) + + def repeat_dataset(self, repeat_num: float | None = None): + if repeat_num is None: + repeat_num = self.repeat_dataset_num + else: + self.repeat_dataset_num = repeat_num + index_pool_size = len(self.index_pool) + repeated_size = int(index_pool_size * repeat_num) + repeated_indices = self.rng.choice( + range(index_pool_size), + size=repeated_size, + replace=True, + ) + self.index_pool = [self.index_pool[i] for i in repeated_indices] + + def split_unused_episodes( + self, + remaining_ratio: float = 1.0, + other_used_episode_indices: Optional[list[int]] = None, + ): + """ + Split unused episodes from the included episodes. + """ + log.info( + f"Splitting unused data with remaining ratio {remaining_ratio} and other used episode ids {other_used_episode_indices}." + ) + unused_dataset = copy.deepcopy(self) + unused_dataset.rng = np.random.default_rng(unused_dataset.seed) + if other_used_episode_indices is None: + other_used_episode_indices = [] + unused_episode_indices = [ + episode_idx + for episode_idx in self.include_episode_indices + if episode_idx not in self.used_episode_indices and episode_idx not in other_used_episode_indices + ] + unused_dataset.used_episode_indices = cast( + list[int], + self.rng.choice( + unused_episode_indices, + size=int(len(unused_episode_indices) * remaining_ratio), + replace=False, + ).tolist(), + ) + unused_dataset.used_episode_indices = sorted(unused_dataset.used_episode_indices) + unused_dataset.used_episode_ratio = len(unused_dataset.used_episode_indices) / len( + unused_dataset.include_episode_indices + ) + unused_dataset._check_data_validity() + unused_dataset._create_index_pool() + assert len(unused_dataset) >= 1, ( + f"Splitted dataset {unused_dataset.name} has no data. Please check the used_data_ratio and the overall dataset size" + ) + return unused_dataset + + def sample_data( + self, + output_entry_names: list[str], + sample_num: int, + augment_data: bool, + normalize_data: bool, + sampled_indices: npt.NDArray[np.int64] | None = None, + ) -> dict[str, torch.Tensor]: + raise NotImplementedError("This method should be implemented in subclasses.") + + def calc_stats_worker_single_process(self, start_idx: int, end_idx: int, entry_names: list[str]): + data: dict[str, torch.Tensor] = self.sample_data( + sample_num=end_idx - start_idx, + output_entry_names=entry_names, + augment_data=False, + normalize_data=False, + sampled_indices=np.arange(start_idx, end_idx), + ) + min_vals: dict[str, torch.Tensor] = {} + max_vals: dict[str, torch.Tensor] = {} + for entry_name in entry_names: + # data[entry_name]: (batch_size, traj_length, ...) + min_vals[entry_name] = torch.min(torch.min(data[entry_name], dim=0).values, dim=0).values + max_vals[entry_name] = torch.max(torch.max(data[entry_name], dim=0).values, dim=0).values + return min_vals, max_vals + + def calc_stats(self, entry_names: list[str], process_num: int = 0): + # assert process_num > 0 + if process_num == 0: + process_num = min(20, multiprocessing.cpu_count() - 2) + log.info(f"Calculating stats with {process_num} processes.") + + index_pool_size = len(self.index_pool) + indices_per_process = index_pool_size // process_num + start_indices = np.arange(0, index_pool_size, indices_per_process)[:process_num] + end_indices = start_indices + indices_per_process + end_indices[-1] = index_pool_size + + with multiprocessing.Pool(process_num) as pool: + results = pool.starmap( + self.calc_stats_worker_single_process, zip(start_indices, end_indices, [entry_names] * process_num) + ) + + stats: dict[str, dict[str, torch.Tensor]] = {} + + for result in results: + for entry_name in entry_names: + if entry_name not in stats: + stats[entry_name] = { + "min": result[0][entry_name], + "max": result[1][entry_name], + } + else: + stats[entry_name]["min"] = torch.min(stats[entry_name]["min"], result[0][entry_name]) + stats[entry_name]["max"] = torch.max(stats[entry_name]["max"], result[1][entry_name]) + + return stats + + def fit_normalizer(self) -> FixedNormalizer: + log.info(f"Fitting normalizer for {self.name}.") + self.normalizer = FixedNormalizer(self.output_data_meta) + + normalize_entries = [ + entry_meta.name for entry_meta in self.output_data_meta.values() if entry_meta.normalizer != "identity" + ] + + stats = self.calc_stats( + entry_names=normalize_entries, + ) + log.info(f"Stats: {stats}") + + self.normalizer.from_dict(stats) + + self.normalizer.to(torch.device("cpu")) + + normalizer_state_dict = self.normalizer.as_dict("list") + + if self.normalizer_dir != "": + os.makedirs(self.normalizer_dir, exist_ok=True) + normalizer_path = os.path.join(self.normalizer_dir, f"{self.name}_normalizer.json") + with open(normalizer_path, "w") as f: + json.dump(normalizer_state_dict, f) + log.info(f"Normalizer dict saved to {normalizer_path}.") + + return self.normalizer + + def process_image_data(self, data: npt.NDArray[Any]) -> npt.NDArray[np.float32]: + if ( + data.shape[-1] <= 4 + ): # (..., H, W, C) where the color dimension is usually a small number (1 (grayscale), 3 (RGB), or 4 (RGBD)) + dims = len(data.shape) + data = data.transpose((*range(dims - 3), -1, -3, -2)) # (..., C, H, W) + + if data.dtype == np.uint8: + return (data / 255.0).astype(np.float32) + + return data + + def __len__(self) -> int: + return len(self.index_pool) + + def __getitem__(self, idx: int) -> dict[str, torch.Tensor]: + raise NotImplementedError("This method should be implemented in subclasses.") + + +class AggregatedDataset(BaseDataset): + """ + Dataset loader for the iPhUMI dataset. + Example structure: + / + ├── data + │ ├── camera0_main_rgb (N, image_size, image_size, 3) uint8 + │ ├── camera0_ultrawide_rgb (N/6, image_size, image_size, 3) uint8 + │ ├── robot0_demo_end_pose (N, 6) float64 + │ ├── robot0_demo_start_pose (N, 6) float64 + │ ├── robot0_eef_pos (N, 3) float32 + │ ├── robot0_eef_rot_axis_angle (N, 3) float32 + │ └── robot0_gripper_width (N, 1) float32 + └── meta + └── episode_ends (K,) int64 + + """ + + def __init__( + self, + down_sample_steps: int, + **kwargs, + ): + self.down_sample_steps: int = down_sample_steps + kwargs["history_padding_length"] = kwargs["history_padding_length"] * down_sample_steps + kwargs["future_padding_length"] = kwargs["future_padding_length"] * down_sample_steps + for meta in kwargs["source_data_meta"].values(): + meta["include_indices"] = [i * down_sample_steps for i in meta["include_indices"]] + + super().__init__(**kwargs) + + data_store = self.zarr_store["data"] + assert isinstance(data_store, zarr.Group) + + self.data_store: zarr.Group = data_store + self.data_store_keys: list[str] = list(data_store.keys()) + + self.episode_ends: npt.NDArray[np.int64] = np.array(self.zarr_store["meta"]["episode_ends"]) + self.store_episode_num: int = len(self.episode_ends) + + self._update_episode_indices() + + self.episode_starts: npt.NDArray[np.int64] = np.zeros_like(self.episode_ends) + + for i, end in enumerate(self.episode_ends): + if i == 0: + self.episode_starts[i] = 0 + else: + self.episode_starts[i] = self.episode_ends[i - 1] + + self.episode_frame_nums[i] = end - self.episode_starts[i] + self.episode_valid_indices_min[i] = self.max_history_length - self.history_padding_length + self.episode_valid_indices_max[i] = ( + self.episode_frame_nums[i] + self.future_padding_length - self.max_future_length + ) + + self._create_index_pool() + + log.info( + f"Dataset: {self.name}, store_episode_num: {self.store_episode_num}, include_episode_num: {self.include_episode_num}, used_episode_num: {self.used_episode_num}" + ) + + def _check_data_validity(self): + # Not implemented yet. Will skip checking for now. + pass + + def _process_source_data(self, data_dict: dict[str, npt.NDArray[Any]]) -> dict[str, npt.NDArray[Any]]: + # Override this function to process the source data + raise NotImplementedError("This function should be overridden by the subclass.") + + def _get_single_traj_data( + self, + episode_idx: int, + traj_idx: int, + output_entry_names: list[str] | None = None, + ): + episode_length = self.episode_frame_nums[episode_idx] + start_idx = self.episode_starts[episode_idx] + + source_data_dict: dict[str, Any] = {} + + if output_entry_names is not None and len(output_entry_names) > 0: + source_entry_names = [ + self.output_data_meta[target_entry_name].source_entry_names for target_entry_name in output_entry_names + ] + source_entry_names = [item for sublist in source_entry_names for item in sublist] + else: + source_entry_names = None + + for entry_meta in self.source_data_meta.values(): + if ( + source_entry_names is not None + and len(source_entry_names) > 0 + and entry_meta.name not in source_entry_names + ): + # Skip the entries that are not needed to make data loading faster + continue + + if entry_meta.name not in self.data_store_keys: + continue + indices = [traj_idx + i for i in entry_meta.include_indices] + # Crop the indices to the valid range. Will introduce padding if the indices are out of range. + indices = [(0 if i < 0 else episode_length - 1 if i >= episode_length else i) for i in indices] + global_indices = [start_idx + i for i in indices] + if entry_meta.name.endswith("ultrawide_rgb"): + global_indices = self.zarr_store["meta"][f"upsample_index_{entry_meta.name}"][global_indices] + + source_data_dict[entry_meta.name] = np.array(self.data_store[entry_meta.name][global_indices]) + + processed_data_dict = self._process_source_data(source_data_dict) + + torch_data_dict: dict[str, Any] = {} + torch_data_dict["episode_idx"] = torch.tensor([episode_idx]) # [1] + torch_data_dict["traj_idx"] = torch.tensor([traj_idx]) # [1] + + for entry_meta in self.output_data_meta.values(): + if ( + output_entry_names is not None + and len(output_entry_names) > 0 + and entry_meta.name not in output_entry_names + ): + # Skip the entries that are not needed to make data loading faster + continue + + assert entry_meta.name in processed_data_dict + processed_data = processed_data_dict[entry_meta.name] + if isinstance(processed_data, np.ndarray): + if entry_meta.data_type == "image": + processed_data = self.process_image_data(processed_data) # -> (..., C, H, W), float32 + processed_data = torch.from_numpy(processed_data) + + torch_data_dict[entry_meta.name] = processed_data + + # Pass through initial_pose (non-normalized route only). + if "initial_pose" in processed_data_dict: + torch_data_dict["initial_pose"] = torch.from_numpy(processed_data_dict["initial_pose"]) + + return torch_data_dict + + def __getitem__(self, idx: int): + """ + output_data_dict: + video: (C, T, H, W) uint8 + camera_poses: (T, 9) # This is actually the TCP pose: the middle point of the gripper tip + eef_poses: (T, 9) (xyz, rot6d), is defined self.eef_z_offset behind the gripper tip + eef_commands: (T, 1) + t5_text_embeddings: (512, 1024) + t5_text_mask: (512,) int64 + fps: int + __key__: int + + if self.use_grasp_state_repr: + grasp_states: (T, 6) (left-finger-xyz, right-finger-xyz) wrt eef_poses + """ + episode_idx, traj_idx = self.index_pool[idx] + + # skip data if corrupted + while True: + try: + torch_data_dict = self._get_single_traj_data(episode_idx, traj_idx) + break + except Exception: + import traceback + + traceback.print_exc() + log.warning( + f"UMI Dataset WARNING: Corrupted data found at episode_idx: {episode_idx}, traj_idx: {traj_idx}. Randomly selecting a new data.", + ) + random_idx = self.rng.choice(len(self.index_pool)) + episode_idx, traj_idx = self.index_pool[random_idx] + + # Pop initial_pose before transforms (transforms crash on unknown keys). + initial_pose = torch_data_dict.pop("initial_pose", None) + + torch_data_dict = self.transforms.apply(torch_data_dict) + + if self.normalizer is not None: + torch_data_dict = self.normalizer.normalize(torch_data_dict) + + # Re-add initial_pose (non-normalized route only, for viewer). + if initial_pose is not None: + torch_data_dict["initial_pose"] = initial_pose + + for entry_meta in self.output_data_meta.values(): + assert torch_data_dict[entry_meta.name].shape == ( + entry_meta.length, + *entry_meta.shape, + ), ( + f"entry_meta: {entry_meta.name}, torch_data_dict[entry_meta.name].shape: {torch_data_dict[entry_meta.name].shape}, entry_meta.length: {entry_meta.length}, entry_meta.shape: {entry_meta.shape}" + ) + + # resize to image_size x image_size if needed + if torch_data_dict["video"].shape[-2:] != (self.image_size, self.image_size): + torch_data_dict["video"] = torch.nn.functional.interpolate( + torch_data_dict["video"], size=(self.image_size, self.image_size), mode="bilinear" + ) + torch_data_dict["video"] = (torch_data_dict["video"] * 255.0).to(torch.uint8) + + if torch_data_dict["video"].shape[1] == 3: # [T,C,H,W] + torch_data_dict["video"] = torch_data_dict["video"].permute(1, 0, 2, 3) # [C,T,H,W] + + torch_data_dict["__key__"] = torch.tensor([idx], dtype=torch.long) # [1] + torch_data_dict["conditioning_fps"] = torch.tensor(60 / self.down_sample_steps, dtype=torch.long) # scalar + + if self.mode == "joint": + mode = random.choice(["forward_dynamics", "inverse_dynamics", "policy"]) + else: + mode = self.mode + torch_data_dict["mode"] = mode + + _name = re.sub(r"(_v?\d+)+$", "", self.name) + torch_data_dict["ai_caption"] = _name.replace("_", " ") + torch_data_dict["domain_id"] = torch.tensor([self.domain_id], dtype=torch.long) # [1] + torch_data_dict["viewpoint"] = "wrist_view" + + if not self.use_grasp_state_repr: + # Use eef_poses or camera_poses for the action based on self.use_eef_pose + action = torch.cat( + [ + torch_data_dict["eef_poses"] if self.use_eef_pose else torch_data_dict["camera_poses"], + torch_data_dict["eef_commands"], + ], + dim=1, + ).to(torch.float32) # [T,camera_pose_dim+1] + else: + action = torch.cat( + [ + torch_data_dict["eef_poses"] if self.use_eef_pose else torch_data_dict["camera_poses"], + torch_data_dict["grasp_states"], + ], + dim=1, + ).to(torch.float32) # [T,eef_pose_dim+grasp_state_dim] + + torch_data_dict["action"] = action + + return torch_data_dict + + def sample_data( + self, + output_entry_names: list[str], + sample_num: int, + augment_data: bool, + normalize_data: bool, + sampled_indices: npt.NDArray[np.int64] | None = None, + ) -> dict[str, torch.Tensor]: + if sample_num == -1: + sample_num = len(self.index_pool) + + if sampled_indices is None: + sampled_indices = self.rng.choice( + len(self.index_pool), min(sample_num, len(self.index_pool)), replace=False + ) + else: + assert len(sampled_indices) == sample_num, ( + f"sampled_indices should be of length {sample_num}, but got {len(sampled_indices)}" + ) + + samples = [] + log.info(f"Sampling {sample_num} data from {len(self.index_pool)} trajectories.") + for idx in tqdm.tqdm(sampled_indices): + episode_idx, traj_idx = self.index_pool[idx] + samples.append(self._get_single_traj_data(episode_idx, traj_idx, output_entry_names)) + + all_samples_data_dict: dict[str, torch.Tensor] = aggregate_batch(samples, aggregate_fn=torch.stack) + + return all_samples_data_dict + + +class MultiTaskDataset(Dataset[dict[str, torch.Tensor]]): + def __init__( + self, + name: str, + sub_dataset_target: str, + dataset_configs: dict[str, dict[str, Any]], + eager_load: bool = False, + **base_config: dict[str, Any], + ): + """ + name: if select certain sub-datasets, should be the name of multiple sub-datasets connected by "," + root, normalizer_dir should be the same for all datasets + dataset_configs: { + "dataset_name_1": { + "sample_ratio": 1.0, + "override_config_1": "value_1", + "override_config_2": "value_2", + ... + }, + ... + } + """ + + if isinstance(dataset_configs, DictConfig): + dataset_configs = cast(dict[str, dict[str, Any]], OmegaConf.to_container(dataset_configs)) + if ";" in name: + name = name.replace(";", ",") + if "," in name: + dataset_names = name.split(",") + dataset_configs = {name: dataset_configs[name] for name in dataset_names} + + log.info(f"dataset_configs: {dataset_configs}") + self.dataset_configs: dict[str, dict[str, Any]] = dataset_configs + assert len(self.dataset_configs) >= 1, "At least one dataset is required" + + if isinstance(base_config, DictConfig): + base_config = cast(dict[str, Any], OmegaConf.to_container(base_config)) + self.base_config: dict[str, Any] = base_config + + self.sample_ratios: dict[str, float] = { + name: config.pop("sample_ratio") for name, config in self.dataset_configs.items() + } + + + self._all_shard_roots = list(self.dataset_configs.keys()) + + self.datasets: dict[str, BaseDataset] = {} + self.index_pool: list[tuple[str, int]] = [] + + self.sub_dataset_target = sub_dataset_target + + if eager_load: + self._register_sources() + + def _register_sources(self, indices: list[int] | None = None): + if indices is None: + indices = list(range(len(self._all_shard_roots))) + log.info(f"Registering UMI Multi-Task datasets: {indices}") + + for idx in indices: + base_name = self._all_shard_roots[idx] + dataset_config = self.dataset_configs[base_name] + num_shards = dataset_config.get("num_shards", 0) + shard_config = {k: v for k, v in dataset_config.items() if k != "num_shards"} + if num_shards > 0: + for i in range(num_shards): + shard_name = f"{base_name}_v{i}" + log.info(f"Initializing dataset shard: {shard_name}") + config = copy.deepcopy(self.base_config) + config.update(copy.deepcopy(shard_config)) + config["name"] = shard_name + config["_target_"] = self.sub_dataset_target + self.datasets[shard_name] = hydra.utils.instantiate(config) + else: + log.info(f"Initializing dataset: {base_name}") + config = copy.deepcopy(self.base_config) + config.update(copy.deepcopy(shard_config)) + config["name"] = base_name + config["_target_"] = self.sub_dataset_target + self.datasets[base_name] = hydra.utils.instantiate(config) + + self._create_index_pool() + + @property + def normalizer(self) -> FixedNormalizer | None: + return next(iter(self.datasets.values())).normalizer + + @property + def action_dim(self) -> int: + """ """ + return 10 + + def _create_index_pool(self): + self.index_pool = [] + for dataset_name, dataset in self.datasets.items(): + self.index_pool.extend((dataset_name, i) for i in range(len(dataset))) + + def __len__(self): + return len(self.index_pool) + + def __getitem__(self, idx: int) -> dict[str, torch.Tensor]: + dataset_name, data_idx = self.index_pool[idx] + data = self.datasets[dataset_name][data_idx] + dataset_index = list(self.datasets.keys()).index(dataset_name) + data["dataset_index"] = torch.tensor([dataset_index]) # [1] + data["data_index"] = torch.tensor([data_idx]) # [1] + return data + + def split_unused_episodes( + self, + remaining_ratio: float = 1.0, + other_used_episode_indices: list[int] | None = None, + ): + unused_dataset = copy.deepcopy(self) + unused_dataset.index_pool = [] + unused_dataset.datasets = {} + + for dataset_name, dataset in self.datasets.items(): + unused_dataset.datasets[dataset_name] = dataset.split_unused_episodes( + remaining_ratio, other_used_episode_indices + ) + unused_dataset._create_index_pool() + + return unused_dataset + + def repeat_dataset(self): + for dataset_name, dataset in self.datasets.items(): + dataset.repeat_dataset() + self._create_index_pool() + + +def _process_source_data( + self: AggregatedDataset, data_dict: dict[str, npt.NDArray[Any]] +) -> dict[str, npt.NDArray[Any]]: + """ + Will calculate the following data: + relative poses + poses wrt episode start + This step does not include normalization and data augmentation + Input data_dict: + camera0_main_rgb: (..., H, W, 3) uint8 + camera0_ultrawide_rgb: (..., H, W, 3) uint8 + robot0_demo_start_pose: (1, 6) float64 (optional) + robot0_eef_pos: (..., 3) float32 + robot0_eef_rot_axis_angle: (..., 3) float32 + robot0_gripper_width: (..., 1) float32 + + Output data_dict: + video: (C, T, H, W) uint8 + camera_poses: (T, 9) # 9dim: [pos_3d, rot_6d], rot_6d is the first two rows of the rotation matrix + eef_poses: (T, 9) (xyz, rot6d), is defined self.eef_z_offset behind the gripper tip + eef_commands: (T, 1) + t5_text_embeddings: (512, 1024) + t5_text_mask: (512,) int64 + fps: int + __key__: int + + if self.use_grasp_state_repr: + grasp_states: (T, 6) (left-finger-xyz, right-finger-xyz) wrt eef_poses + + """ + + processed_data_dict: dict[str, npt.NDArray[Any]] = {} + + pose_indices = self.source_data_meta["robot0_eef_pos"].include_indices + assert pose_indices == self.source_data_meta["robot0_eef_rot_axis_angle"].include_indices, ( + f"robot0_eef_pos and robot0_eef_rot_axis_angle must be aligned" + ) + + gripper_width_indices = self.source_data_meta["robot0_gripper_width"].include_indices + + ## Use absolute gripper width + if self.use_relative_gripper_width: + assert len(pose_indices) == len(gripper_width_indices), f"{len(pose_indices)=}, {len(gripper_width_indices)=}" + else: + assert len(pose_indices) == len(gripper_width_indices) + 1, ( + f"{len(pose_indices)=}, {len(gripper_width_indices)=}" + ) + + zero_idx = pose_indices.index(0) + + assert self.robot_num == 1, "Currently only support one robot" + + for i in range(self.robot_num): + assert len(pose_indices) == self.output_data_meta[f"camera_poses"].length + 1, ( + f"{len(pose_indices)=}, {self.output_data_meta[f'camera_poses'].length=}" + ) + + + camera_poses_meta = self.output_data_meta[f"camera_poses"] + camera_poses = np.zeros((camera_poses_meta.length, *camera_poses_meta.shape), dtype=np.float32) + + eef_commands_meta = self.output_data_meta[f"eef_commands"] + eef_commands = np.zeros((eef_commands_meta.length, *eef_commands_meta.shape), dtype=np.float32) + + video_meta = self.output_data_meta[f"video"] + if video_meta.source_entry_names[0] in data_dict: + video = data_dict[video_meta.source_entry_names[0]] + processed_data_dict["video"] = video + + if f"robot{i}_eef_pos" in data_dict and f"robot{i}_eef_rot_axis_angle" in data_dict: + pose_mat = build_abs_pose_from_components( + data_dict[f"robot{i}_eef_pos"], + data_dict[f"robot{i}_eef_rot_axis_angle"], + "axisangle", + ) + assert self.use_relative_pose, "Currently only support relative pose" + pose_convention = "backward_framewise" if self.relative_pose_mode == "frame_wise" else "backward_anchored" + + pose = pose_abs_to_rel(pose_mat, rotation_format="rot6d", pose_convention=pose_convention) + assert len(pose) == camera_poses_meta.length, ( + f"pose should be one step longer (zero index is excluded) than camera_poses_meta.length, but got {len(pose)} and {camera_poses_meta.length}" + ) + processed_data_dict[f"camera_poses"] = ( + pose # The camera poses here are actually TCP pose (the middle point of the gripper tip) + ) + + # Stash the absolute first-frame pose for viewer trajectory reconstruction. + # Only useful on the non-normalized route (normalizer_dir=""). + if self.normalizer is None: + processed_data_dict["initial_pose"] = pose_mat[zero_idx].astype(np.float32) + + offset_eef_pose_mat = pose_mat.copy() + offset_eef_pose_mat[:, :3, 3] += ( + offset_eef_pose_mat[:, :3, 2] * self.eef_z_offset + ) # Apply a z offset in the local coordinate frame instead of the global frame + offset_eef_pose = pose_abs_to_rel( + offset_eef_pose_mat, rotation_format="rot6d", pose_convention=pose_convention + ) + processed_data_dict[f"eef_poses"] = offset_eef_pose + + if f"robot{i}_gripper_width" in data_dict: + ## Use absolute gripper width + # eef_commands[:] = data_dict[f"robot{i}_gripper_width"] + # processed_data_dict[f"eef_commands"] = eef_commands + + assert not self.use_relative_gripper_width, "Currently only support absolute gripper width" + processed_gripper_width = data_dict[f"robot{i}_gripper_width"] + + eef_commands[:] = processed_gripper_width + processed_data_dict[f"eef_commands"] = eef_commands + + if self.use_grasp_state_repr: + processed_data_dict[f"grasp_states"] = np.zeros((eef_commands_meta.length, 6), dtype=np.float32) + processed_data_dict[f"grasp_states"][:, 2] = -self.eef_z_offset + processed_data_dict[f"grasp_states"][:, 5] = -self.eef_z_offset + processed_data_dict[f"grasp_states"][:, :1] = eef_commands / 2 + processed_data_dict[f"grasp_states"][:, 3:4] = -eef_commands / 2 + + # if f"robot{i}_demo_start_pose" in data_dict and f"robot{i}_eef_rot_axis_angle_wrt_start" in self.output_data_meta: + # # Calculate relative poses wrt episode start + # try: + # wrt_start_entry_meta = self.output_data_meta[ + # f"robot{i}_eef_rot_axis_angle_wrt_start" + # ] + # assert ( + # data_dict[f"robot{i}_demo_start_pose"].shape[0] == 1 + # ), "robot0_demo_start_pose must be (1, 6)" + # # HACK: add noise to episode start pose. Copied from the original UMI codebase. + # start_pose: npt.NDArray[np.float64] = data_dict[ + # f"robot{i}_demo_start_pose" + # ][0] + # start_pose += self.rng.normal( + # scale=[0.05, 0.05, 0.05, 0.05, 0.05, 0.05], + # size=start_pose.shape, + # ) + # start_pose_mat = pose_to_mat(start_pose) + # rel_pose_mat = convert_pose_mat_rep( + # pose_mat, + # base_pose_mat=start_pose_mat, + # pose_rep="relative", + # backward=False, + # ) + # except ValueError: + # # No wrt_start_entry_meta, so no relative poses wrt episode start + # # print(f"No wrt_start_entry_meta for robot{i}") + # pass + + return processed_data_dict + + +class iPhUMISingleTrajDataset(AggregatedDataset): + def __init__(self, align_to_ultrawide: bool, **kwargs): + """ + If align_to_ultrawide is True, the dataset will be aligned to the ultrawide camera. (will only use 1/6 of the data, but will guarantee the main camera is aligned to the ultrawide camera) + """ + if "camera0_ultrawide_rgb" in kwargs["source_data_meta"]: + # assert kwargs["down_sample_steps"] == 6, "down_sample_steps must be 6 for iPhUMI dataset if using ultrawide camera. This is because the ultrawide camera is captured at 10Hz while others are at 60Hz." + assert len(kwargs["source_data_meta"]["camera0_ultrawide_rgb"].include_indices) == 1, ( + "camera0_ultrawide_rgb should not include history, because it is captured at 10Hz" + ) + self.align_to_ultrawide: bool = align_to_ultrawide + + super().__init__(**kwargs) + if self.align_to_ultrawide: + assert self.source_data_meta["camera0_ultrawide_rgb"].include_indices == [0], ( + "camera0_ultrawide_rgb should only include the current frame" + ) + + def _create_index_pool(self): + super()._create_index_pool() + if self.align_to_ultrawide: + new_index_pool: list[tuple[int, int]] = [] + + ultrawide_indices: npt.NDArray[np.int32] = np.array( + self.zarr_store["meta"]["upsample_index_camera0_ultrawide_rgb"], dtype=np.int32 + ) + regular_indices: npt.NDArray[np.int32] = np.array( + self.zarr_store["meta"]["downsample_index_camera0_ultrawide_rgb"], dtype=np.int32 + ) + # Only use the data that is aligned to the ultrawide camera + for episode_idx, traj_idx in self.index_pool: + start_idx = self.episode_starts[episode_idx] + global_idx = start_idx + traj_idx + if regular_indices[ultrawide_indices[global_idx]] == global_idx: + new_index_pool.append((episode_idx, traj_idx)) + + log.info( + f"Aligning to ultrawide camera, index pool size changed from {len(self.index_pool)} to {len(new_index_pool)}" + ) + self.index_pool = new_index_pool + + +iPhUMISingleTrajDataset._process_source_data = _process_source_data + + +class UMISingleTrajDataset(AggregatedDataset): + pass + + +UMISingleTrajDataset._process_source_data = _process_source_data + + +def get_umi_dataset( + dataset_name: str, + dataset_type: str, + is_val: bool, + mode: str = "policy", + image_size: int = 256, + root: str = "", + normalizer_dir: str | None = None, + **kwargs, +): + # assert len(kwargs) == 0, f"get_umi_dataset does not accept any kwargs, but got {kwargs}" + log.info(f"get_umi_dataset override kwargs: {kwargs=}") + try: + OmegaConf.register_new_resolver("eval", eval) + except Exception: + pass + if dataset_type == "umi": + cfg = OmegaConf.load("cosmos_framework/data/vfm/action/umi_dataset.yaml") + elif dataset_type == "umi_multi_task": + if hydra.core.global_hydra.GlobalHydra().is_initialized(): + cfg = hydra.compose(config_name="umi_dataset_multi_task") + else: + with hydra.initialize("."): + cfg = hydra.compose(config_name="umi_dataset_multi_task") + else: + raise ValueError(f"Invalid dataset type: {dataset_type}. Available dataset types: umi, umi_multi_task") + + cfg.update(kwargs) + + cfg.name = dataset_name + cfg.mode = mode + if root is not None: + cfg.root = root + if normalizer_dir is not None: + cfg.normalizer_dir = normalizer_dir + assert mode in ["policy", "joint", "forward_dynamics", "inverse_dynamics", "image2video"], ( + f"Invalid mode: {mode}. Available modes: policy, joint, forward_dynamics, inverse_dynamics, video" + ) + cfg.image_size = image_size + if is_val: + cfg.output_data_meta = cfg.val_output_data_meta + cfg.eager_load = True + cfg.val_output_data_meta = None # type: ignore + OmegaConf.resolve(cfg) + dataset = hydra.utils.instantiate(cfg) + if is_val: + return dataset.split_unused_episodes() + else: + return dataset + + +if __name__ == "__main__": + # OmegaConf.register_new_resolver("eval", eval) + # cfg = OmegaConf.load("cosmos_framework/data/vfm/action/umi_dataset.yaml") + # os.environ["HYDRA_FULL_ERROR"] = "1" + args = sys.argv[1:] + dataset_type = args[0] + dataset_name = args[1] + + dataset: BaseDataset = get_umi_dataset( + dataset_name, dataset_type, is_val=False, use_grasp_state_repr=False, use_eef_pose=False + ) + + # dataset.fit_normalizer() + + def save_np_array_as_video(rollout_images: list[np.ndarray] | torch.Tensor, video_path: str, fps: int = 10): + """ + rollout_images: (T, H, W, C) or (T, C, H, W) + Saves an MP4 replay of an episode. + """ + + if isinstance(rollout_images, torch.Tensor): + assert rollout_images.ndim == 4, f"Rollout images must be a 4D tensor, but got {rollout_images.ndim}" + if rollout_images.shape[1] == 3: + rollout_images = rollout_images.permute(0, 2, 3, 1) # [T,H,W,C] + rollout_images = rollout_images.cpu().numpy() + + import imageio + + video_writer = imageio.get_writer(video_path, fps=fps) + for img in rollout_images: + video_writer.append_data(img) + video_writer.close() + print(f"Saved rollout MP4 at path {video_path}") + + print(dataset) + ld = len(dataset) + iss = [0, ld // 4, ld // 2, ld // 4 * 3, -1] + for i in iss: + # for i in range(60, 70): + data = dataset[i] + + # print(data["video"].shape) # [C,T,H,W] + # video_thwc = data["video"].permute(1, 2, 3, 0) # [T,H,W,C] + # save_np_array_as_video(video_thwc, f"video_{i}.mp4") + # print(data["camera_poses"]) + # for k, v in data.items(): + # print(k, v.shape) + print("camera_poses", data["camera_poses"]) + print("eef_commands", data["eef_commands"]) + print("eef_poses", data["eef_poses"]) + if dataset.use_grasp_state_repr: + print("grasp_states", data["grasp_states"]) + print("action", data["action"]) diff --git a/cosmos_framework/data/vfm/action/umi_dataset.yaml b/cosmos_framework/data/vfm/action/umi_dataset.yaml new file mode 100644 index 0000000..8f7fe80 --- /dev/null +++ b/cosmos_framework/data/vfm/action/umi_dataset.yaml @@ -0,0 +1,216 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +_target_: projects.cosmos3.vfm.datasets.action.umi_dataset.UMISingleTrajDataset + +root: null + +name: ??? +mode: policy +image_size: 256 + +robot_num: 1 +down_sample_steps: 3 # 60 fps -> 20 fps + +include_episode_num: -1 # To specify how many episodes to include. -1 means using all episodes. +include_episode_indices: [] # If not empty, will override used_episode_ratio +used_episode_ratio: 0.99 # How many episodes are used for training +history_padding_length: 0 +future_padding_length: 16 +starting_percentile_max: 1.0 +starting_percentile_min: 0.0 +index_pool_size_per_episode: -1 # Use all indices +seed: 0 +use_relative_pose: true # UMI uses relative pose +relative_pose_mode: frame_wise # When changing this to anchored, please also update the normalizer_dir +use_relative_gripper_width: false # UMI dataset usually does not have intential gripper recovery actions +# Using relative gripper width often leads to gripper width out of distribution which can be seen in the camera +normalizer_sample_num: -1 + + +# If relative_pose_mode == "anchored", use the following normalizer +# normalizer_dir: uva_umi_single_task_anchored_normalizer.json + +# If relative_pose_mode == "frame_wise", use the following normalizer +# normalizer_dir: uva_umi_single_task_frame_wise_normalizer.json +normalizer_dir: uva_umi_single_task_frame_wise_normalizer.json + +repeat_dataset_num: 20 # How many times to repeat the dataset. This will avoid the dataset being consumed too fast and the reload time being too long. + + +apply_image_augmentation_in_cpu: true +use_grasp_state_repr: false +eef_z_offset: -0.10 # set the eef pose to be 0.10m behind the gripper position. This is to align with the wrist settings in egocentric datasets. For different gripper designs, this offset might need to be adjusted. +use_eef_pose: false # If true, use the eef pose for the action. Please specify it in the dataset kwargs when calling `get_umi_dataset` + +source_data_meta: + camera0_rgb: + include_indices: ${eval:"ListConfig(list(range(0, 17)))"} + shape: [224, 224, 3] + + robot0_eef_pos: + # include_indices: ${eval:"ListConfig(list(range(-1, 17)))"} + # 1 frame into the past, 16 frames into the future. 0 is the current frame + # obs: relative pose of [-1] wrt to [0] + # action: relative pose of [1:17] wrt to [0] + include_indices: ${eval:"ListConfig(list(range(0, 17)))"} + shape: [3] + + robot0_eef_rot_axis_angle: + # include_indices: ${eval:"ListConfig(list(range(-1, 17)))"} + include_indices: ${eval:"ListConfig(list(range(0, 17)))"} + shape: [3] + + robot0_gripper_width: + # include_indices: ${eval:"ListConfig(list(range(0, 17)))"} + # Only use absolute width so no need to calculate relative values + # obs: absolute width of [0] + # action: absolute width of [1:17] + include_indices: ${eval:"ListConfig(list(range(1, 17)))"} # Not include the current gripper width + shape: [1] + + + +output_data_meta: + video: + name: video + data_type: image + length: 17 + normalizer: identity + augmentation: + - name: RandomCrop + size: [208, 208] + p: 0.3 + - name: Resize + size: [256, 256] + antialias: True + - name: ColorJitter + brightness: 0.3 + contrast: 0.4 + saturation: 0.5 + hue: 0.08 + p: 0.8 + - name: RandomSharpness + sharpness: 2 + p: 0.3 + - name: RandomAutoContrast + p: 0.3 + - name: RandomGrayscale + p: 0.2 + - name: RandomGaussianBlur + kernel_size: [5, 5] + sigma: [0.1, 2.0] + p: 0.3 + shape: [3, 256, 256] + source_entry_names: + - camera0_rgb + + camera_poses: # This is actually the TCP pose: the middle point of the gripper tip + name: camera_poses + data_type: low_dim + # length: 17 # 1 history relative camera pose (velocity) + length: 16 # Don't include history camera pose + # normalizer: range # Normalize to [-1, 1] + normalizer: clamped_range + augmentation: [] + shape: [9] + source_entry_names: + - robot0_eef_pos + - robot0_eef_rot_axis_angle + + eef_commands: + name: eef_commands + data_type: low_dim + # length: 17 # 1 history eef command (gripper width) + length: 16 # Don't include history eef command + normalizer: clamped_range # Normalize to [-1, 1] + augmentation: [] + shape: [1] + source_entry_names: + - robot0_gripper_width + + eef_poses: + name: eef_poses + data_type: low_dim + # length: 17 # 1 history relative camera pose (velocity) + length: 16 # Don't include history camera pose + # normalizer: range # Normalize to [-1, 1] + normalizer: clamped_range + augmentation: [] + shape: [9] + source_entry_names: + - robot0_eef_pos + - robot0_eef_rot_axis_angle + + grasp_states: + name: grasp_states + data_type: low_dim + # length: 17 # 1 history eef command (gripper width) + length: 16 # Don't include history eef command + normalizer: clamped_range # Normalize to [-1, 1] + augmentation: [] + shape: [6] + source_entry_names: + - robot0_gripper_width + +val_output_data_meta: + video: + name: video + data_type: image + length: 17 + normalizer: identity + augmentation: + - name: Resize + size: [256, 256] + antialias: True + shape: [3, 256, 256] + source_entry_names: + - camera0_rgb + + camera_poses: # This is actually the TCP pose: the middle point of the gripper tip + name: camera_poses + data_type: low_dim + # length: 17 # 1 history relative camera pose (velocity) + length: 16 # Don't include history camera pose + # normalizer: range # Normalize to [-1, 1] + normalizer: clamped_range + augmentation: [] + shape: [9] + source_entry_names: + - robot0_eef_pos + - robot0_eef_rot_axis_angle + + eef_commands: + name: eef_commands + data_type: low_dim + # length: 17 # 1 history eef command (gripper width) + length: 16 # Don't include history eef command + normalizer: clamped_range # Normalize to [-1, 1] + augmentation: [] + shape: [1] + source_entry_names: + - robot0_gripper_width + + eef_poses: + name: eef_poses + data_type: low_dim + # length: 17 # 1 history relative camera pose (velocity) + length: 16 # Don't include history camera pose + # normalizer: range # Normalize to [-1, 1] + normalizer: clamped_range + augmentation: [] + shape: [9] + source_entry_names: + - robot0_eef_pos + - robot0_eef_rot_axis_angle + + grasp_states: + name: grasp_states + data_type: low_dim + # length: 17 # 1 history eef command (gripper width) + length: 16 # Don't include history eef command + normalizer: clamped_range # Normalize to [-1, 1] + augmentation: [] + shape: [6] + source_entry_names: + - robot0_gripper_width diff --git a/cosmos_framework/data/vfm/action/umi_dataset_multi_task.yaml b/cosmos_framework/data/vfm/action/umi_dataset_multi_task.yaml new file mode 100644 index 0000000..175402f --- /dev/null +++ b/cosmos_framework/data/vfm/action/umi_dataset_multi_task.yaml @@ -0,0 +1,174 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +defaults: + - umi_dataset + - _self_ + +_target_: projects.cosmos3.vfm.datasets.action.umi_dataset.MultiTaskDataset + +sub_dataset_target: projects.cosmos3.vfm.datasets.action.umi_dataset.UMISingleTrajDataset + +eager_load: false + +name: umi_multi_tasks + +normalizer_dir: uva_umi_single_task_frame_wise_normalizer.json + +dataset_configs: + data_scaling_law/mouse_arrangement_0: + sample_ratio: 0.0 + data_scaling_law/mouse_arrangement_1: + sample_ratio: 0.0 + data_scaling_law/towel_folding_0: + sample_ratio: 0.0 + data_scaling_law/unplug_charger: + sample_ratio: 0.0 + data_scaling_law/water_pouring_0: + sample_ratio: 0.0 + data_scaling_law/water_pouring_1: + sample_ratio: 0.0 + umi/cup_arrangement_0: + sample_ratio: 0.0 + umi/cup_arrangement_1: + sample_ratio: 0.0 + umi/dynamic_tossing_0: + sample_ratio: 0.0 + maniwav/bagle_flipping_0: + sample_ratio: 0.0 + maniwav/bagle_flipping_1: + sample_ratio: 0.0 + maniwav/dice_pouring_0: + sample_ratio: 0.0 + maniwav/whiteboard_wiping_0: + sample_ratio: 0.0 + maniwav/wire_strapping_0: + sample_ratio: 0.0 + + ## MV-UMI uses negative gripper action - looks like relative action + # mv_umi/bottles_rack_insertion_0: + # sample_ratio: 0.0 + # mv_umi/markers_placement_0: + # sample_ratio: 0.0 + # mv_umi/markers_placement_raw_0: + # sample_ratio: 0.0 + + ## Touch in the Wild uses a different gripper scale. Will be fixed in the next UMI dataset version + # touch_in_the_wild/fluid_transfer_0: + # sample_ratio: 0.0 + # eef_z_offset: -0.067 # touch_in_the_wild uses a shorter gripper design: it is 0.033m shorter than the default gripper + # touch_in_the_wild/hex_key_insertion_0: + # sample_ratio: 0.0 + # eef_z_offset: -0.067 + # touch_in_the_wild/moving_cup_0: + # sample_ratio: 0.0 + # eef_z_offset: -0.067 + # touch_in_the_wild/moving_tape_0: + # sample_ratio: 0.0 + # eef_z_offset: -0.067 + # touch_in_the_wild/peg_insertion_0: + # sample_ratio: 0.0 + # eef_z_offset: -0.067 + # touch_in_the_wild/pencil_insertion_0: + # sample_ratio: 0.0 + # eef_z_offset: -0.067 + # touch_in_the_wild/test_tube_collection_0: + # sample_ratio: 0.0 + # eef_z_offset: -0.067 + # touch_in_the_wild/tossing_0: + # sample_ratio: 0.0 + # eef_z_offset: -0.067 + # touch_in_the_wild/whiteboard_erasing_0: + # sample_ratio: 0.0 + # eef_z_offset: -0.067 + # touch_in_the_wild/whiteboard_erasing_1: + # sample_ratio: 0.0 + # eef_z_offset: -0.067 + # touch_in_the_wild/writing_0: + # sample_ratio: 0.0 + # eef_z_offset: -0.067 + + umi_on_legs/kettlebell_pushing_0: + sample_ratio: 0.0 + umi_on_legs/tennis_ball_tossing_0: + sample_ratio: 0.0 + vitamin/articulated_object_manipulation_0: + sample_ratio: 0.0 + vitamin/dynamic_peg_insertion_0: + sample_ratio: 0.0 + vitamin/orange_placement_0: + sample_ratio: 0.0 + vitamin/scissor_hanging_0: + sample_ratio: 0.0 + vitamin/sponge_insertion_0: + sample_ratio: 0.0 + vitamin/test_tube_reorientation_0: + sample_ratio: 0.0 + FastUMI/clean_table: + sample_ratio: 0.0 + num_shards: 5 + FastUMI/close_ricecooker: + sample_ratio: 0.0 + num_shards: 1 + FastUMI/cover_beef: + sample_ratio: 0.0 + num_shards: 11 + FastUMI/fold_towel: + sample_ratio: 0.0 + num_shards: 7 + FastUMI/hotdog_in_rice_cooker: + sample_ratio: 0.0 + num_shards: 2 + FastUMI/hotdog_in_roaster: + sample_ratio: 0.0 + num_shards: 10 + FastUMI/open_container: + sample_ratio: 0.0 + num_shards: 7 + FastUMI/open_drawer: + sample_ratio: 0.0 + num_shards: 17 + FastUMI/open_ricecooker: + sample_ratio: 0.0 + num_shards: 5 + FastUMI/open_roaster: + sample_ratio: 0.0 + num_shards: 10 + FastUMI/open_suitcase: + sample_ratio: 0.0 + num_shards: 1 + FastUMI/pick_bear: + sample_ratio: 0.0 + num_shards: 11 + FastUMI/pick_bread: + sample_ratio: 0.0 + num_shards: 10 + FastUMI/pick_cup: + sample_ratio: 0.0 + num_shards: 18 + FastUMI/pick_lid: + sample_ratio: 0.0 + num_shards: 10 + FastUMI/pick_pen: + sample_ratio: 0.0 + num_shards: 12 + FastUMI/place_plate: + sample_ratio: 0.0 + num_shards: 10 + FastUMI/place_pot: + sample_ratio: 0.0 + num_shards: 6 + FastUMI/pour_coke: + sample_ratio: 0.0 + num_shards: 10 + FastUMI/rearrange_coke: + sample_ratio: 0.0 + num_shards: 10 + FastUMI/sweep_trash: + sample_ratio: 0.0 + num_shards: 11 + FastUMI/tossing: + sample_ratio: 0.0 + FastUMI/unplug_charger: + sample_ratio: 0.0 + num_shards: 10 diff --git a/cosmos_framework/data/vfm/action/unified_dataset.py b/cosmos_framework/data/vfm/action/unified_dataset.py new file mode 100644 index 0000000..4ba94d9 --- /dev/null +++ b/cosmos_framework/data/vfm/action/unified_dataset.py @@ -0,0 +1,660 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Unified iterable dataset for Action multi-embodiment robot data. + +``ActionUnifiedIterableDataset`` is the Layer 2 component of the Action data loading +pipeline. It wraps *all* Action datasets into a single ``IterableDataset`` and +handles: + +- **Rank-level dataset assignment** (Hare-Niemeyer proportional allocation) +- **Worker-level shard distribution** (round-robin within a dataset family) +- **Per-sample transforms** via :class:`~.transforms.ActionTransformPipeline` +- **Weighted random fallback** when worker assignment is not active + +See ``docs/dataloader.md`` for the full design document. +""" + +from __future__ import annotations + +import gc +import random +import warnings +from collections.abc import Iterator, Mapping, Sequence +from typing import Any + +from torch.utils.data import Dataset, IterableDataset + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.transforms import ActionTransformPipeline +from cosmos_framework.data.vfm.utils import VIDEO_RES_SIZE_INFO + +_iterable_dataset_len_warning_suppressed = False + + +def _suppress_iterable_dataset_len_warning() -> None: + """Register a one-time filter for PyTorch's IterableDataset len() warning. + + The inner datasets may not implement ``__len__``, so the wrapper reports + ``len()=0``. PyTorch's iterator then warns on every ``__next__`` when + samples are fetched. This filter suppresses that warning. + """ + global _iterable_dataset_len_warning_suppressed + if _iterable_dataset_len_warning_suppressed: + return + _iterable_dataset_len_warning_suppressed = True + warnings.filterwarnings( + "ignore", + message="Length of IterableDataset.*was reported to be 0", + category=UserWarning, + module="torch.utils.data.dataloader", + ) + + +# --------------------------------------------------------------------------- +# Worker-side periodic garbage collection +# --------------------------------------------------------------------------- +# DataLoader workers running IterableDatasets are long-lived forked processes. +# Complex sample dictionaries (nested dicts, tensors, Arrow references) can +# create circular-reference chains that Python's reference counting alone +# cannot free. The generational GC *does* collect them eventually, but its +# default thresholds are too conservative for high-throughput data loading, +# causing RSS to grow monotonically until the node OOMs. +# +# Calling ``gc.collect()`` periodically inside the worker iteration loop +# eliminates the leak with negligible overhead (<1 ms per call vs ~6 s +# iteration time). +# +_GC_INTERVAL: int = 10 + + +def _maybe_gc(interval: int, count: int) -> int: + """Increment *count* and run ``gc.collect()`` every *interval* samples.""" + if interval <= 0: + return count + count += 1 + if count % interval == 0: + gc.collect() + return count + + +class ActionUnifiedIterableDataset(IterableDataset): + """Single IterableDataset wrapping all Action datasets. + + Handles worker-to-dataset assignment, shard distribution, and transforms. + + Args: + datasets: List of dicts, each with keys ``"name"`` (str identifier), + ``"dataset"`` (the dataset instance), and ``"ratio"`` (float + sampling weight). + transform: Transform pipeline applied to every yielded sample. + shard_across_workers: When ``True``, ranks are assigned to + dataset families via Hare-Niemeyer and workers get round-robin + shards. When ``False`` (default), every worker loads all + datasets and iterates with weighted random selection. + """ + + def __init__( + self, + datasets: list[dict[str, Any]], + transform: ActionTransformPipeline, + shard_across_workers: bool = False, + ) -> None: + super().__init__() + self._datasets = datasets + self._transform = transform + self._shard_across_workers = shard_across_workers + + # Set per-worker by assign_worker; None means not yet assigned. + self._dataset: Any | None = None + self._resolution: str | None = None # resolution for single-dataset path + self._family_name: str | None = None # set in assign_worker (single-family path) + self._sources_initialized = False + + # Backward compat: expose ``self.dataset`` pointing to the first + # inner dataset and ``self.transform`` exposing the pipeline + # (mirrors old TransformedIterableDataset interface). + self.dataset = datasets[0]["dataset"] if datasets else None + self.transform = transform + + # -- source initialization ------------------------------------------------ + + def _ensure_sources_registered(self) -> None: + if self._sources_initialized: + return + self._sources_initialized = True + for entry in self._datasets: + ds = entry["dataset"] + shard_roots = getattr(ds, "_all_shard_roots", []) + if shard_roots and hasattr(ds, "_register_sources"): + ds._register_sources() + + # -- backward-compat helpers ----------------------------------------------- + + def __len__(self) -> int: # type: ignore[override] + total = 0 + for entry in self._datasets: + ds = entry["dataset"] + try: + total += len(ds) # type: ignore[arg-type] + except TypeError: + pass + return total + + def __getattr__(self, name: str) -> Any: + """Forward attribute lookups to the first inner dataset.""" + if name.startswith("_") or not self._datasets: + raise AttributeError(name) + return getattr(self._datasets[0]["dataset"], name) + + # -- Hare-Niemeyer rank allocation ----------------------------------------- + + @staticmethod + def _compute_rank_ranges( + datasets: list[dict[str, Any]], + world_size: int, + ) -> list[tuple[int, int]]: + """Hare-Niemeyer allocation of ranks to datasets. + + Guarantees at least 1 rank per dataset, distributes the rest + proportionally. Returns a list of ``(start_rank, end_rank)`` ranges. + + Raises: + ValueError: If ``world_size < len(datasets)``. + """ + n_ds = len(datasets) + if world_size < n_ds: + raise ValueError(f"world_size ({world_size}) must be >= number of datasets ({n_ds})") + ratios = [d["ratio"] for d in datasets] + total = sum(ratios) + + # Hare-Niemeyer (largest-remainder) method: + # 1. Give every dataset a guaranteed minimum of 1 rank. + # 2. Distribute the leftover ranks proportionally to each dataset's + # ratio. Take the floor of each fractional allocation, then award + # the still-unassigned ranks one-by-one to datasets with the + # largest fractional remainders. + # Example: world_size=8, ratios=[3, 1] (2 datasets) + # remaining = 8 - 2 = 6 + # fractional = [6*3/4, 6*1/4] = [4.5, 1.5] + # floors = [4, 1], remainders = [0.5, 0.5], leftover = 1 + # award 1 extra to first dataset -> floors = [5, 1] + # counts = [1+5, 1+1] = [6, 2] + counts = [1] * n_ds + remaining = world_size - n_ds + if remaining > 0: + fractional = [remaining * r / total for r in ratios] + floors = [int(f) for f in fractional] + remainders = [f - fl for f, fl in zip(fractional, floors)] + leftover = remaining - sum(floors) + for idx in sorted(range(n_ds), key=lambda j: -remainders[j])[:leftover]: + floors[idx] += 1 + counts = [1 + f for f in floors] + + # Convert per-dataset counts into contiguous rank intervals. + # Example continued: counts=[6, 2] -> ranges=[(0,6), (6,8)] + # ranks 0..5 serve dataset 0, ranks 6..7 serve dataset 1. + ranges: list[tuple[int, int]] = [] + cursor = 0 + for c in counts: + ranges.append((cursor, cursor + c)) + cursor += c + return ranges + + @classmethod + def resolve_family_for_rank( + cls, + datasets: list[dict[str, Any]], + rank: int, + world_size: int, + ) -> tuple[int, str]: + """Return ``(family_index, family_name)`` for *rank* under Hare-Niemeyer. + + Same allocation as :meth:`assign_worker`, exposed so callers can + determine the family this rank will serve *before* worker init + (e.g. :class:`~.dataloaders.InfiniteDataLoader` patching DataLoader + kwargs per-family at construction time). + + Raises: + ValueError: If ``world_size < len(datasets)``. + IndexError: If ``rank`` is outside ``[0, world_size)``. + """ + if not (0 <= rank < world_size): + raise IndexError(f"rank {rank} outside [0, {world_size})") + rank_ranges = cls._compute_rank_ranges(datasets, world_size) + for i, (start_rank, end_rank) in enumerate(rank_ranges): + if start_rank <= rank < end_rank: + return i, datasets[i].get("name", f"family_{i}") + raise IndexError(f"rank {rank} not in any of {rank_ranges}") + + # -- worker assignment ----------------------------------------------------- + + def assign_worker( + self, + worker_id: int, + num_workers: int, + rank: int, + world_size: int, + ) -> None: + """Assign this worker to a dataset family and distribute shards. + + Called by the DataLoader's ``worker_init_fn`` (via + :func:`~.dataloaders.create_action_worker_init_fn`) -- not by the + dataset itself. + + Two-level assignment: + + 1. **Rank -> dataset family** (Hare-Niemeyer over *world_size* + ranks). Every rank is fully dedicated to one family. + 2. **Workers -> shards** (round-robin within the family's worker + pool). ``family_worker_id = rank_within_family * num_workers + + worker_id``. + + When ``shard_across_workers=False``: no assignment is performed. + Every worker loads all datasets and ``__iter__`` uses weighted + random selection. + """ + self._sources_initialized = True + if not self._shard_across_workers: + for entry in self._datasets: + ds = entry["dataset"] + shard_roots = getattr(ds, "_all_shard_roots", []) + if shard_roots and hasattr(ds, "_register_sources"): + ds._register_sources() + return + + rank_ranges = self._compute_rank_ranges(self._datasets, world_size) + + # Step 1: which dataset family does this rank belong to? + # ``rank_ranges`` is a list of (start_rank, end_rank) intervals -- one + # per dataset family -- produced by ``_compute_rank_ranges()`` above + # using Hare-Niemeyer allocation. The intervals are contiguous and + # non-overlapping, covering [0, world_size), so every rank belongs to + # exactly one family. + # + # We scan through the intervals to find the one containing this rank, + # then derive two values: + # - rank_within_family: this rank's 0-based position inside its + # family (used in Step 2 to build a globally unique worker id). + # - num_family_ranks: total number of ranks assigned to this family + # (used in Step 2 to compute the family's worker pool size). + # + # ``self._dataset`` is set to the matched family's dataset object so + # that ``__iter__`` only yields samples from this one dataset. + # + # Example with world_size=8, ratios=[3,1] -> ranges=[(0,6), (6,8)]: + # rank 3 -> family 0, rank_within_family=3, num_family_ranks=6 + # rank 6 -> family 1, rank_within_family=0, num_family_ranks=2 + num_family_ranks = 1 + rank_within_family = 0 + for i, (start_rank, end_rank) in enumerate(rank_ranges): + if start_rank <= rank < end_rank: + entry = self._datasets[i] + self._dataset = entry["dataset"] + self._resolution = entry["resolution"] + # Stash the family name so per-sample stamping in + # ``_transform_and_stamp`` can attach it as ``_action_family`` + # on every yielded sample. The outer ``JointDataLoader`` + # overwrites ``output["dataset_name"]`` with the *stream* + # name (e.g. ``"action_data"``), which would otherwise lose + # the per-family identity needed by the + # ``DetailedDataLoadingSpeedMonitor`` per-rank action + # metric. ``_action_family`` is a separate (underscore- + # prefixed) field that the joint loader passes through + # untouched via ``_update_output_batch``. + self._family_name = str(entry.get("name", f"family_{i}")) + rank_within_family = rank - start_rank + num_family_ranks = end_rank - start_rank + break + + # Step 2: distribute shards across workers within the family. + # Each rank spawns ``num_workers`` DataLoader workers (set by the + # DataLoader's ``num_workers`` arg). So the family's total worker + # pool is ``num_family_ranks * num_workers``. + # + # We flatten the 2D index (rank_within_family, worker_id) into a + # single linear ``family_worker_id`` so every worker in the family + # gets a globally unique id within that family: + # family_worker_id = rank_within_family * num_workers + worker_id + # + # Example: family has 3 ranks, each rank spawns 2 workers -> 6 total: + # rank_within_family=0: worker_id 0 -> fwid 0, worker_id 1 -> fwid 1 + # rank_within_family=1: worker_id 0 -> fwid 2, worker_id 1 -> fwid 3 + # rank_within_family=2: worker_id 0 -> fwid 4, worker_id 1 -> fwid 5 + # + # This linear id is then used for round-robin shard assignment below. + family_total_workers = num_family_ranks * num_workers + family_worker_id = rank_within_family * num_workers + worker_id + + # Round-robin assignment: worker k gets shards k, k+stride, k+2*stride, ... + # This ensures shards are evenly spread across the family's workers. + # + # When family_total_workers > num_shards, some workers get an empty + # list from range() (any worker with family_worker_id >= num_shards, + # since start >= stop). The ``if not my_shards`` guard catches this + # and falls back to ``family_worker_id % num_shards``, wrapping the + # worker around to an existing shard so it shares rather than idles. + # + # Example: AgiBotWorld with 190 shards and 256 family workers: + # Workers 0-189 -> each gets 1 unique shard via range() + # Workers 190-255 -> empty range, fallback to family_worker_id % 190, + # sharing a shard with an earlier worker. + # + # Multiple workers reading the same shard is fine because each worker + # has a different RNG seed (``seed + rank * 9999 + worker_id``), so + # they produce different sample orderings from the same underlying data. + my_shards: list[int] = [] + shard_roots = getattr(self._dataset, "_all_shard_roots", []) + if shard_roots and hasattr(self._dataset, "_register_sources"): + num_shards = len(shard_roots) + my_shards = list(range(family_worker_id, num_shards, family_total_workers)) + if not my_shards: + my_shards = [family_worker_id % num_shards] + self._dataset._register_sources(my_shards) + + # Per-rank/worker assignment dump so we can grep the log to see which + # action dataset family each rank serves and how shards distribute. + # Useful for diagnosing cross-rank memory skew (heavy families like + # camera_720 / av_480 land on the unlucky ranks). + family_name = self._datasets[i].get("name", f"family_{i}") if self._datasets else "" + log.info( + f"ActionUnifiedIterableDataset assign_worker: rank={rank}/{world_size} " + f"worker={worker_id}/{num_workers} family={family_name!r} " + f"resolution={self._resolution} family_ranks={num_family_ranks} " + f"family_worker_id={family_worker_id}/{family_total_workers} " + f"num_shards={len(my_shards)}", + rank0_only=False, + ) + + # -- iteration ------------------------------------------------------------- + + def _transform_and_stamp( + self, + sample: dict[str, Any], + resolution: str | None, + family_name: str | None = None, + ) -> dict[str, Any]: + """Apply ``self._transform`` and stamp ``_action_family`` on the sample. + + Single chokepoint for both ``__iter__`` (per-rank single-family path) + and ``_iter_all_datasets_weighted`` (debug all-families path). The + outer ``JointDataLoader`` overwrites ``"dataset_name"`` with the + *stream* name (e.g. ``"action_data"``), so the per-family identity + needed by + :class:`~.callbacks.dataloading_monitor.DetailedDataLoadingSpeedMonitor` + rides on the underscore-prefixed ``_action_family`` field instead; + ``_update_output_batch`` passes underscore-prefixed keys through + untouched. + """ + sample = self._transform(sample, resolution=resolution) + if family_name is not None: + sample["_action_family"] = family_name + return sample + + def _iter_all_datasets_weighted(self) -> Iterator[dict[str, Any]]: + """Iterate all datasets with weighted random selection. + + Used when ``shard_across_workers=False`` (every worker sees all + datasets) or as the ``num_workers=0`` fallback. + """ + iterators = [iter(d["dataset"]) for d in self._datasets] + ratios = [d["ratio"] for d in self._datasets] + total = sum(ratios) + weights = [r / total for r in ratios] + + gc_count = 0 + + while True: + chosen = random.choices(range(len(self._datasets)), weights=weights, k=1)[0] + resolution = self._datasets[chosen]["resolution"] + chosen_family_name = str(self._datasets[chosen].get("name", f"family_{chosen}")) + try: + sample = next(iterators[chosen]) + except StopIteration: + iterators[chosen] = iter(self._datasets[chosen]["dataset"]) + try: + sample = next(iterators[chosen]) + except StopIteration: + continue + yield self._transform_and_stamp(sample, resolution, chosen_family_name) + gc_count = _maybe_gc(_GC_INTERVAL, gc_count) + + def __iter__(self) -> Iterator[dict[str, Any]]: + if self._dataset is not None: + gc_count = 0 + for sample in self._dataset: + yield self._transform_and_stamp(sample, self._resolution, self._family_name) + gc_count = _maybe_gc(_GC_INTERVAL, gc_count) + return + + if not self._shard_across_workers: + self._ensure_sources_registered() + yield from self._iter_all_datasets_weighted() + return + + # num_workers=0 fallback (shard_across_workers=True but no worker + # processes exist, so assign_worker was never called). + log.warning( + "ActionUnifiedIterableDataset: num_workers=0 fallback — " + "loading ALL datasets in main process. Use only for debugging." + ) + self._ensure_sources_registered() + yield from self._iter_all_datasets_weighted() + + +class MapToIterableAdapter(IterableDataset): + """Wraps a map-style ``Dataset`` as an ``IterableDataset``. + + Each iteration yields a sample from a uniformly random index, using + ``random.randint`` for O(1) time and zero extra memory. The per-worker + RNG seed (set by :func:`~.dataloaders.create_action_worker_init_fn`) ensures + different DataLoader workers produce different random sequences. + + Args: + dataset: A map-style ``Dataset`` with ``__len__`` and ``__getitem__``. + """ + + def __init__(self, dataset: Dataset) -> None: + super().__init__() + self.dataset = dataset + + def __len__(self) -> int: # type: ignore[override] + return len(self.dataset) # type: ignore[arg-type] + + def __iter__(self) -> Iterator: + n = len(self.dataset) # type: ignore[arg-type] + while True: + yield self.dataset[random.randint(0, n - 1)] + + def __getattr__(self, name: str) -> Any: + """Forward attribute lookups to the inner dataset for transparency.""" + if name == "dataset": + raise AttributeError(name) + return getattr(self.dataset, name) + + +def dataset_entry( + name: str, + dataset: Dataset | IterableDataset, + ratio: float = 1.0, + resolution: str | None = None, +) -> dict: + """Factory for a single dataset descriptor used inside ``wrap_dataset``. + + Wrapping each entry with ``LazyCall(dataset_entry)(...)`` gives it a + ``_target_`` so that ``instantiate`` recurses into the nested dataset + config automatically. + + Args: + name: Identifier for the dataset. + dataset: The dataset instance. + ratio: Sampling weight. Defaults to 1.0. + resolution: Optional resolution tier (e.g. ``"256"``, ``"480"``) for + this dataset. When ``None``, falls back to ``wrap_dataset``'s + global ``resolution`` (which may be ``None`` for auto-detect). + """ + return {"name": name, "dataset": dataset, "ratio": ratio, "resolution": resolution} + + +def wrap_dataset( + list_of_datasets: Sequence[dict] | list[dict] | Dataset | IterableDataset, + resolution: str | None = None, + pad_keys: list[str] | None = None, + keep_aspect_ratio: bool = True, + tokenizer_config: dict | None = None, + cfg_dropout_rate: float = 0.0, + caption_key: str = "ai_caption", + text_token_key: str = "text_token_ids", + video_temporal_downsample: int = 4, + max_action_dim: int = 32, + shard_across_workers: bool = False, + action_channel_masking: bool = True, + append_duration_fps_timestamps: bool = True, + append_resolution_info: bool = True, + append_idle_frames: bool = False, + idle_frames_dropout: float = 0.05, + format_prompt_as_json: bool = False, +) -> ActionUnifiedIterableDataset: + """Factory that wraps one or more datasets with the Action transform pipeline. + + ``list_of_datasets`` accepts either: + + * A **list of dicts**, where each dict has the keys: + - ``name`` (``str``): identifier for the dataset. + - ``dataset`` (``Dataset | IterableDataset``): the dataset instance. + - ``ratio`` (``float``, optional): sampling weight. Defaults to ``1``. + - ``resolution`` (``str | None``, optional): resolution tier for this + dataset. When missing, falls back to ``wrap_dataset``'s global + ``resolution`` (which may be ``None`` for auto-detect). + * A **single** ``Dataset`` or ``IterableDataset`` for backward compatibility + (auto-wrapped as ``[{"name": "default", "dataset": , "ratio": 1}]``). + + Map-style datasets are automatically wrapped with + :class:`MapToIterableAdapter` so the returned dataset is always an + ``IterableDataset``. This means callers can mix map-style and + iterable-style datasets freely. + + Args: + list_of_datasets: The dataset(s) to wrap. + resolution: Resolution tier key (e.g. ``"256"``, ``"480"``, ``"720"``). + Spatial dimensions are resized and reflection-padded to the closest + predefined target from ``VIDEO_RES_SIZE_INFO``. When ``None``, the + tier is auto-detected per sample via ``get_vision_data_resolution``. + Defaults to ``None``. + pad_keys: Data-dict keys whose values should be resized and padded. Pass + an empty list or ``None`` to disable padding. Defaults to ``["video"]``. + tokenizer_config: A lazy-instantiable config dict for the VLM tokenizer. When + ``None``, text tokenization is skipped. Defaults to ``None``. + cfg_dropout_rate: Probability of replacing the caption with an empty string for + classifier-free guidance. Defaults to ``0.0``. + caption_key: The data-dict key that contains the input caption string. + Defaults to ``"ai_caption"``. + text_token_key: The data-dict key where tokenized text IDs will be stored. + Defaults to ``"text_token_ids"``. + video_temporal_downsample: Temporal downsampling factor of the video tokenizer. + Used when building a ``SequencePlan`` for ``"inverse_dynamics"`` mode. + Defaults to 4. + max_action_dim: Target action dimension to pad to. The ``"action"`` tensor + in every sample is padded along its last dimension. Defaults to 32. + action_channel_masking: When ``True`` (default), stores the original action + dimension in ``"raw_action_dim"`` so the model masks loss/noise/velocity + on padded channels. Set to ``False`` to disable (original behavior). + shard_across_workers: When ``True``, the returned dataset + supports rank-level dataset assignment and worker-level shard + distribution via ``assign_worker()``. When ``False`` (default), + every worker iterates all datasets with weighted random selection. + append_duration_fps_timestamps: Whether to append duration and FPS metadata to the + caption before tokenization. Defaults to ``True``. + append_resolution_info: Whether to append resolution metadata to the + caption before tokenization. Defaults to ``True``. + append_idle_frames: Whether to append the idle-frame count out of the + total action frames (Pi0.7-style metadata) to the caption before + tokenization. The dataset is responsible for populating + ``data_dict["idle_frames"]``; samples without it are silently + skipped. Defaults to ``False`` so existing experiments are + unaffected. + idle_frames_dropout: Per-field dropout rate for the idle-frame segment. + Independent of ``cfg_dropout_rate`` (which empties the whole + caption). Defaults to 0.05. + format_prompt_as_json: Whether to replace the plain text prompt with a + structured JSON-compatible dictionary before tokenization. Defaults + to ``False``. + + Returns: + A :class:`ActionUnifiedIterableDataset` wrapping the dataset(s) with the + configured transforms applied. + + Raises: + TypeError: If the dataset(s) are not ``Dataset`` or ``IterableDataset``. + ValueError: If ``list_of_datasets`` is an empty list. + """ + if pad_keys is None: + pad_keys = ["video"] + + # ------------------------------------------------------------------ + # Backward compatibility: single dataset -> list-of-dicts + # ------------------------------------------------------------------ + if isinstance(list_of_datasets, (Dataset, IterableDataset)): + list_of_datasets = [{"name": "default", "dataset": list_of_datasets, "ratio": 1}] + + if ( + not isinstance(list_of_datasets, Sequence) + or isinstance(list_of_datasets, (str, bytes)) + or len(list_of_datasets) == 0 + ): + raise ValueError( + "list_of_datasets must be a non-empty list/sequence of dicts or a single Dataset/IterableDataset, " + f"got {type(list_of_datasets).__name__}" + ) + + # ------------------------------------------------------------------ + # Parse list-of-dicts, wrapping map-style datasets with + # MapToIterableAdapter so every dataset is iterable. Compute effective + # resolution per entry (per-entry overrides global). + # ------------------------------------------------------------------ + datasets: list[dict] = [] + for entry in list_of_datasets: + if not isinstance(entry, Mapping): + raise TypeError(f"Each entry in list_of_datasets must be a dict/mapping, got {type(entry).__name__}") + name: str = entry["name"] + dataset: Dataset | IterableDataset = entry["dataset"] + ratio: float = float(entry.get("ratio", 1)) + resolution: str | None = entry.get("resolution", None) + if resolution is not None: + res_key = str(resolution) if isinstance(resolution, int) else resolution + if res_key not in VIDEO_RES_SIZE_INFO: + raise ValueError( + f"Resolution '{resolution}' for dataset '{name}' not found in VIDEO_RES_SIZE_INFO. " + f"Available: {list(VIDEO_RES_SIZE_INFO.keys())}" + ) + if not isinstance(dataset, IterableDataset): + dataset = MapToIterableAdapter(dataset) + datasets.append({"name": name, "dataset": dataset, "ratio": ratio, "resolution": resolution}) + + # ------------------------------------------------------------------ + # Build the transform pipeline (resolution supplied at call time) + # ------------------------------------------------------------------ + transform = ActionTransformPipeline( + pad_keys=pad_keys, + keep_aspect_ratio=keep_aspect_ratio, + tokenizer_config=tokenizer_config, + cfg_dropout_rate=cfg_dropout_rate, + caption_key=caption_key, + text_token_key=text_token_key, + video_temporal_downsample=video_temporal_downsample, + max_action_dim=max_action_dim, + action_channel_masking=action_channel_masking, + append_duration_fps_timestamps=append_duration_fps_timestamps, + append_resolution_info=append_resolution_info, + append_idle_frames=append_idle_frames, + idle_frames_dropout=idle_frames_dropout, + format_prompt_as_json=format_prompt_as_json, + ) + + _suppress_iterable_dataset_len_warning() + + return ActionUnifiedIterableDataset( + datasets=datasets, + transform=transform, + shard_across_workers=shard_across_workers, + ) diff --git a/cosmos_framework/data/vfm/action/unified_dataset_test.py b/cosmos_framework/data/vfm/action/unified_dataset_test.py new file mode 100644 index 0000000..87d196a --- /dev/null +++ b/cosmos_framework/data/vfm/action/unified_dataset_test.py @@ -0,0 +1,537 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from __future__ import annotations + +import itertools +from typing import Any +from unittest.mock import MagicMock, patch + +import pytest +import torch +from torch.utils.data import IterableDataset + +from cosmos_framework.data.vfm.action.unified_dataset import ActionUnifiedIterableDataset +from cosmos_framework.model.vfm.omni_mot_model_test import _maybe_init_distributed + +""" +pytest -v -s cosmos_framework/data/vfm/action/unified_dataset_test.py --L0 +pytest -v -s cosmos_framework/data/vfm/action/unified_dataset_test.py::test_hare_niemeyer --L0 +pytest -v -s cosmos_framework/data/vfm/action/unified_dataset_test.py::test_assign_worker --L0 +pytest -v -s cosmos_framework/data/vfm/action/unified_dataset_test.py::test_iter --L0 +pytest -v -s cosmos_framework/data/vfm/action/unified_dataset_test.py::test_backward_compat --L0 +pytest -v -s cosmos_framework/data/vfm/action/unified_dataset_test.py::test_worker_init_fn --L0 +pytest -v -s cosmos_framework/data/vfm/action/unified_dataset_test.py::test_register_sources --L0 +torchrun --nproc_per_node=4 --standalone -m pytest -v -s cosmos_framework/data/vfm/action/unified_dataset_test.py::test_distributed_4rank_shard_assignment --L1 +""" + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- +# Dataset entries are dicts with "name", "dataset", "ratio", and optionally +# "resolution". Resolution is always in the dict (use None when not needed); +# it is not passed to the underlying dataset or to wrap_dataset. + + +def _make_sample( + video_shape: tuple[int, ...] = (3, 17, 256, 256), + action_dim: int = 10, + mode: str = "policy", +) -> dict: + """Minimal sample dict compatible with ActionTransformPipeline.""" + action_length = video_shape[1] - 1 + return { + "video": torch.randint(0, 256, video_shape, dtype=torch.uint8), + "action": torch.randn(action_length, action_dim), + "ai_caption": "A test caption.", + "mode": mode, + "domain_id": torch.tensor(0, dtype=torch.long), + } + + +class _TaggedIterableDataset(IterableDataset): + """Iterable dataset that tags samples with a ``dataset_tag``.""" + + def __init__(self, tag: str, length: int = 50) -> None: + self._tag = tag + self._length = length + + def __len__(self) -> int: + return self._length + + def __iter__(self): + while True: + sample = _make_sample() + sample["dataset_tag"] = self._tag + yield sample + + +class _FiniteIterableDataset(IterableDataset): + """Iterable dataset that yields a fixed number of samples then stops.""" + + def __init__(self, tag: str, count: int = 3) -> None: + self._tag = tag + self._count = count + + def __len__(self) -> int: + return self._count + + def __iter__(self): + for _ in range(self._count): + sample = _make_sample() + sample["dataset_tag"] = self._tag + yield sample + + +class _ShardableDataset(IterableDataset): + """Dataset with ``_all_shard_roots`` and ``_register_sources`` for testing.""" + + def __init__(self, tag: str, num_shards: int = 10) -> None: + self._tag = tag + self._all_shard_roots = [{"shard_id": i} for i in range(num_shards)] + self._registered_shards: list[int] = [] + + def _register_sources(self, indices: list[int] | None = None) -> None: + if indices is None: + indices = list(range(len(self._all_shard_roots))) + self._registered_shards = indices + + def __iter__(self): + while True: + sample = _make_sample() + sample["dataset_tag"] = self._tag + sample["registered_shards"] = list(self._registered_shards) + yield sample + + +def _identity_transform(data_dict: dict, resolution: str | None = None) -> dict: + return data_dict + + +def _make_unified( + datasets: list[dict[str, Any]], + shard_across_workers: bool = True, +) -> ActionUnifiedIterableDataset: + transform = MagicMock(side_effect=_identity_transform) + return ActionUnifiedIterableDataset( + datasets=datasets, + transform=transform, + shard_across_workers=shard_across_workers, + ) + + +# --------------------------------------------------------------------------- +# Hare-Niemeyer rank allocation +# --------------------------------------------------------------------------- + + +@pytest.mark.L0 +def test_hare_niemeyer_basic_proportional(): + """4 datasets, 8 ranks with ratios [4,2,1,1]. + + Hare-Niemeyer with 1-rank guarantee: + - Each dataset gets 1 guaranteed rank, leaving 4 to distribute. + - Remaining 4 allocated proportionally: A=2.0, B=1.0, C=0.5, D=0.5 + - Floors: [2,1,0,0], leftover=1 goes to C (highest remainder). + - Final: [3,2,2,1]. + """ + datasets = [ + {"name": "A", "ratio": 4}, + {"name": "B", "ratio": 2}, + {"name": "C", "ratio": 1}, + {"name": "D", "ratio": 1}, + ] + ranges = ActionUnifiedIterableDataset._compute_rank_ranges(datasets, 8) + counts = [end - start for start, end in ranges] + print(f" Counts: {counts}") + assert counts == [3, 2, 2, 1] + assert ranges[0] == (0, 3) + assert ranges[1] == (3, 5) + assert ranges[2] == (5, 7) + assert ranges[3] == (7, 8) + print(" Success: basic proportional allocation correct") + + +@pytest.mark.L0 +def test_hare_niemeyer_minimum_one_rank(): + """Even with very skewed ratios, each dataset gets at least 1 rank.""" + datasets = [ + {"name": "big", "ratio": 100}, + {"name": "tiny", "ratio": 1}, + ] + ranges = ActionUnifiedIterableDataset._compute_rank_ranges(datasets, 3) + counts = [end - start for start, end in ranges] + print(f" Counts: {counts}") + assert all(c >= 1 for c in counts), f"All datasets must get >= 1 rank: {counts}" + assert sum(counts) == 3 + print(" Success: minimum 1-rank guarantee holds") + + +@pytest.mark.L0 +def test_hare_niemeyer_insufficient_ranks(): + """world_size < num_datasets should raise ValueError.""" + datasets = [{"name": f"ds_{i}", "ratio": 1} for i in range(5)] + with pytest.raises(ValueError, match="must be >= number of datasets"): + ActionUnifiedIterableDataset._compute_rank_ranges(datasets, 3) + print(" Success: insufficient ranks raises ValueError") + + +@pytest.mark.L0 +def test_hare_niemeyer_total_matches_world_size(): + """Sum of allocated ranks always equals world_size.""" + for world_size in [4, 8, 16, 32, 64]: + datasets = [ + {"name": "A", "ratio": 5}, + {"name": "B", "ratio": 3}, + {"name": "C", "ratio": 1}, + ] + ranges = ActionUnifiedIterableDataset._compute_rank_ranges(datasets, world_size) + total = sum(end - start for start, end in ranges) + assert total == world_size, f"world_size={world_size}: total={total}" + print(f" world_size={world_size}: total={total} OK") + print(" Success: totals match for all world sizes") + + +@pytest.mark.L0 +def test_hare_niemeyer_contiguous_non_overlapping(): + """Ranges are contiguous and non-overlapping.""" + datasets = [ + {"name": "A", "ratio": 4}, + {"name": "B", "ratio": 2}, + {"name": "C", "ratio": 1}, + ] + ranges = ActionUnifiedIterableDataset._compute_rank_ranges(datasets, 10) + print(f" Ranges: {ranges}") + for i in range(1, len(ranges)): + assert ranges[i][0] == ranges[i - 1][1], f"Gap between range {i - 1} and {i}" + print(" Success: ranges are contiguous") + + +# --------------------------------------------------------------------------- +# assign_worker +# --------------------------------------------------------------------------- + + +@pytest.mark.L0 +def test_assign_worker_dataset_assignment(): + """Each rank is assigned to the correct dataset family.""" + ds_a = _ShardableDataset("A", num_shards=20) + ds_b = _ShardableDataset("B", num_shards=5) + datasets = [ + {"name": "A", "dataset": ds_a, "ratio": 3, "resolution": None}, + {"name": "B", "dataset": ds_b, "ratio": 1, "resolution": None}, + ] + + # 4 ranks, ratios [3,1]: guarantee 1 each, remaining 2 -> A=1.5 B=0.5 + # floors [1,0], leftover 1 -> A. Final: [2,2] -> A=[0,1], B=[2,3] + print(" Testing rank 0 -> dataset A...") + unified = _make_unified(datasets) + unified.assign_worker(worker_id=0, num_workers=2, rank=0, world_size=4) + assert unified._dataset is ds_a + + print(" Testing rank 3 -> dataset B...") + unified2 = _make_unified( + [ + {"name": "A", "dataset": _ShardableDataset("A", num_shards=20), "ratio": 3, "resolution": None}, + {"name": "B", "dataset": ds_b, "ratio": 1, "resolution": None}, + ] + ) + unified2.assign_worker(worker_id=0, num_workers=2, rank=3, world_size=4) + assert unified2._dataset is ds_b + print(" Success: rank-to-dataset assignment correct") + + +@pytest.mark.L0 +def test_assign_worker_shard_round_robin(): + """Shards are distributed round-robin across the family's workers.""" + print(" 1 rank, 4 workers, 10 shards...") + ds = _ShardableDataset("A", num_shards=10) + datasets = [{"name": "A", "dataset": ds, "ratio": 1, "resolution": None}] + unified = _make_unified(datasets) + + # Worker 0 -> shards [0, 4, 8] + unified.assign_worker(worker_id=0, num_workers=4, rank=0, world_size=1) + print(f" Worker 0 shards: {ds._registered_shards}") + assert ds._registered_shards == [0, 4, 8] + + # Worker 1 -> shards [1, 5, 9] + ds2 = _ShardableDataset("A", num_shards=10) + datasets2 = [{"name": "A", "dataset": ds2, "ratio": 1, "resolution": None}] + unified2 = _make_unified(datasets2) + unified2.assign_worker(worker_id=1, num_workers=4, rank=0, world_size=1) + print(f" Worker 1 shards: {ds2._registered_shards}") + assert ds2._registered_shards == [1, 5, 9] + print(" Success: round-robin shard distribution correct") + + +@pytest.mark.L0 +def test_assign_worker_shard_wrap_around(): + """When workers > shards, workers wrap around.""" + ds = _ShardableDataset("A", num_shards=3) + datasets = [{"name": "A", "dataset": ds, "ratio": 1, "resolution": None}] + unified = _make_unified(datasets) + + # Worker 5 of 8 -> family_worker_id = 5, range(5, 3, 8) empty -> fallback [5 % 3] = [2] + unified.assign_worker(worker_id=5, num_workers=8, rank=0, world_size=1) + print(f" Worker 5 shards: {ds._registered_shards}") + assert ds._registered_shards == [2] + print(" Success: wrap-around fallback correct") + + +@pytest.mark.L0 +def test_assign_worker_no_shard_roots(): + """Datasets without _all_shard_roots skip shard distribution (Case C). + + Most datasets (LIBERO, PushT, CameraDataset, AVDataset) don't populate + _all_shard_roots. They are still assigned to a rank family via + Hare-Niemeyer, but every worker in the family iterates the full dataset + with different RNG seeds — no round-robin shard splitting occurs. + See dataloader.md "Case C" for details. + """ + ds = _TaggedIterableDataset("plain") + datasets = [{"name": "plain", "dataset": ds, "ratio": 1, "resolution": None}] + unified = _make_unified(datasets) + + unified.assign_worker(worker_id=0, num_workers=4, rank=0, world_size=1) + assert unified._dataset is ds + assert not hasattr(ds, "_registered_shards") + print(" Success: no shard roots -> no registration") + + +@pytest.mark.L0 +def test_assign_worker_unsharded_mode(): + """shard_across_workers=False: every worker loads all datasets (default). + + When shard_across_workers=False (the default), Hare-Niemeyer rank + assignment is skipped entirely. Every worker on every rank sees all + datasets, and __iter__ uses weighted random selection based on the + ratio values. This is the safe default that works without any dataset + changes — opt in to shard_across_workers=True for large sharded + datasets like AgiBotWorld. + """ + ds_a = _ShardableDataset("A", num_shards=10) + ds_b = _ShardableDataset("B", num_shards=5) + datasets = [ + {"name": "A", "dataset": ds_a, "ratio": 1}, + {"name": "B", "dataset": ds_b, "ratio": 1}, + ] + unified = _make_unified(datasets, shard_across_workers=False) + + unified.assign_worker(worker_id=0, num_workers=4, rank=0, world_size=4) + + assert unified._dataset is None + assert ds_a._registered_shards == list(range(10)) + assert ds_b._registered_shards == list(range(5)) + print(" Success: unsharded mode registers all shards") + + +@pytest.mark.L0 +def test_assign_worker_multi_rank_family(): + """Shards distribute across workers on multiple ranks of the same family.""" + print(" 2 ranks, 4 workers each, 20 shards -> family_total_workers = 8") + + # Rank 0, worker 2 -> family_worker_id = 0*4+2 = 2 -> shards [2, 10, 18] + ds = _ShardableDataset("A", num_shards=20) + datasets = [{"name": "A", "dataset": ds, "ratio": 1, "resolution": None}] + unified = _make_unified(datasets) + unified.assign_worker(worker_id=2, num_workers=4, rank=0, world_size=2) + print(f" Rank 0, Worker 2 shards: {ds._registered_shards}") + assert ds._registered_shards == [2, 10, 18] + + # Rank 1, worker 1 -> family_worker_id = 1*4+1 = 5 -> shards [5, 13] + ds2 = _ShardableDataset("A", num_shards=20) + datasets2 = [{"name": "A", "dataset": ds2, "ratio": 1, "resolution": None}] + unified2 = _make_unified(datasets2) + unified2.assign_worker(worker_id=1, num_workers=4, rank=1, world_size=2) + print(f" Rank 1, Worker 1 shards: {ds2._registered_shards}") + assert ds2._registered_shards == [5, 13] + print(" Success: multi-rank family distribution correct") + + +# --------------------------------------------------------------------------- +# create_action_worker_init_fn +# --------------------------------------------------------------------------- + + +@pytest.mark.L0 +def test_worker_init_fn_calls_assign_worker(): + """worker_init_fn calls dataset.assign_worker with correct args.""" + from cosmos_framework.data.vfm.action.dataloaders import create_action_worker_init_fn + + seed = 42 + + mock_dataset = MagicMock(spec=ActionUnifiedIterableDataset) + mock_info = MagicMock() + mock_info.dataset = mock_dataset + mock_info.num_workers = 8 + + with patch("cosmos_framework.data.vfm.action.dataloaders.distributed") as mock_dist: + mock_dist.get_rank.return_value = 3 + mock_dist.get_world_size.return_value = 8 + fn = create_action_worker_init_fn(seed) + with patch("torch.utils.data.get_worker_info", return_value=mock_info): + fn(worker_id=5) + + mock_dataset.assign_worker.assert_called_once_with(5, 8, 3, 8) + print(" Success: assign_worker called with (worker_id=5, num_workers=8, rank=3, world_size=8)") + + +# --------------------------------------------------------------------------- +# BaseActionLeRobotDataset._register_sources +# --------------------------------------------------------------------------- + + +@pytest.mark.L0 +def test_register_sources_all(): + """_register_sources() with no args registers all shard roots.""" + from cosmos_framework.data.vfm.action.cosmos3_action_lerobot import BaseActionLeRobotDataset + + ds = MagicMock(spec=BaseActionLeRobotDataset) + ds._all_shard_roots = ["/data/a", "/data/b", "/data/c"] + ds._delta_timestamps = {"obs": [0.0, 0.1]} + ds._tolerance_s = 0.01 + ds._enable_fast_init = False + BaseActionLeRobotDataset._register_sources(ds, indices=None) + print(f" _register_source called {ds._register_source.call_count} times") + assert ds._register_source.call_count == 3 + print(" Success: all 3 shard roots registered") + + +@pytest.mark.L0 +def test_register_sources_subset(): + """_register_sources(indices=[1]) registers only the specified shard.""" + from cosmos_framework.data.vfm.action.cosmos3_action_lerobot import BaseActionLeRobotDataset + + ds = MagicMock(spec=BaseActionLeRobotDataset) + ds._all_shard_roots = ["/data/a", "/data/b"] + ds._delta_timestamps = {"obs": [0.0]} + ds._tolerance_s = 0.01 + ds._enable_fast_init = False + BaseActionLeRobotDataset._register_sources(ds, indices=[1]) + ds._register_source.assert_called_once_with( + root="/data/b", + delta_timestamps={"obs": [0.0]}, + tolerance_s=0.01, + dataset_label="b", + prefetched_meta=None, + ) + print(" Success: only shard [1] registered") + + +@pytest.mark.L0 +def test_register_sources_empty(): + """_register_sources() on empty _all_shard_roots is a no-op.""" + from cosmos_framework.data.vfm.action.cosmos3_action_lerobot import BaseActionLeRobotDataset + + ds = MagicMock(spec=BaseActionLeRobotDataset) + ds._all_shard_roots = [] + BaseActionLeRobotDataset._register_sources(ds, indices=None) + ds._register_source.assert_not_called() + print(" Success: empty shard roots -> no calls") + + +# --------------------------------------------------------------------------- +# Distributed: 4-rank end-to-end test +# --------------------------------------------------------------------------- + + +@pytest.mark.L1 +def test_distributed_4rank_shard_assignment(): + """End-to-end test: 4 ranks, 2 datasets, 2 workers per rank via InfiniteDataLoader. + + Verifies that across 4 real distributed ranks: + - Hare-Niemeyer assigns ranks to the correct dataset family + - assign_worker is called by create_action_worker_init_fn inside DataLoader workers + - Each rank's workers only yield samples from the assigned dataset + - Shardable datasets get round-robin shard distribution + + Run with: + torchrun --nproc_per_node=4 --standalone -m pytest -v -s \\ + cosmos_framework/data/vfm/action/unified_dataset_test.py::test_distributed_4rank_shard_assignment --L1 + """ + _maybe_init_distributed() + + world_size = torch.distributed.get_world_size() + if world_size != 4: + pytest.skip(f"This test requires exactly 4 ranks (got world_size={world_size})") + + rank = torch.distributed.get_rank() + + from cosmos_framework.data.vfm.action.dataloaders import InfiniteDataLoader, create_action_worker_init_fn + + # Dataset A: shardable with 20 shards, ratio=3 + # Dataset B: plain iterable (no shards), ratio=1 + # Hare-Niemeyer with 4 ranks, ratios [3,1]: + # guarantee 1 each -> remaining 2 -> A=1.5 B=0.5 + # floors [1,0], leftover 1 -> A (tied remainder, lower index wins) + # Final: A=3 ranks [0,1,2], B=1 rank [3] + ds_a = _ShardableDataset("A", num_shards=20) + ds_b = _TaggedIterableDataset("B", length=100) + + datasets = [ + {"name": "A", "dataset": ds_a, "ratio": 3, "resolution": None}, + {"name": "B", "dataset": ds_b, "ratio": 1, "resolution": None}, + ] + + transform = MagicMock(side_effect=_identity_transform) + unified = ActionUnifiedIterableDataset( + datasets=datasets, + transform=transform, + shard_across_workers=True, + ) + + num_workers = 2 + loader = InfiniteDataLoader( + dataset=unified, + batch_size=1, + num_workers=num_workers, + worker_init_fn=create_action_worker_init_fn(seed=42), + ) + + # Consume samples and verify dataset assignment + n_samples = 20 + samples = list(itertools.islice(loader, n_samples)) + assert len(samples) == n_samples, f"[Rank {rank}] Expected {n_samples} batches, got {len(samples)}" + + # Each sample is a collated batch dict; extract the dataset_tag + tags = set() + for batch in samples: + if "dataset_tag" in batch: + tag = batch["dataset_tag"] + if isinstance(tag, list): + tags.update(tag) + else: + tags.add(tag) + + # Ranks 0-2 -> dataset A, rank 3 -> dataset B + if rank in (0, 1, 2): + expected_tag = "A" + else: + expected_tag = "B" + + print(f"[Rank {rank}] Tags seen: {tags}, expected: {{{expected_tag}}}") + assert tags == {expected_tag}, f"[Rank {rank}] Expected only '{expected_tag}' samples, got tags: {tags}" + + # For shardable dataset A (ranks 0-2): verify shards were distributed + if rank in (0, 1, 2): + # After workers ran, ds_a._registered_shards should have been set + # by _register_sources in the worker processes. Since workers are + # separate processes, we can't inspect ds_a directly. Instead we + # check that the "registered_shards" key in the yielded samples + # contains a non-empty subset. + for batch in samples: + if "registered_shards" in batch: + shards = batch["registered_shards"] + if isinstance(shards, torch.Tensor): + shards = shards.tolist() + elif isinstance(shards, list) and len(shards) > 0 and isinstance(shards[0], torch.Tensor): + shards = [s.item() for s in shards] + print(f"[Rank {rank}] Sample registered_shards: {shards}") + assert len(shards) > 0, f"[Rank {rank}] Expected non-empty registered_shards" + assert all(0 <= s < 20 for s in shards), f"[Rank {rank}] Shard indices out of range: {shards}" + break + + # Barrier to ensure all ranks completed + torch.distributed.barrier() + print(f"[Rank {rank}] Success: distributed shard assignment test passed!") diff --git a/cosmos_framework/data/vfm/action/urdf_visualizer/G1_omnipicker_calibrated.urdf b/cosmos_framework/data/vfm/action/urdf_visualizer/G1_omnipicker_calibrated.urdf new file mode 100644 index 0000000..bd83679 --- /dev/null +++ b/cosmos_framework/data/vfm/action/urdf_visualizer/G1_omnipicker_calibrated.urdf @@ -0,0 +1,1350 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/cosmos_framework/data/vfm/action/urdf_visualizer/__init__.py b/cosmos_framework/data/vfm/action/urdf_visualizer/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/data/vfm/action/urdf_visualizer/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/data/vfm/action/urdf_visualizer/compare_viewer.py b/cosmos_framework/data/vfm/action/urdf_visualizer/compare_viewer.py new file mode 100755 index 0000000..1cbc9e6 --- /dev/null +++ b/cosmos_framework/data/vfm/action/urdf_visualizer/compare_viewer.py @@ -0,0 +1,1019 @@ +#!/usr/bin/env python +# Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# NVIDIA CORPORATION & AFFILIATES and its licensors retain all intellectual property +# and proprietary rights in and to this software and related materials and are subject +# to, without limitation, any intellectual property or other proprietary rights that +# may be applicable to such software and related materials, and any terms and +# conditions to which such software and related materials are subject. +"""Overlay GT and predicted action trajectories from eval outputs. + +Loads ground truth (batch_data.safetensors or output.safetensors) alongside +predicted (output.safetensors) action tensors directory and renders both +trajectories overlaid in the same viser scene. + +Color scheme: GT trajectories in green, pred trajectories in red. The +robot mesh is driven only by the pred trajectory. + +Usage — browse an eval dir: + uv run python cosmos_framework/data/vfm/action/urdf_visualizer/compare_viewer.py \\ + --eval-dir /mnt/cosmos-eval/ --share + +Usage — single pair: + uv run python cosmos_framework/data/vfm/action/urdf_visualizer/compare_viewer.py \\ + --gt ground_truth/bridge/inverse_dynamics/0/batch_data.safetensors \\ + --pred outputs/bridge/inverse_dynamics/0/output.safetensors \\ + --dataset bridge + +Dependencies: viser safetensors numpy cv2 torch (mujoco pin for robot meshes) +""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +import time as _time +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Callable + +import numpy as np + +_REPO_ROOT = str(Path(__file__).resolve().parents[6]) +if _REPO_ROOT not in sys.path: + sys.path.insert(0, _REPO_ROOT) + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.urdf_visualizer.unified_action import ( + ActionFormat, + SceneState, + build_scene_state, + to_unified, +) +from cosmos_framework.data.vfm.action.urdf_visualizer.unified_renderer import UnifiedRenderer +from cosmos_framework.data.vfm.action.urdf_visualizer.viewer import ( + DatasetEntry, + _build_datasets, + _load_symbol, +) + +# ── Minimal dataset entries ──────────────────────────── + + +def _build_minimal_entries() -> dict[str, DatasetEntry]: + """Hardcoded subset of the registry + + Source ``to_opencv`` matrices from each dataset module + """ + # All matrices inlined to avoid importing dataset modules that pull in lerobot, + # which is not available on CPU Lepton jobs. + _BRIDGE_TO_OPENCV = np.array([[0, 0, 1], [-1, 0, 0], [0, -1, 0]], dtype=np.float32) + _DROID_TO_OPENCV = np.array([[0.0, -1.0, 0.0], [1.0, 0.0, 0.0], [0.0, 0.0, 1.0]], dtype=np.float32) + _GOOGLE_ROBOT_TO_OPENCV = np.array([[0, 1, 0], [-1, 0, 0], [0, 0, 1]], dtype=np.float32) + _ROBOMIND_FRANKA_TO_OPENCV = np.array( + [[0.0, -1.0, 0.0, 0.0], [1.0, 0.0, 0.0, 0.0], [0.0, 0.0, 1.0, 0.0], [0.0, 0.0, 0.0, 1.0]], + dtype=np.float32, + ) + AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST = { + "left_wrist": np.array([[0.0, 1.0, 0.0], [-1.0, 0.0, 0.0], [0.0, 0.0, 1.0]], dtype=np.float32), + "right_wrist": np.array([[0.0, -1.0, 0.0], [1.0, 0.0, 0.0], [0.0, 0.0, 1.0]], dtype=np.float32), + } + + franka_to_opencv = _ROBOMIND_FRANKA_TO_OPENCV[:3, :3] + eye3 = np.eye(3, dtype=np.float32) + + def E(**kw: Any) -> DatasetEntry: + return DatasetEntry(action_format=kw.pop("action_format", ActionFormat.SINGLE_ARM_10D), **kw) + + return { + "bridge": E(name="bridge", robot_name="widowx", max_finger_width=0.06, fps=5, to_opencv=_BRIDGE_TO_OPENCV), + "droid": E(name="droid", robot_name="franka_panda", max_finger_width=0.08, fps=15, to_opencv=_DROID_TO_OPENCV), + "fractal": E( + name="fractal", + robot_name="google_robot", + max_finger_width=0.05, + fps=3, + to_opencv=_GOOGLE_ROBOT_TO_OPENCV, + camera_fov_deg=69.0, + camera_aspect=320 / 256, + ), + "robomind_franka": E( + name="robomind_franka", robot_name="franka_panda", max_finger_width=0.08, fps=10, to_opencv=franka_to_opencv + ), + "robomind_franka_dual": E( + name="robomind_franka_dual", + robot_name="franka_panda", + max_finger_width=0.08, + fps=10, + action_format=ActionFormat.DUAL_ARM_20D, + to_opencv=franka_to_opencv, + dual_base_left=np.array( + [[1, 0, 0, 0.0], [0, 1, 0, 0.3], [0, 0, 1, 0.0], [0, 0, 0, 1.0]], + dtype=np.float32, + ), + dual_base_right=np.array( + [[1, 0, 0, 0.0], [0, 1, 0, -0.3], [0, 0, 1, 0.0], [0, 0, 0, 1.0]], + dtype=np.float32, + ), + ), + "robomind_ur": E(name="robomind_ur", robot_name="ur5e", max_finger_width=0.085, fps=10, to_opencv=eye3), + "umi": E(name="umi", robot_name="", max_finger_width=0.08, fps=10), + "hand_pose": E( + name="hand_pose", robot_name="", max_finger_width=0.0, fps=15, action_format=ActionFormat.UNIFIED_57D + ), + "hwb_egoverse": E( + name="hwb_egoverse", robot_name="", max_finger_width=0.0, fps=15, action_format=ActionFormat.UNIFIED_57D + ), + "av": E(name="av", robot_name="", max_finger_width=0.0, fps=10, action_format=ActionFormat.EGO_9D), + "camera": E( + name="camera", + robot_name="", + max_finger_width=0.0, + fps=10, + action_format=ActionFormat.EGO_9D, + camera_fov_deg=69.0, + camera_aspect=640 / 480, + ), + "embodiment_c_gripper": E( + name="embodiment_c_gripper", + robot_name="embodiment_c", + max_finger_width=0.12, + fps=10, + camera_fov_deg=69.0, + camera_aspect=640 / 480, + robot_embodiment_type="embodiment_c_gripper", + to_opencv=AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST, + ), + "embodiment_c_gripper_ext": E( + name="embodiment_c_gripper_ext", + robot_name="embodiment_c", + max_finger_width=0.12, + fps=10, + camera_fov_deg=69.0, + camera_aspect=640 / 480, + robot_embodiment_type="embodiment_c_gripper_ext", + to_opencv=AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST, + ), + "agibotworld_beta": E( + name="agibotworld_beta", + robot_name="embodiment_c", + max_finger_width=0.12, + fps=10, + camera_fov_deg=69.0, + camera_aspect=640 / 480, + robot_embodiment_type="embodiment_c_gripper", + to_opencv=AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST, + ), + } + + +# ── Color palettes ──────────────────────────────────────────────────────────── + +PALETTE_GT = { + "ego": (39, 174, 96), # green (matches right/left for ego-only datasets) + "ego_top": (39, 174, 96), + "right": (39, 174, 96), # green + "left": (46, 204, 113), # light green +} + +PALETTE_PRED = { + "ego": (231, 76, 60), # red (matches right/left for ego-only datasets) + "ego_top": (192, 57, 43), + "right": (231, 76, 60), # red + "left": (192, 57, 43), # dark red +} + +# ── Safetensors I/O ─────────────────────────────────────────────────────────── + + +def _load_safetensors(path: Path) -> dict[str, np.ndarray]: + """Load all tensors from a safetensors file as float32 numpy arrays.""" + from safetensors.numpy import load_file + + return {k: np.asarray(v, dtype=np.float32) for k, v in load_file(str(path)).items()} + + +def _extract_action(data: dict[str, np.ndarray], target_dim: int | None = None) -> np.ndarray: + """Return (T, D) action array from a safetensors dict. + + If ``target_dim`` is given, truncate the trailing dim to that size — model + predictions are saved at the full untruncated dim while GT is already + truncated to ``raw_action_dim``. + """ + if "action" not in data: + raise KeyError(f"No 'action' key found. Available: {sorted(data)}") + a = data["action"] + if a.ndim == 3 and a.shape[0] == 1: + a = a[0] + a = a.astype(np.float32) + if target_dim is not None and a.shape[-1] > target_dim: + a = a[..., :target_dim] + return a + + +def _extract_pose(data: dict[str, np.ndarray], key: str) -> np.ndarray | None: + """Extract a (4,4) pose matrix from a safetensors dict, or None.""" + v = data.get(key) + if v is None: + return None + v = np.asarray(v, dtype=np.float32) + if v.ndim == 3 and v.shape[0] == 1: + v = v[0] + if v.shape == (4, 4): + return v + return None + + +# ── Action denormalization (inverse of training-time normalization) ────────── + +_NORMALIZER_DIR = Path(__file__).resolve().parents[1] / "normalizers" + +# Dataset name → (normalizer JSON filename, stats key, method). +# Both eval GT and pred are saved AFTER normalization, so the viewer must +# invert it to get raw body-frame deltas back. +_DATASET_NORMALIZER: dict[str, tuple[str, str, str]] = { + "bridge": ("bridge_orig_lerobot_backward_framewise_rot6d.json", "global", "quantile"), + "droid": ("droid_lerobot_backward_framewise_rot6d.json", "global", "quantile"), + "fractal": ("fractal_backward_framewise_rot6d.json", "global", "quantile"), + "robomind_franka": ("robomind-franka_backward_framewise_rot6d.json", "global", "quantile"), + "robomind_franka_dual": ("robomind-franka-dual_backward_framewise_rot6d.json", "global", "quantile"), + "robomind_ur": ("robomind-ur_backward_framewise_rot6d.json", "global", "quantile"), + "embodiment_c_gripper": ("embodiment_c_gripper_backward_framewise_rot6d.json", "global", "quantile"), + "embodiment_c_gripper_ext": ("embodiment_c_gripper_backward_framewise_rot6d.json", "global", "quantile"), + "hand_pose": ("hand_pose_backward_framewise_rot6d.json", "global", "quantile"), + "hwb_egoverse": ("hand_pose_backward_framewise_rot6d.json", "global", "quantile"), +} + + +def _load_norm_stats(dataset_name: str | None) -> tuple[dict[str, np.ndarray], str] | None: + """Return (stats, method) for ``dataset_name`` or None if unknown.""" + if dataset_name is None: + return None + cfg = _DATASET_NORMALIZER.get(dataset_name) + if cfg is None: + return None + fname, key, method = cfg + path = _NORMALIZER_DIR / fname + if not path.exists(): + log.warning(f"Normalizer JSON not found: {path}") + return None + with path.open() as f: + raw = json.load(f) + block = raw.get(key, raw) + stat_keys = {"mean", "std", "min", "max", "q01", "q99"} + stats = {k: np.asarray(v, dtype=np.float32) for k, v in block.items() if k in stat_keys} + return stats, method + + +def _denormalize_action(action: np.ndarray, stats: dict[str, np.ndarray], method: str) -> np.ndarray: + """Inverse of training normalization. Returns raw body-frame action.""" + D = action.shape[-1] + if method == "quantile": + q01, q99 = stats["q01"][:D], stats["q99"][:D] + denom = np.maximum(q99 - q01, 1e-8) + return (action + 1.0) / 2.0 * denom + q01 + if method == "minmax": + lo, hi = stats["min"][:D], stats["max"][:D] + denom = np.maximum(hi - lo, 1e-8) + return (action + 1.0) / 2.0 * denom + lo + if method == "meanstd": + mean, std = stats["mean"][:D], np.maximum(stats["std"][:D], 1e-8) + return action * std + mean + raise ValueError(f"Unknown normalization method: {method!r}") + + +# ── Auto action-format inference ────────────────────────────────────────────── + +_DIM_TO_FORMAT: dict[int, ActionFormat] = { + 9: ActionFormat.EGO_9D, + 10: ActionFormat.SINGLE_ARM_10D, + 20: ActionFormat.DUAL_ARM_20D, + 57: ActionFormat.UNIFIED_57D, +} + + +def _infer_action_format(action: np.ndarray) -> ActionFormat: + dim = int(action.shape[-1]) + fmt = _DIM_TO_FORMAT.get(dim) + if fmt is None: + raise ValueError( + f"Cannot infer action format from trailing dim {dim}. " + f"Known: {sorted(_DIM_TO_FORMAT)}. " + "Pass --action-format explicitly." + ) + return fmt + + +# ── State building ──────────────────────────────────────────────────────────── + + +def _build_state_from_action( + action: np.ndarray, + gt_data: dict[str, np.ndarray], + entry: DatasetEntry | None, + action_format: ActionFormat, +) -> SceneState: + """Convert a raw action array to a render-ready SceneState. + + Uses initial pose and pose convention from the dataset entry (or GT data). + The same initial pose is applied to both GT and pred states so trajectories + start at the same anchor point. + """ + initial_pose = _extract_pose(gt_data, "initial_pose") + if initial_pose is None: + initial_pose = np.eye(4, dtype=np.float32) + initial_pose_right = _extract_pose(gt_data, "initial_pose_right") + initial_pose_left = _extract_pose(gt_data, "initial_pose_left") + + pose_convention = entry.pose_convention if entry else "backward_framewise" + right_base_pose = entry.dual_base_right if entry else None + left_base_pose = entry.dual_base_left if entry else None + + if entry and entry.to_unified_fn: + import inspect as _inspect + + converter = _load_symbol(entry.to_unified_fn) + params = _inspect.signature(converter).parameters + embodiment_type = entry.robot_embodiment_type or str(entry.dataset_kwargs.get("embodiment_type", "")) + if "embodiment_type" in params: + unified = converter({"action": action}, embodiment_type=embodiment_type) + elif "kind" in params: + unified = converter(action, kind="gripper") + else: + unified = converter(action) + else: + unified = to_unified(action, action_format=action_format) + + uses_dual = action_format is ActionFormat.DUAL_ARM_20D + if uses_dual and initial_pose_left is None: + initial_pose_left = initial_pose + + return build_scene_state( + unified, + initial_pose=initial_pose, + initial_pose_right=initial_pose_right, + initial_pose_left=initial_pose_left, + right_base_pose=right_base_pose, + left_base_pose=left_base_pose, + pose_convention=pose_convention, + ) + + +# ── Sample discovery ────────────────────────────────────────────────────────── + + +@dataclass +class SamplePair: + gt_path: Path + pred_path: Path + label: str + dataset_key: str | None = None # canonical key ("bridge" from "bridge_20260416") + gt_video_path: Path | None = None + pred_video_path: Path | None = None + + +def _dataset_key_from_versioned(versioned_name: str) -> str: + """Strip the ``_YYYYMMDD`` suffix from a versioned dataset name. + + e.g. ``"bridge_20260416"`` → ``"bridge"``, + ``"robomind_franka_dual_20260414"`` → ``"robomind_franka_dual"``. + Falls back to the input if no version suffix is found. + """ + return re.sub(r"_\d{8}$", "", versioned_name) + + +def _load_mp4_frames(path: Path) -> np.ndarray | None: + """Decode an mp4 to ``(T, H, W, 3)`` uint8 RGB. Returns None on failure.""" + if path is None or not path.exists(): + return None + try: + import cv2 # type: ignore[import-not-found] + except ImportError: + log.warning("cv2 not available — video panels disabled") + return None + cap = cv2.VideoCapture(str(path)) + frames: list[np.ndarray] = [] + while True: + ok, frame_bgr = cap.read() + if not ok: + break + frames.append(cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)) + cap.release() + if not frames: + log.warning(f"No frames decoded from {path}") + return None + return np.stack(frames, axis=0).astype(np.uint8) + + +_GT_FILENAMES = ("batch_data.safetensors", "output.safetensors") + + +def _find_gt_file(gt_batch_dir: Path) -> Path | None: + """Return the first GT safetensors file present in ``gt_batch_dir``.""" + for fname in _GT_FILENAMES: + candidate = gt_batch_dir / fname + if candidate.exists(): + return candidate + return None + + +def _discover_pairs(eval_dir: Path) -> list[SamplePair]: + """Walk an eval job directory and collect (gt, pred) safetensors pairs. + + Expected layout (Lepton/orchestrated eval output): + eval_dir/ + ├── outputs/{ds}/{mode}/{batch_id}/[{sample?}/]output.safetensors + └── ground_truth/{ds}/{mode}/{batch_id}/{batch_data,output}.safetensors + """ + outputs_root = eval_dir / "outputs" + gt_root = eval_dir / "ground_truth" + + if not outputs_root.exists(): + raise FileNotFoundError(f"outputs/ dir not found under {eval_dir}") + if not gt_root.exists(): + raise FileNotFoundError(f"ground_truth/ dir not found under {eval_dir}") + + pairs: list[SamplePair] = [] + for pred_file in sorted(outputs_root.rglob("output.safetensors")): + rel = pred_file.relative_to(outputs_root) + parts = rel.parts # (ds, mode, batch_id, [seed], output.safetensors) + if len(parts) < 4: + continue + ds, mode, batch_id = parts[0], parts[1], parts[2] + seed = parts[3] if len(parts) == 5 else None + gt_file = _find_gt_file(gt_root / ds / mode / batch_id) + if gt_file is None: + log.warning(f"No GT found for {pred_file.relative_to(eval_dir)}") + continue + label = f"{ds}/{mode}/{batch_id}" + (f"/{seed}" if seed else "") + + # Optional video files: pred = vision.mp4 next to the pred safetensors; + # GT = first *.mp4 in the GT batch dir (typically vision_gt.mp4). + pred_video = pred_file.parent / "vision.mp4" + gt_videos = sorted((gt_root / ds / mode / batch_id).glob("*.mp4")) + pairs.append( + SamplePair( + gt_path=gt_file, + pred_path=pred_file, + label=label, + dataset_key=_dataset_key_from_versioned(ds), + gt_video_path=gt_videos[0] if gt_videos else None, + pred_video_path=pred_video if pred_video.exists() else None, + ) + ) + + return pairs + + +# ── Viewer session ──────────────────────────────────────────────────────────── + + +def _collect_scene_points_both(gt_state: SceneState, pred_state: SceneState) -> np.ndarray: + """Collect all trajectory points from both states for camera framing.""" + points: list[np.ndarray] = [] + for state in (gt_state, pred_state): + for poses in (state.ego_poses, state.right_poses, state.left_poses): + if poses is not None and len(poses) > 0: + points.append(poses[:, :3, 3].astype(np.float32)) + return np.concatenate(points, axis=0) if points else np.zeros((1, 3), dtype=np.float32) + + +def _format_compare_text(gt_state: SceneState, pred_state: SceneState, t: int) -> str: + """Show per-step GT vs pred action values and per-component L2 error.""" + if t == 0: + return "*t=0: anchor pose (identity)*" + + gt_raw = gt_state.action_raw + pred_raw = pred_state.action_raw + if gt_raw is None or pred_raw is None: + return "" + idx = t - 1 + if idx >= len(gt_raw) or idx >= len(pred_raw): + return "" + + g = gt_raw[idx] + p = pred_raw[idx] + mask = gt_state.mask + min_d = min(len(g), len(p)) + diff = p[:min_d] - g[:min_d] + mse_total = float(np.mean(diff**2)) + + lines = [ + f"step {idx} → {t}", + f"MSE (all dims): {mse_total:.6f}", + "═" * 32, + ] + + def _fmt3(v: np.ndarray) -> str: + return " ".join(f"{x:+.4f}" for x in v[:3]) + + def _mse3(a: np.ndarray, b: np.ndarray) -> str: + return f"err={np.mean((b - a) ** 2):.5f}" + + if mask.ego and min_d >= 9: + lines += [ + f"Ego pos GT {_fmt3(g[0:3])}", + f" Pr {_fmt3(p[0:3])} {_mse3(g[0:3], p[0:3])}", + ] + if mask.right_wrist and min_d >= 18: + lines += [ + "", + f"R wrist GT {_fmt3(g[9:12])}", + f" Pr {_fmt3(p[9:12])} {_mse3(g[9:12], p[9:12])}", + ] + if gt_state.gripper_right is not None and t < len(gt_state.gripper_right): + gr_gt = float(gt_state.gripper_right[t]) + gr_pr = float(pred_state.gripper_right[t]) if pred_state.gripper_right is not None else float("nan") + lines.append(f" grip GT {gr_gt:+.4f} Pr {gr_pr:+.4f}") + if mask.left_wrist and min_d >= 42: + lines += [ + "", + f"L wrist GT {_fmt3(g[33:36])}", + f" Pr {_fmt3(p[33:36])} {_mse3(g[33:36], p[33:36])}", + ] + if gt_state.gripper_left is not None and t < len(gt_state.gripper_left): + gl_gt = float(gt_state.gripper_left[t]) + gl_pr = float(pred_state.gripper_left[t]) if pred_state.gripper_left is not None else float("nan") + lines.append(f" grip GT {gl_gt:+.4f} Pr {gl_pr:+.4f}") + + return "```\n" + "\n".join(lines) + "\n```" + + +# ── Main viewer ─────────────────────────────────────────────────────────────── + + +def _dummy_entry(action_format: ActionFormat) -> DatasetEntry: + """Stub entry for samples whose dataset isn't in the registry. + + With ``robot_name=""`` the renderer skips the URDF mesh + IK; trajectories + and EE frames still render correctly. + """ + return DatasetEntry( + name="unknown", + robot_name="", + max_finger_width=0.05, + fps=10, + action_format=action_format, + ) + + +def launch_compare_viewer( + pairs: list[SamplePair], + entry: DatasetEntry | None, + action_format_override: ActionFormat | None, + port: int = 8014, + share: bool = False, + denorm_gt: bool = False, + denorm_pred: bool = False, + datasets: dict[str, DatasetEntry] | None = None, + on_share_url: Callable[[str], None] | None = None, + idle_timeout_s: int | None = None, +) -> None: + """Launch the interactive compare viewer. + + Args: + on_share_url: Called once with the public share URL when share=True. + idle_timeout_s: Exit after this many seconds with no connected clients + (only starts counting after the first client has connected and left). + """ + import threading as _threading + + import viser + + server = viser.ViserServer(host="0.0.0.0", port=port) + + @dataclass + class ViewerSession: + time_slider: Any + speed_slider: Any + load_lock: Any = field(default_factory=_threading.Lock) + gt_state: SceneState | None = None + pred_state: SceneState | None = None + playing: bool = False + last_frame_time: float = 0.0 + + sessions: dict[int, ViewerSession] = {} + sessions_lock = _threading.Lock() + idle_state = {"had_connection": False, "last_activity": _time.time()} + + @server.on_client_connect + def _(client) -> None: + client.scene.reset() + client.scene.set_up_direction("+z") + client.gui.reset() + + gt_renderer = UnifiedRenderer(client, name_prefix="/gt", palette=PALETTE_GT) + pred_renderer = UnifiedRenderer(client, name_prefix="/pred", palette=PALETTE_PRED) + + with client.gui.add_folder("Sample"): + sample_labels = [p.label for p in pairs] + sample_dropdown = client.gui.add_dropdown("Sample", options=sample_labels, initial_value=sample_labels[0]) + status_text = client.gui.add_markdown("*Ready*") + + with client.gui.add_folder("Display", expand_by_default=False): + show_robot = client.gui.add_checkbox("Show robot mesh", initial_value=True) + show_frames = client.gui.add_checkbox("Show wrist frames", initial_value=True) + show_traj = client.gui.add_checkbox("Show trajectory", initial_value=True) + show_fingertips = client.gui.add_checkbox("Show fingertips", initial_value=True) + show_gt = client.gui.add_checkbox("Show GT", initial_value=True) + show_pred = client.gui.add_checkbox("Show Pred", initial_value=True) + axis_scale = client.gui.add_slider("Axis scale", min=0.1, max=20.0, step=0.1, initial_value=1.0) + + with client.gui.add_folder("Playback"): + time_slider = client.gui.add_slider("Time", min=0, max=1, step=1, initial_value=0) + play_button = client.gui.add_button("▶ Play") + speed_slider = client.gui.add_slider("Speed (fps)", min=1, max=30, step=1, initial_value=3) + + with client.gui.add_folder("Camera", expand_by_default=True): + image_panel = client.gui.add_image(np.zeros((64, 64, 3), dtype=np.uint8)) + gt_renderer.set_video_panel(image_panel) + + with client.gui.add_folder("GT vs Pred", expand_by_default=True): + compare_text = client.gui.add_markdown("*No sample loaded*") + + with client.gui.add_folder("Legend", expand_by_default=False): + client.gui.add_markdown("**GT** trajectories 🟢 green | **Pred** trajectories 🔴 red") + + show = { + "frames": True, + "traj": True, + "fingertips": True, + "ego": True, + "robot": True, + "robot_frame_filters": {}, + "gt": True, + "pred": True, + } + session = ViewerSession(time_slider=time_slider, speed_slider=speed_slider) + + def _effective_show(renderer_key: str) -> dict: + """Build show dict with ego/frames/traj gated by per-renderer toggle. + + Only the pred renderer draws the robot mesh — GT contributes only + its trajectory + frames so we have one mesh that follows pred. + """ + s = dict(show) + if not show.get(renderer_key, True): + s["frames"] = False + s["traj"] = False + s["fingertips"] = False + s["ego"] = False + s["robot"] = False + if renderer_key == "gt": + s["robot"] = False # always hide GT mesh; pred mesh is the only one + return s + + def _update_both(t: int) -> None: + if session.gt_state is not None: + gt_renderer.update(t, _effective_show("gt")) + if session.pred_state is not None: + pred_renderer.update(t, _effective_show("pred")) + if session.gt_state is not None and session.pred_state is not None: + compare_text.content = _format_compare_text(session.gt_state, session.pred_state, t) + + def do_load() -> None: + label = sample_dropdown.value + idx = sample_labels.index(label) + pair = pairs[idx] + status_text.content = f"⏳ Loading {label}..." + try: + gt_data = _load_safetensors(pair.gt_path) + pred_data = _load_safetensors(pair.pred_path) + + gt_action = _extract_action(gt_data) + # Pred is saved at the full model action dim; truncate to GT's dim. + pred_action = _extract_action(pred_data, target_dim=int(gt_action.shape[-1])) + + # Resolve the dataset entry: explicit --dataset override beats + # the per-pair auto-detection from the eval directory layout. + # Unknown datasets fall back to a dummy entry that renders + # trajectories + frames without a robot mesh. + pair_entry = entry + if pair_entry is None and datasets is not None and pair.dataset_key is not None: + pair_entry = datasets.get(pair.dataset_key) + + # Denormalization is per-side: depends on the eval's training + # config (action_normalization). Toggle via --denorm-gt / + # --denorm-pred CLI flags. + ds_key = pair_entry.name if pair_entry else pair.dataset_key + norm = _load_norm_stats(ds_key) if ds_key else None + if norm is not None and (denorm_gt or denorm_pred): + stats, method = norm + if denorm_gt: + gt_action = _denormalize_action(gt_action, stats, method) + if denorm_pred: + pred_action = _denormalize_action(pred_action, stats, method) + + has_custom_converter = pair_entry is not None and pair_entry.to_unified_fn is not None + if action_format_override: + fmt = action_format_override + elif has_custom_converter: + fmt = ActionFormat.SINGLE_ARM_10D # dummy — custom converter owns raw-format handling + else: + fmt = _infer_action_format(gt_action) + + gt_state = _build_state_from_action(gt_action, gt_data, pair_entry, fmt) + pred_state = _build_state_from_action(pred_action, gt_data, pair_entry, fmt) + + # Load GT camera video + gt_state.video = _load_mp4_frames(pair.gt_video_path) + + session.gt_state = gt_state + session.pred_state = pred_state + + _entry = pair_entry or _dummy_entry(fmt) + fov_deg = _entry.camera_fov_deg + gt_renderer.load(gt_state, _entry, to_opencv=_entry.to_opencv) + pred_renderer.load(pred_state, _entry, to_opencv=_entry.to_opencv) + + T = max(gt_state.T, pred_state.T) + time_slider.max = max(T, 1) + time_slider.value = 0 + + all_pts = _collect_scene_points_both(gt_state, pred_state) + _reset_camera_to_trajectory_both(client, all_pts, gt_state, fov_deg) + + gt_renderer.update_axis_scale(axis_scale.value) + pred_renderer.update_axis_scale(axis_scale.value) + _update_both(0) + + status_text.content = f"✅ {label} | GT T={gt_state.T} | Pred T={pred_state.T} | format={fmt.value}" + log.info(f"Loaded pair: {label}, format={fmt.value}") + + except Exception as e: + status_text.content = f"❌ {e}" + log.error(f"Load failed: {e}") + import traceback + + traceback.print_exc() + + def _load_threaded() -> None: + if not session.load_lock.acquire(blocking=False): + return + + def _run() -> None: + try: + do_load() + finally: + session.load_lock.release() + + _threading.Thread(target=_run, daemon=True).start() + + @sample_dropdown.on_update + def _(_) -> None: + _load_threaded() + + @time_slider.on_update + def _(_) -> None: + _update_both(time_slider.value) + + @play_button.on_click + def _(_) -> None: + session.playing = not session.playing + session.last_frame_time = _time.time() + play_button.label = "⏸ Pause" if session.playing else "▶ Play" + + @show_robot.on_update + def _(_) -> None: + show["robot"] = show_robot.value + _update_both(time_slider.value) + + @show_frames.on_update + def _(_) -> None: + show["frames"] = show_frames.value + _update_both(time_slider.value) + + @show_traj.on_update + def _(_) -> None: + show["traj"] = show_traj.value + _update_both(time_slider.value) + + @show_fingertips.on_update + def _(_) -> None: + show["fingertips"] = show_fingertips.value + _update_both(time_slider.value) + + @show_gt.on_update + def _(_) -> None: + show["gt"] = show_gt.value + _update_both(time_slider.value) + + @show_pred.on_update + def _(_) -> None: + show["pred"] = show_pred.value + _update_both(time_slider.value) + + @axis_scale.on_update + def _(_) -> None: + gt_renderer.update_axis_scale(axis_scale.value) + pred_renderer.update_axis_scale(axis_scale.value) + + with sessions_lock: + sessions[client.client_id] = session + idle_state["had_connection"] = True + idle_state["last_activity"] = _time.time() + _load_threaded() + + @server.on_client_disconnect + def _(client) -> None: + with sessions_lock: + sessions.pop(client.client_id, None) + idle_state["last_activity"] = _time.time() + + log.info(f"✅ Compare viewer ready at http://0.0.0.0:{port}") + if share: + share_url = server.request_share_url() + if share_url: + log.info(f"🌐 Share URL: {share_url}") + if on_share_url is not None: + on_share_url(share_url) + + try: + while True: + now = _time.time() + with sessions_lock: + active = list(sessions.values()) + if idle_timeout_s is not None: + _had = idle_state["had_connection"] + _idle = now - idle_state["last_activity"] + _empty = len(sessions) == 0 + if idle_timeout_s is not None and _had and _empty and _idle > idle_timeout_s: + log.info(f"No active connections for {idle_timeout_s}s — shutting down.") + break + for session in active: + if not session.playing: + continue + if session.gt_state is None and session.pred_state is None: + continue + frame_period = 1.0 / max(float(session.speed_slider.value), 1.0) + if now - session.last_frame_time < frame_period: + continue + T = max( + session.gt_state.T if session.gt_state else 0, + session.pred_state.T if session.pred_state else 0, + ) + t = (session.time_slider.value + 1) % max(T, 1) + session.time_slider.value = t + session.last_frame_time = now + _time.sleep(0.02) + except KeyboardInterrupt: + log.info("Shutting down.") + + +# ── Helpers ─────────────────────────────────────────────────────────────────── + + +def _reset_camera_to_trajectory_both( + client: Any, + all_points: np.ndarray, + gt_state: SceneState, + camera_fov_deg: float, +) -> None: + """Frame the viewport around all trajectory points from both states.""" + from cosmos_framework.data.vfm.action.urdf_visualizer.viewer import ( + _get_observation_forward_direction, + _get_observation_up_direction, + ) + + center = all_points.mean(axis=0) + extent = all_points - center[None, :] + radius = float(np.linalg.norm(extent, axis=1).max()) if len(all_points) > 0 else 0.0 + radius = max(radius, 0.15) + + fov_rad = float(np.deg2rad(camera_fov_deg)) + fit_distance = radius / max(np.tan(fov_rad / 2.0), 0.35) + + view_forward = _get_observation_forward_direction(gt_state) + if view_forward is None: + view_dir = np.array([1.0, -1.0, 0.7], dtype=np.float32) + view_dir /= np.linalg.norm(view_dir) + camera_position = center + view_dir * max(fit_distance * 1.35, 0.5) + view_forward = center - camera_position + view_forward /= np.linalg.norm(view_forward) + else: + camera_position = center - view_forward * max(fit_distance * 1.35, 0.5) + + view_forward = center - camera_position + view_forward /= np.linalg.norm(view_forward) + up_direction = _get_observation_up_direction(gt_state, view_forward) + + camera = client.camera + deadline = _time.time() + 1.0 + while getattr(camera._state, "update_timestamp", 0.0) == 0.0 and _time.time() < deadline: + _time.sleep(0.01) + if getattr(camera._state, "update_timestamp", 0.0) == 0.0: + return + + camera.fov = fov_rad + camera.up_direction = tuple(up_direction.tolist()) + camera.position = tuple(camera_position.tolist()) + camera.look_at = tuple(center.tolist()) + client.flush() + + +# ── CLI ─────────────────────────────────────────────────────────────────────── + + +def main() -> None: + parser = argparse.ArgumentParser(description="Compare GT vs predicted action trajectories") + group = parser.add_mutually_exclusive_group(required=True) + group.add_argument( + "--eval-dir", + type=Path, + help="Eval job directory containing outputs/ and ground_truth/ subdirs", + ) + group.add_argument( + "--gt", + type=Path, + help="Path to ground-truth batch_data.safetensors", + ) + parser.add_argument( + "--pred", + type=Path, + help="Path to predicted output.safetensors (required with --gt)", + ) + parser.add_argument( + "--dataset", + type=str, + default=None, + help=( + "Optional dataset key (e.g. bridge, droid). When omitted, auto-detected" + " per sample from the eval-dir layout (ground_truth//...)." + " Pass to pin a single registry entry for all samples." + ), + ) + parser.add_argument( + "--action-format", + choices=[fmt.value for fmt in ActionFormat], + default=None, + help="Override the raw action format (default: inferred from tensor shape)", + ) + parser.add_argument("--port", type=int, default=8014) + parser.add_argument("--share", action="store_true") + parser.add_argument( + "--denorm-gt", + action="store_true", + help="Denormalize GT action (use when training experiment had action_normalization!=None).", + ) + parser.add_argument( + "--denorm-pred", + action="store_true", + help="Denormalize pred action (use when training experiment had action_normalization!=None).", + ) + args = parser.parse_args() + + if args.gt is not None and args.pred is None: + parser.error("--pred is required when --gt is specified") + + # ── Build pairs list ── + if args.eval_dir is not None: + pairs = _discover_pairs(args.eval_dir) + if not pairs: + log.error(f"No (GT, pred) pairs found under {args.eval_dir}") + raise SystemExit(1) + log.info(f"Found {len(pairs)} sample pairs in {args.eval_dir}") + else: + gt_path: Path = args.gt + pred_path: Path = args.pred + if not gt_path.exists(): + parser.error(f"GT file not found: {gt_path}") + if not pred_path.exists(): + parser.error(f"Pred file not found: {pred_path}") + label = gt_path.parent.name + pairs = [SamplePair(gt_path=gt_path, pred_path=pred_path, label=label)] + + # ── Build dataset registry (used for per-pair auto-detection from eval-dir + # layout) and optionally pin to a single entry via --dataset. Fall back to + # the hardcoded minimal entries when the full registry can't be built + try: + datasets = _build_datasets() + except Exception as e: + log.warning(f"Could not build full dataset registry ({e}); using hardcoded minimal entries.") + datasets = _build_minimal_entries() + + entry: DatasetEntry | None = None + if args.dataset: + entry = datasets.get(args.dataset) + if entry is None: + log.warning( + f"Dataset '{args.dataset}' not found. Known: {sorted(datasets)}. Running without robot metadata." + ) + + action_format_override = ActionFormat(args.action_format) if args.action_format else None + + launch_compare_viewer( + pairs=pairs, + entry=entry, + action_format_override=action_format_override, + port=args.port, + share=args.share, + denorm_gt=args.denorm_gt, + denorm_pred=args.denorm_pred, + datasets=datasets, + ) + + +if __name__ == "__main__": + main() diff --git a/cosmos_framework/data/vfm/action/urdf_visualizer/contracts_test.py b/cosmos_framework/data/vfm/action/urdf_visualizer/contracts_test.py new file mode 100644 index 0000000..e544e47 --- /dev/null +++ b/cosmos_framework/data/vfm/action/urdf_visualizer/contracts_test.py @@ -0,0 +1,306 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import numpy as np +import pytest + +from cosmos_framework.data.vfm.action.urdf_visualizer import ik_solver, robot_scene_model, unified_renderer +from cosmos_framework.data.vfm.action.urdf_visualizer.robot_scene_model import RobotSceneModel +from cosmos_framework.data.vfm.action.urdf_visualizer.unified_action import ( + ALL_FINGERS, + Action57DMask, + ActionFormat, + UnifiedAction, + build_scene_state, + to_unified, +) +from cosmos_framework.data.vfm.action.urdf_visualizer.unified_renderer import _agibot_gripper_tip_positions + + +@pytest.mark.L0 +@pytest.mark.parametrize( + ("action_format", "action", "expected_mask"), + [ + ( + ActionFormat.EGO_9D, + np.zeros((2, 9), dtype=np.float32), + Action57DMask(ego=True), + ), + ( + ActionFormat.SINGLE_ARM_10D, + np.zeros((2, 10), dtype=np.float32), + Action57DMask(right_wrist=True), + ), + ( + ActionFormat.DUAL_ARM_20D, + np.zeros((2, 20), dtype=np.float32), + Action57DMask(right_wrist=True, left_wrist=True), + ), + ( + ActionFormat.UNIFIED_57D, + np.zeros((2, 57), dtype=np.float32), + Action57DMask( + ego=True, + right_wrist=True, + right_fingers=ALL_FINGERS, + left_wrist=True, + left_fingers=ALL_FINGERS, + ), + ), + ], +) +def test_to_unified_uses_explicit_action_format( + action_format: ActionFormat, + action: np.ndarray, + expected_mask: Action57DMask, +) -> None: + """Explicit action formats should decode to one canonical 57D layout.""" + unified = to_unified(action, action_format=action_format) + + assert unified.action.shape == (2, 57) + assert unified.mask == expected_mask + + +@pytest.mark.L0 +def test_to_unified_rejects_shape_mismatches() -> None: + """Declared raw action formats should fail fast on incompatible tensors.""" + with pytest.raises(ValueError, match="expects trailing dim 10"): + to_unified(np.zeros((2, 9), dtype=np.float32), action_format=ActionFormat.SINGLE_ARM_10D) + + +@pytest.mark.L0 +def test_build_scene_state_outputs_world_space_contract() -> None: + """SceneState should be fully canonicalized into world-space during construction.""" + action = np.zeros((1, 57), dtype=np.float32) + action[0, 9:18] = np.array([0.1, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0], dtype=np.float32) + action[0, 18:21] = np.array([0.0, 0.0, 0.05], dtype=np.float32) + action[0, 33:42] = np.array([0.2, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0], dtype=np.float32) + unified = UnifiedAction( + action=action, + mask=Action57DMask( + right_wrist=True, + right_fingers=ALL_FINGERS, + left_wrist=True, + ), + ) + + right_base = np.eye(4, dtype=np.float32) + right_base[1, 3] = -0.3 + left_base = np.eye(4, dtype=np.float32) + left_base[1, 3] = 0.3 + + state = build_scene_state( + unified, + right_base_pose=right_base, + left_base_pose=left_base, + ) + + np.testing.assert_allclose(state.right_poses[1, :3, 3], np.array([0.1, -0.3, 0.0], dtype=np.float32)) + np.testing.assert_allclose(state.left_poses[1, :3, 3], np.array([0.2, 0.3, 0.0], dtype=np.float32)) + np.testing.assert_allclose(state.right_fingers[0, 0], np.array([0.0, -0.3, 0.05], dtype=np.float32)) + np.testing.assert_allclose(state.right_fingers[1, 0], np.array([0.1, -0.3, 0.05], dtype=np.float32)) + + +@pytest.mark.L0 +def test_robot_scene_model_caches_home_meshes_and_applies_base_pose(monkeypatch: pytest.MonkeyPatch) -> None: + """RobotSceneModel should own home mesh loading and world-frame placement.""" + calls = {"count": 0} + mesh_transform = np.eye(4, dtype=np.float32) + mesh_transform[2, 3] = 0.2 + + monkeypatch.setattr(robot_scene_model, "get_robot_config", lambda _: {"ee_frame": "tool"}) + monkeypatch.setattr(robot_scene_model, "get_mjcf_path", lambda _: "/tmp/fake.xml") + + def _loader() -> tuple[list[tuple[str, object, np.ndarray]], np.ndarray]: + calls["count"] += 1 + return [("geom", object(), mesh_transform)], np.eye(4, dtype=np.float32) + + monkeypatch.setattr(robot_scene_model, "get_robot_loaders", lambda: {"fake_robot": _loader}) + + model = RobotSceneModel("fake_robot") + meshes_local = model.get_home_meshes() + base_pose = np.eye(4, dtype=np.float32) + base_pose[0, 3] = 1.5 + meshes_world = model.get_home_meshes(base_pose=base_pose) + + assert calls["count"] == 1 + np.testing.assert_allclose(meshes_local[0][2][:3, 3], np.array([0.0, 0.0, 0.2], dtype=np.float32)) + np.testing.assert_allclose(meshes_world[0][2][:3, 3], np.array([1.5, 0.0, 0.2], dtype=np.float32)) + + +@pytest.mark.L0 +def test_robot_scene_model_reapplies_base_pose_after_solving(monkeypatch: pytest.MonkeyPatch) -> None: + """RobotSceneModel should solve in arm-local space but return world-space geometry.""" + captured: dict[str, np.ndarray] = {} + + monkeypatch.setattr(robot_scene_model, "get_robot_config", lambda _: {"ee_frame": "tool"}) + monkeypatch.setattr(robot_scene_model, "get_mjcf_path", lambda _: "/tmp/fake.xml") + + def _fake_solve( + _mjcf_path: str, + world_ee_positions: np.ndarray, + gripper_openings: np.ndarray | None = None, + world_ee_orientations: np.ndarray | None = None, + robot_name: str | None = None, + ) -> np.ndarray: + captured["positions"] = world_ee_positions.copy() + captured["orientations"] = world_ee_orientations.copy() + captured["grippers"] = ( + gripper_openings.copy() if gripper_openings is not None else np.array([], dtype=np.float32) + ) + assert robot_name == "fake_robot" + return np.zeros((2, 1), dtype=np.float32) + + def _fake_fk( + _mjcf_path: str, + _joint_configs: np.ndarray, + robot_name: str | None = None, + ) -> tuple[np.ndarray, np.ndarray]: + assert robot_name == "fake_robot" + positions = np.array([[0.0, 0.0, 0.0], [0.2, 0.0, 0.0]], dtype=np.float32) + rotations = np.tile(np.eye(3, dtype=np.float32), (2, 1, 1)) + return positions, rotations + + def _fake_geom( + _mjcf_path: str, + _joint_configs: np.ndarray, + ) -> tuple[ + list[list[tuple[np.ndarray, np.ndarray]]], + None, + None, + dict[str, list[tuple[np.ndarray, np.ndarray]]], + ]: + transforms = [ + [(np.array([0.0, 0.0, 0.0], dtype=np.float32), np.eye(3, dtype=np.float32))], + [(np.array([0.2, 0.0, 0.0], dtype=np.float32), np.eye(3, dtype=np.float32))], + ] + frames = { + "body:tool": [ + (np.array([0.0, 0.0, 0.0], dtype=np.float32), np.eye(3, dtype=np.float32)), + (np.array([0.2, 0.0, 0.0], dtype=np.float32), np.eye(3, dtype=np.float32)), + ] + } + return transforms, None, None, frames + + monkeypatch.setattr(ik_solver, "solve_trajectory_ik", _fake_solve) + monkeypatch.setattr(ik_solver, "compute_fk_ee_poses", _fake_fk) + monkeypatch.setattr(ik_solver, "compute_mujoco_geom_transforms", _fake_geom) + + model = RobotSceneModel("fake_robot") + wrist_poses_world = np.tile(np.eye(4, dtype=np.float32), (2, 1, 1)) + wrist_poses_world[:, 0, 3] = np.array([1.0, 1.2], dtype=np.float32) + base_pose = np.eye(4, dtype=np.float32) + base_pose[0, 3] = 1.0 + grippers = np.array([0.0, 0.25], dtype=np.float32) + + result = model.solve_visual_trajectory( + wrist_poses_world, + gripper_openings=grippers, + base_pose=base_pose, + ) + + assert result is not None + np.testing.assert_allclose( + captured["positions"][:, 0], + np.array([0.0, 0.2], dtype=np.float32), + atol=1e-6, + ) + np.testing.assert_allclose( + result.mesh_transforms[1][0][0], + np.array([1.2, 0.0, 0.0], dtype=np.float32), + atol=1e-6, + ) + np.testing.assert_allclose( + result.named_frames["ik:tool"][1][0], + np.array([1.2, 0.0, 0.0], dtype=np.float32), + atol=1e-6, + ) + + +@pytest.mark.L0 +def test_agibot_gripper_tip_spheres_open_along_opencv_x_axis() -> None: + """AgiBot synthetic tip spheres should open horizontally in the corrected wrist frame.""" + + pos = np.array([0.1, 0.2, 0.3], dtype=np.float32) # [3] + rot = np.eye(3, dtype=np.float32) # [3,3] + + tip_l, tip_r = _agibot_gripper_tip_positions( + pos=pos, + rot=rot, + opening=1.0, + max_finger_width=0.12, + ) + + expected_center = pos + np.array([0.0, 0.0, 0.14308], dtype=np.float32) # [3] + np.testing.assert_allclose(tip_l, expected_center + np.array([0.06, 0.0, 0.0], dtype=np.float32), atol=1e-6) + np.testing.assert_allclose(tip_r, expected_center + np.array([-0.06, 0.0, 0.0], dtype=np.float32), atol=1e-6) + + +@pytest.mark.L0 +def test_agibot_renderer_uses_direct_link_poses_before_ik(monkeypatch: pytest.MonkeyPatch) -> None: + """Dataset FK link poses should drive AgiBot meshes directly when present.""" + + class _Scene: + def __init__(self) -> None: + self.handles: list[object] = [] + + def add_mesh_trimesh(self, *_args, **_kwargs): + class _Handle: + visible = True + + def remove(self) -> None: + self.visible = False + + handle = _Handle() + self.handles.append(handle) + return handle + + class _Server: + def __init__(self) -> None: + self.scene = _Scene() + + class _Entry: + robot_name = "embodiment_c" + robot_embodiment_type = "embodiment_c_gripper" + dataset_kwargs: dict[str, str] = {} + + geometry = type( + "Geometry", + (), + { + "name": "moving_link_geometry_0", + "link_name": "moving_link", + "mesh": object(), + "local_transform": np.eye(4, dtype=np.float32), + }, + )() + monkeypatch.setattr(unified_renderer, "get_embodiment_c_collision_geometries", lambda: [geometry]) + + def _fail_ik(*_args, **_kwargs): + raise AssertionError("IK should not run when direct AgiBot FK link poses are present") + + monkeypatch.setattr(unified_renderer, "solve_agibot_trajectory_ik", _fail_ik) + + poses = np.tile(np.eye(4, dtype=np.float32), (2, 1, 1)) # [T+1,4,4] + poses[:, 0, 3] = np.array([0.1, 0.4], dtype=np.float32) # [T+1] + state = build_scene_state( + UnifiedAction(np.zeros((1, 57), dtype=np.float32), Action57DMask()), + sample={"agibot_link_poses": {"moving_link": poses}}, + ) + + renderer = object.__new__(unified_renderer.UnifiedRenderer) + renderer.server = _Server() + renderer._name_prefix = "" + renderer.robot_right = [] + renderer.robot_left = [] + renderer._robot_frame_handles_right = {} + renderer._robot_frame_handles_left = {} + renderer._robot_link_names = [] + renderer._robot_local_transforms = [] + renderer._current_robot = None + + renderer._load_robot_and_ik(state, _Entry()) + + assert renderer._ik_left is None + assert len(renderer._ik_right) == 2 + np.testing.assert_allclose(renderer._ik_right[1][0][0], np.array([0.4, 0.0, 0.0], dtype=np.float32)) diff --git a/cosmos_framework/data/vfm/action/urdf_visualizer/droid_franka_robotiq_2f85.xml b/cosmos_framework/data/vfm/action/urdf_visualizer/droid_franka_robotiq_2f85.xml new file mode 100644 index 0000000..ab3ca8d --- /dev/null +++ b/cosmos_framework/data/vfm/action/urdf_visualizer/droid_franka_robotiq_2f85.xml @@ -0,0 +1,377 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/cosmos_framework/data/vfm/action/urdf_visualizer/ik_solver.py b/cosmos_framework/data/vfm/action/urdf_visualizer/ik_solver.py new file mode 100644 index 0000000..8c17650 --- /dev/null +++ b/cosmos_framework/data/vfm/action/urdf_visualizer/ik_solver.py @@ -0,0 +1,852 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Robot-agnostic IK solver using pinocchio + MuJoCo. + +Supports any robot loaded from MJCF (Google Robot, Franka Panda, WidowX, etc). +Auto-detects EE frame, arm vs finger joints, and uses multi-start random seeding. + +The solver: +1. Auto-discovers the EE frame from a list of candidate names +2. Determines which joints are "arm" joints (actuated for IK) vs "finger" joints +3. Uses multi-start random sampling to avoid local minima +4. Optionally sets finger joint angles from gripper opening fractions +""" + +from functools import lru_cache +from typing import Any + +import numpy as np + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.embodiment_c_spec import ( + AGIBOT_GEAR_GRIPPER_OPEN_ANGLE_RAD, + AGIBOT_GEAR_HEAD_CAMERA_LINK_NAME, + AGIBOT_GEAR_LEFT_EE_LINK_NAME, + AGIBOT_GEAR_LEFT_GRIPPER_JOINT_MIMICS, + AGIBOT_GEAR_RIGHT_EE_LINK_NAME, + AGIBOT_GEAR_RIGHT_GRIPPER_JOINT_MIMICS, +) +from cosmos_framework.data.vfm.action.urdf_visualizer.robot_scene_model import ( + get_ee_frame_candidates, + get_mujoco_to_pinocchio_world_transform, + get_robot_config, + get_urdf_path, + get_visual_geom_ids, + resolve_robot_name_from_mjcf, +) + +# ── IK Solver ──────────────────────────────────────────────────────────────── + + +def _find_ee_frame(model, robot_name: str | None = None) -> int | None: + """Find the end-effector frame ID by trying candidate names. + + If robot_name is given and its config specifies ``ee_frame``, that name + is tried first before falling through to the generic candidates. + """ + cfg = get_robot_config(robot_name) if robot_name else {} + override = cfg.get("ee_frame") + if override: + fid = model.getFrameId(override) + if fid < model.nframes: + return fid + log.warning(f"Configured ee_frame '{override}' not found in Pinocchio model") + + for name in get_ee_frame_candidates(): + fid = model.getFrameId(name) + if fid < model.nframes: + return fid + log.warning(f"Could not find EE frame by name for robot '{robot_name}', skipping IK") + return None + + +@lru_cache(maxsize=1) +def _get_agibot_model() -> tuple[Any, dict[str, int]]: + """Load the AgiBot URDF model and cache the key frame IDs.""" + + import pinocchio as pin # pyright: ignore[reportMissingImports] + + urdf_path = get_urdf_path("embodiment_c") + if urdf_path is None: + raise FileNotFoundError("AgiBot URDF path is unavailable") + + model = pin.buildModelFromUrdf(urdf_path) + frame_ids = { + "head": model.getFrameId(AGIBOT_GEAR_HEAD_CAMERA_LINK_NAME), + "left": model.getFrameId(AGIBOT_GEAR_LEFT_EE_LINK_NAME), + "right": model.getFrameId(AGIBOT_GEAR_RIGHT_EE_LINK_NAME), + } + for frame_name, frame_id in frame_ids.items(): + if frame_id >= model.nframes: + raise ValueError(f"AgiBot frame {frame_name!r} missing from Pinocchio model") + return model, frame_ids + + +def _set_agibot_gripper_joint_configs( + model: Any, + joint_configs: np.ndarray, + *, + left_gripper_openings: np.ndarray | None, + right_gripper_openings: np.ndarray | None, +) -> None: + """Write AgiBot omnipicker finger joint angles from open fractions.""" + + side_specs = ( + ("left", left_gripper_openings, AGIBOT_GEAR_LEFT_GRIPPER_JOINT_MIMICS), + ("right", right_gripper_openings, AGIBOT_GEAR_RIGHT_GRIPPER_JOINT_MIMICS), + ) + lower = model.lowerPositionLimit + upper = model.upperPositionLimit + num_steps = int(joint_configs.shape[0]) + + for side, openings, joint_mimics in side_specs: + if openings is None: + continue + if len(openings) != num_steps: + log.warning( + f"AgiBot {side} gripper openings length {len(openings)} does not match IK frames {num_steps}; " + "leaving finger joints at neutral." + ) + continue + + joint_indices: list[tuple[int, float, float]] = [] + for joint_name, multiplier, offset in joint_mimics: + joint_id = model.getJointId(joint_name) + if joint_id >= model.njoints: + log.warning(f"AgiBot {side} gripper joint '{joint_name}' not found in Pinocchio model") + continue + joint_indices.append((model.idx_qs[joint_id], multiplier, offset)) + + for step_idx, opening in enumerate(openings): + primary_angle = -float(np.clip(opening, 0.0, 1.0)) * AGIBOT_GEAR_GRIPPER_OPEN_ANGLE_RAD + for q_idx, multiplier, offset in joint_indices: + joint_value = multiplier * primary_angle + offset + joint_configs[step_idx, q_idx] = np.clip(joint_value, lower[q_idx], upper[q_idx]) + + +def solve_agibot_trajectory_ik( + head_camera_poses: np.ndarray | None, + left_wrist_poses: np.ndarray | None, + right_wrist_poses: np.ndarray | None, + left_gripper_openings: np.ndarray | None = None, + right_gripper_openings: np.ndarray | None = None, + max_iter: int = 200, + dt: float = 0.2, + damp: float = 1e-4, + pos_tol_m: float = 5e-3, + rot_tol_rad: float = 5e-2, +) -> np.ndarray | None: + """Solve full-body AgiBot IK from calibrated head-camera and gripper-base trajectories.""" + + import pinocchio as pin # pyright: ignore[reportMissingImports] + + targets: list[np.ndarray] = [ + poses for poses in (head_camera_poses, left_wrist_poses, right_wrist_poses) if poses is not None + ] + if not targets: + return None + + num_steps = int(targets[0].shape[0]) + if any(int(poses.shape[0]) != num_steps for poses in targets): + raise ValueError("AgiBot IK requires head/left/right trajectories to have matching lengths") + + model, frame_ids = _get_agibot_model() + data = model.createData() + lower = model.lowerPositionLimit.copy() + upper = model.upperPositionLimit.copy() + + head_targets = head_camera_poses.astype(np.float32, copy=False) if head_camera_poses is not None else None + left_targets = left_wrist_poses.astype(np.float32, copy=False) if left_wrist_poses is not None else None + right_targets = right_wrist_poses.astype(np.float32, copy=False) if right_wrist_poses is not None else None + + task_specs: list[tuple[str, np.ndarray, int, float]] = [] + if head_targets is not None: + task_specs.append(("head", head_targets, frame_ids["head"], 0.75)) + if left_targets is not None: + task_specs.append(("left", left_targets, frame_ids["left"], 1.0)) + if right_targets is not None: + task_specs.append(("right", right_targets, frame_ids["right"], 1.0)) + + joint_configs = np.empty((num_steps, model.nq), dtype=np.float32) + q = pin.neutral(model) + rot_weight = 0.35 + + for step_idx in range(num_steps): + best_q = q.copy() + best_total = float("inf") + best_pos = float("inf") + best_rot = float("inf") + + for _ in range(max_iter): + pin.forwardKinematics(model, data, q) + pin.updateFramePlacements(model, data) + + errors: list[np.ndarray] = [] + jacobians: list[np.ndarray] = [] + max_pos_error = 0.0 + max_rot_error = 0.0 + + for _, poses, frame_id, task_weight in task_specs: + target = poses[step_idx] + placement = data.oMf[frame_id] + pos_err = target[:3, 3] - placement.translation + rot_err = pin.log3(target[:3, :3] @ placement.rotation.T) + max_pos_error = max(max_pos_error, float(np.linalg.norm(pos_err))) + max_rot_error = max(max_rot_error, float(np.linalg.norm(rot_err))) + + err6 = np.concatenate([pos_err, rot_weight * rot_err]).astype(np.float64, copy=False) + errors.append(task_weight * err6) + + jacobian = pin.computeFrameJacobian(model, data, q, frame_id, pin.LOCAL_WORLD_ALIGNED).copy() + jacobian[3:, :] *= rot_weight + jacobians.append(task_weight * jacobian) + + error_vec = np.concatenate(errors, axis=0) + total_error = float(np.linalg.norm(error_vec)) + if total_error < best_total: + best_total = total_error + best_q = q.copy() + best_pos = max_pos_error + best_rot = max_rot_error + + if max_pos_error < pos_tol_m and max_rot_error < rot_tol_rad: + break + + stacked_jacobian = np.concatenate(jacobians, axis=0) + normal = stacked_jacobian @ stacked_jacobian.T + damp * np.eye(stacked_jacobian.shape[0]) + velocity = stacked_jacobian.T @ np.linalg.solve(normal, error_vec) + q = pin.integrate(model, q, velocity * dt) + q = np.clip(q, lower, upper) + + q = best_q.copy() + joint_configs[step_idx] = q.astype(np.float32, copy=False) + log.info(f"AgiBot IK frame {step_idx}: max_pos={best_pos * 1000:.2f}mm, max_rot={np.degrees(best_rot):.2f}deg") + + _set_agibot_gripper_joint_configs( + model, + joint_configs, + left_gripper_openings=left_gripper_openings, + right_gripper_openings=right_gripper_openings, + ) + return joint_configs + + +def compute_agibot_link_poses_batch_from_configs( + joint_configs: np.ndarray, + link_names: list[str], +) -> dict[str, np.ndarray]: + """Compute AgiBot URDF link poses from solved full-body joint configurations.""" + + import pinocchio as pin # pyright: ignore[reportMissingImports] + + if joint_configs.size == 0: + return {link_name: np.empty((0, 4, 4), dtype=np.float32) for link_name in link_names} + + model, _ = _get_agibot_model() + data = model.createData() + frame_ids = {link_name: model.getFrameId(link_name) for link_name in link_names} + for link_name, frame_id in frame_ids.items(): + if frame_id >= model.nframes: + raise ValueError(f"AgiBot link frame {link_name!r} missing from Pinocchio model") + + num_steps = int(joint_configs.shape[0]) + link_poses = {link_name: np.empty((num_steps, 4, 4), dtype=np.float32) for link_name in link_names} + + for step_idx in range(num_steps): + q = joint_configs[step_idx].astype(np.float64, copy=False) + pin.forwardKinematics(model, data, q) + pin.updateFramePlacements(model, data) + for link_name, frame_id in frame_ids.items(): + placement = data.oMf[frame_id] + transform = np.eye(4, dtype=np.float32) + transform[:3, :3] = placement.rotation.astype(np.float32, copy=False) + transform[:3, 3] = placement.translation.astype(np.float32, copy=False) + link_poses[link_name][step_idx] = transform + + return link_poses + + +def solve_trajectory_ik( + mjcf_path: str, + world_ee_positions: np.ndarray, + gripper_openings: np.ndarray | None = None, + world_ee_orientations: np.ndarray | None = None, + robot_name: str | None = None, + max_random_samples: int = 50_000, + seed: int = 42, +) -> np.ndarray | None: + """Solve IK for a sequence of world-space EE poses (robot-agnostic). + + Args: + mjcf_path: Path to the MJCF XML file. + world_ee_positions: (T, 3) target EE positions. + gripper_openings: (T,) gripper opening fractions [0=closed, 1=open]. + world_ee_orientations: (T, 3, 3) target EE rotation matrices. + robot_name: Robot name for config lookup (optional, inferred from path). + max_random_samples: Number of random configs to try for initial seed. + seed: Random seed. + + Returns: + (T, nq) joint configurations, or None if IK fails. + """ + import pinocchio as pin + + model, cfg = _build_pinocchio_model(mjcf_path, robot_name) + log.info(f"IK: pinocchio model nq={model.nq}") + data = model.createData() + ee_id = _find_ee_frame(model, robot_name) + if ee_id is None: + return None + ee_name = model.frames[ee_id].name + log.info(f"IK: using EE frame '{ee_name}' (id={ee_id}), nq={model.nq}") + + T = len(world_ee_positions) + use_6dof = world_ee_orientations is not None and len(world_ee_orientations) == T + + # Apply TCP offset: dataset EE poses may be at the TCP (e.g. ee_gripper_link) + # while the Pinocchio frame is at the kinematic link origin (e.g. gripper_link). + # Convert target TCP poses → IK link-frame targets: + # p_link = p_tcp - R_tcp @ tcp_offset + tcp_offset = cfg.get("tcp_offset") + if tcp_offset is not None: + tcp_offset = np.asarray(tcp_offset, dtype=np.float32) + log.info(f"IK: applying TCP offset {tcp_offset} to target positions") + world_ee_positions = world_ee_positions.copy() + for t in range(T): + if use_6dof: + world_ee_positions[t] -= world_ee_orientations[t] @ tcp_offset + else: + world_ee_positions[t] -= tcp_offset + + lower = model.lowerPositionLimit.copy() + upper = model.upperPositionLimit.copy() + + # Determine arm joints vs finger joints + # After model reduction, arm joints are first, fingers follow + n_arm = cfg.get("n_arm_joints", model.nq - 2) # default: all but last 2 are arm + n_finger = model.nq - n_arm + + log.info(f"IK: {n_arm} arm joints + {n_finger} finger joints") + + # ── 6-DoF CLIK (position + orientation) ── + def _ik_6dof(target_pos, target_rot, q_init, max_iter=800, eps_pos=5e-5, eps_rot=1e-3, dt=0.1, damp=1e-4): + q = q_init.copy() + best_q = q.copy() + best_total = float("inf") + stall_count = 0 + rot_weight = 1.0 + + for it in range(max_iter): + pin.forwardKinematics(model, data, q) + pin.updateFramePlacements(model, data) + + pos_err = target_pos - data.oMf[ee_id].translation + pos_norm = np.linalg.norm(pos_err) + R_err = target_rot @ data.oMf[ee_id].rotation.T + rot_err = pin.log3(R_err) + rot_norm = np.linalg.norm(rot_err) + + if pos_norm < eps_pos and rot_norm < eps_rot: + return q, pos_norm, rot_norm, it + 1 + + total = pos_norm + 0.05 * rot_norm + if total < best_total: + best_total = total + best_q = q.copy() + stall_count = 0 + else: + stall_count += 1 + + if stall_count > 50 and rot_weight > 0.1: + rot_weight *= 0.8 + stall_count = 0 + if stall_count > 150: + break + + err6 = np.concatenate([pos_err, rot_weight * rot_err]) + J = pin.computeFrameJacobian(model, data, q, ee_id, pin.LOCAL_WORLD_ALIGNED).copy() + J[3:, :] *= rot_weight + # Zero out finger joint columns + J[:, n_arm:] = 0 + JJt = J @ J.T + damp * np.eye(6) + v = J.T @ np.linalg.solve(JJt, err6) + v[n_arm:] = 0 # don't move finger joints + + q = pin.integrate(model, q, v * dt) + q = np.clip(q, lower, upper) + + pin.forwardKinematics(model, data, best_q) + pin.updateFramePlacements(model, data) + pos_norm = np.linalg.norm(target_pos - data.oMf[ee_id].translation) + rot_norm = np.linalg.norm(pin.log3(target_rot @ data.oMf[ee_id].rotation.T)) + return best_q, pos_norm, rot_norm, max_iter + + # ── 3-DoF CLIK (position only) ── + def _ik_3dof(target_pos, q_init, max_iter=500, eps=5e-5, dt=0.15, damp=1e-4): + q = q_init.copy() + for it in range(max_iter): + pin.forwardKinematics(model, data, q) + pin.updateFramePlacements(model, data) + err = target_pos - data.oMf[ee_id].translation + if np.linalg.norm(err) < eps: + return q, np.linalg.norm(err), it + 1 + J = pin.computeFrameJacobian(model, data, q, ee_id, pin.LOCAL_WORLD_ALIGNED)[:3] + J[:, n_arm:] = 0 + v = J.T @ np.linalg.solve(J @ J.T + damp * np.eye(3), err) + v[n_arm:] = 0 + q = pin.integrate(model, q, v * dt) + q = np.clip(q, lower, upper) + return q, np.linalg.norm(err), max_iter + + def _solve_full_trajectory(seed_q): + configs = [] + max_pe = 0.0 + max_re = 0.0 + q = seed_q.copy() + for t in range(T): + if use_6dof: + q, pe, re, _ = _ik_6dof(world_ee_positions[t], world_ee_orientations[t], q) + max_pe = max(max_pe, float(pe)) + max_re = max(max_re, float(re)) + else: + q, pe, _ = _ik_3dof(world_ee_positions[t], q) + max_pe = max(max_pe, float(pe)) + configs.append(q.copy()) + return np.array(configs), max_pe, max_re + + # ── Multi-start seed search ── + # For robots with a base rotation joint (Google Robot torso, WidowX waist), + # search multiple rotation basins. For Franka (no base rotation freedom), + # use a single wider search. + base_joint_range = upper[0] - lower[0] + if base_joint_range > 4.0: + # Wide base rotation — split into basins (Google Robot, WidowX) + n_basins = 4 + basin_size = base_joint_range / n_basins + basins = [] + for i in range(n_basins): + b_lo = lower[0] + i * basin_size + b_hi = lower[0] + (i + 1) * basin_size + basins.append((b_lo, b_hi)) + else: + # No wide base rotation — single basin (Franka) + basins = [(lower[0], upper[0])] + + samples_per_basin = max_random_samples // len(basins) + target0_pos = world_ee_positions[0] + target0_rot = world_ee_orientations[0] if use_6dof else None + + best_overall_configs = None + best_overall_score = float("inf") + best_basin_info = "" + best_max_pe = 0.0 + best_max_re = 0.0 + + for basin_idx, (b_lo, b_hi) in enumerate(basins): + rng = np.random.RandomState(seed + basin_idx) + basin_lower = lower.copy() + basin_upper = upper.copy() + basin_lower[0] = max(lower[0], b_lo) + basin_upper[0] = min(upper[0], b_hi) + + # Find best seed in this basin + basin_best_q = pin.neutral(model) + basin_best_q[0] = (b_lo + b_hi) / 2 + basin_best_score = float("inf") + + for _ in range(samples_per_basin): + q = rng.uniform(basin_lower, basin_upper) + pin.forwardKinematics(model, data, q) + pin.updateFramePlacements(model, data) + pos_err = np.linalg.norm(data.oMf[ee_id].translation - target0_pos) + + if target0_rot is not None: + rot_err = np.linalg.norm(pin.log3(target0_rot.T @ data.oMf[ee_id].rotation)) + score = pos_err + 0.3 * rot_err + else: + score = pos_err + + if score < basin_best_score: + basin_best_score = score + basin_best_q = q.copy() + if pos_err < 0.005 and (target0_rot is None or rot_err < 0.1): + break + + if basin_best_score > 0.5: + continue + + configs, max_pe, max_re = _solve_full_trajectory(basin_best_q) + traj_score = max_pe + 0.05 * max_re + log.info( + f" Basin [{b_lo:+.1f}, {b_hi:+.1f}]: seed_score={basin_best_score:.4f}, " + f"traj max_pos={max_pe * 1000:.1f}mm, max_rot={np.degrees(max_re):.1f}°, " + f"j0={basin_best_q[0]:+.2f}rad" + ) + + if traj_score < best_overall_score: + best_overall_score = traj_score + best_overall_configs = configs + best_basin_info = f"j0_basin=[{b_lo:+.1f},{b_hi:+.1f}], seed_j0={basin_best_q[0]:+.2f}rad" + best_max_pe = max_pe + best_max_re = max_re + + if best_overall_configs is None: + log.warning("IK failed: no basin converged") + return None + + configs = best_overall_configs + if use_6dof: + log.info( + f"IK solved ({T} frames, 6-DoF): max_pos={best_max_pe * 1000:.2f}mm, max_rot={np.degrees(best_max_re):.1f}° [{best_basin_info}]" + ) + else: + log.info(f"IK solved ({T} frames, 3-DoF): max_pos={best_max_pe * 1000:.2f}mm [{best_basin_info}]") + + # ── Set finger joints from gripper openings ── + if n_finger > 0: + finger_min = cfg.get("finger_min", lower[n_arm]) + finger_max = cfg.get("finger_max", upper[n_arm]) + else: + finger_min = 0.0 + finger_max = 0.0 + close_is_max = cfg.get("finger_close_is_max", True) + finger_joint_names = cfg.get("finger_joint_names") + + if gripper_openings is not None and len(gripper_openings) == T: + # Find finger joint indices by name if specified (e.g., Robotiq driver joints) + if finger_joint_names: + # Use Pinocchio joint name lookup + finger_indices = [] + for fjn in finger_joint_names: + # Pinocchio joint names include the joint name from MJCF + jid = model.getJointId(fjn) + if jid < model.njoints: + # Pinocchio joint index → qpos index + qi = model.idx_qs[jid] + finger_indices.append(qi) + else: + log.warning(f"Finger joint '{fjn}' not found in Pinocchio model") + if not finger_indices: + finger_indices = list(range(n_arm, n_arm + n_finger)) + else: + finger_indices = list(range(n_arm, n_arm + n_finger)) + + for t in range(T): + g = float(np.clip(gripper_openings[t], 0.0, 1.0)) + if close_is_max: + # Robotiq/Google Robot: high angle = closed + angle = finger_max - g * (finger_max - finger_min) + else: + # Franka/WidowX: high value = open + angle = finger_min + g * (finger_max - finger_min) + + for ji in finger_indices: + # WidowX right finger is negative range + if lower[ji] < 0 and upper[ji] < 0: + configs[t, ji] = -(finger_min + g * (finger_max - finger_min)) + else: + configs[t, ji] = angle + log.info( + f"Finger joints set from gripper openings ({gripper_openings.min():.2f} to {gripper_openings.max():.2f})" + ) + + return configs + + +def _build_pinocchio_model(mjcf_path: str, robot_name: str | None = None): + """Build a pinocchio model, reducing to arm+finger joints if configured. + + For URDF models (SimplerEnv), builds from URDF and locks non-arm joints. + For MJCF models (Menagerie), builds from MJCF and locks non-arm joints. + Shared by solve_trajectory_ik and compute_fk_ee_poses. + """ + import pinocchio as pin + + cfg = get_robot_config(robot_name) if robot_name else {} + urdf_path = get_urdf_path(robot_name) if robot_name else None + + if urdf_path: + full_model = pin.buildModelFromUrdf(urdf_path) + else: + full_model = pin.buildModelFromMJCF(mjcf_path) + + # Reduce model to arm + finger joints only (lock base, wheels, etc.) + arm_joint_names = cfg.get("arm_joints", []) + finger_jnames = cfg.get("finger_joint_names", []) + keep_names = set(arm_joint_names) | set(finger_jnames) + + if keep_names: + lock_ids = [ji for ji in range(1, full_model.njoints) if full_model.names[ji] not in keep_names] + if lock_ids: + q_ref = pin.neutral(full_model) + model = pin.buildReducedModel(full_model, lock_ids, q_ref) + else: + model = full_model + else: + model = full_model + + return model, cfg + + +def compute_fk_ee_poses( + mjcf_path: str, + joint_configs: np.ndarray, + robot_name: str | None = None, +) -> tuple[np.ndarray, np.ndarray]: + """Run FK and return EE positions and orientations.""" + import pinocchio as pin + + model, _ = _build_pinocchio_model(mjcf_path, robot_name) + data = model.createData() + ee_id = _find_ee_frame(model, robot_name) + if ee_id is None: + log.warning(f"Skipping FK — no EE frame found for robot '{robot_name}'") + T = len(joint_configs) + return np.zeros((T, 3)), np.zeros((T, 3, 3)) + + T = len(joint_configs) + fk_positions = np.zeros((T, 3)) + fk_orientations = np.zeros((T, 3, 3)) + + for t in range(T): + pin.forwardKinematics(model, data, joint_configs[t]) + pin.updateFramePlacements(model, data) + fk_positions[t] = data.oMf[ee_id].translation.copy() + fk_orientations[t] = data.oMf[ee_id].rotation.copy() + + return fk_positions, fk_orientations + + +def verify_ik_with_fk( + mjcf_path: str, + joint_configs: np.ndarray, + target_positions: np.ndarray, + target_orientations: np.ndarray | None = None, +) -> dict: + """Verify IK solution by running FK and comparing to targets.""" + import pinocchio as pin + + fk_pos, fk_rot = compute_fk_ee_poses(mjcf_path, joint_configs) + if fk_pos is None: + return None + + T = len(joint_configs) + pos_errors_mm = np.linalg.norm(fk_pos - target_positions, axis=1) * 1000 + + rot_errors_deg = None + if target_orientations is not None: + rot_errors_deg = np.zeros(T) + for t in range(T): + R_err = target_orientations[t].T @ fk_rot[t] + angle = np.linalg.norm(pin.log3(R_err)) + rot_errors_deg[t] = np.degrees(angle) + + summary = f"FK Verification ({T} frames): pos mean={pos_errors_mm.mean():.2f}mm max={pos_errors_mm.max():.2f}mm" + if rot_errors_deg is not None: + summary += f", rot mean={rot_errors_deg.mean():.1f}° max={rot_errors_deg.max():.1f}°" + + return { + "fk_positions": fk_pos, + "fk_orientations": fk_rot, + "pos_errors_mm": pos_errors_mm, + "rot_errors_deg": rot_errors_deg, + "summary": summary, + } + + +def compute_mujoco_geom_transforms( + mjcf_path: str, + joint_configs: np.ndarray, +) -> tuple[ + list[list[tuple[np.ndarray, np.ndarray]]], + list[tuple[np.ndarray, np.ndarray]] | None, + list[tuple[np.ndarray, np.ndarray]] | None, + dict[str, list[tuple[np.ndarray, np.ndarray]]] | None, +]: + """Compute MuJoCo geom/body/site transforms in the Pinocchio-aligned world. + + Also extracts camera site pose, EE body pose, and named body/site frames. + + Important: some MJCFs (notably MuJoCo Menagerie's Google Robot) include a + fixed ``worldbody -> root_body`` transform that Pinocchio's + ``buildModelFromMJCF()`` omits. Dataset poses and IK targets already live in + Pinocchio's root-free world, so we explicitly remove that MuJoCo root + transform here before returning any MuJoCo-derived poses. + + Returns: + (all_geom_transforms, camera_poses_or_None, ee_poses_or_None, robot_frames_or_None) + - all_geom_transforms: list of per-frame geom transforms [(pos, mat), ...] + - camera_poses: list of (pos, mat) per frame for 'camera_site', or None if no site. + - ee_poses: list of (pos, mat) per frame for the EE body, or None. + - robot_frames: dict mapping ``body:`` / ``site:`` to per-frame poses. + """ + import mujoco + + model = mujoco.MjModel.from_xml_path(mjcf_path) + data = mujoco.MjData(model) + + visual_geom_ids = get_visual_geom_ids(model) + + # Determine which robot config applies (by matching MJCF filename) + robot_name = resolve_robot_name_from_mjcf(mjcf_path) + cfg = get_robot_config(robot_name) if robot_name is not None else {} + + # Find camera_site if it exists + camera_site_id = mujoco.mj_name2id(model, mujoco.mjtObj.mjOBJ_SITE, "camera_site") + has_camera_site = camera_site_id >= 0 + if has_camera_site: + body_id = model.site_bodyid[camera_site_id] + body_name = mujoco.mj_id2name(model, mujoco.mjtObj.mjOBJ_BODY, body_id) or "?" + log.info(f"Found camera_site (id={camera_site_id}) on body '{body_name}'") + + # Also look up camera body (e.g. zed_mini for DROID) + camera_body_id = -1 + camera_body_name = cfg.get("camera_body") + if camera_body_name and not has_camera_site: + camera_body_id = mujoco.mj_name2id(model, mujoco.mjtObj.mjOBJ_BODY, camera_body_name) + if camera_body_id >= 0: + log.info(f"Found camera body '{camera_body_name}' (id={camera_body_id})") + + all_transforms = [] + camera_poses = [] if (has_camera_site or camera_body_id >= 0) else None + robot_frame_specs = [] + for body_id in range(1, model.nbody): + body_name = mujoco.mj_id2name(model, mujoco.mjtObj.mjOBJ_BODY, body_id) or "" + if body_name: + robot_frame_specs.append(("body", body_name, body_id)) + for site_id in range(model.nsite): + site_name = mujoco.mj_id2name(model, mujoco.mjtObj.mjOBJ_SITE, site_id) or "" + if site_name: + robot_frame_specs.append(("site", site_name, site_id)) + robot_frames = {f"{kind}:{name}": [] for kind, name, _ in robot_frame_specs} if robot_frame_specs else None + + # Find EE body for extracting FK-derived EE pose + ee_body_id = -1 + ee_override = cfg.get("ee_frame") + ee_candidates = get_ee_frame_candidates(robot_name) + for candidate in ee_candidates: + bid = mujoco.mj_name2id(model, mujoco.mjtObj.mjOBJ_BODY, candidate) + if bid >= 0: + ee_body_id = bid + break + ee_poses = [] if ee_body_id >= 0 else None + + # Find driver joint indices for robots with finger_joint_names + finger_joint_names = cfg.get("finger_joint_names", []) + arm_jnames = cfg.get("arm_joints", []) if cfg else [] + # Indices to pin during constraint settling: arm joints + driver joints + n_arm = cfg.get("n_arm_joints", 7) if cfg else 7 + if arm_jnames: + # Use name-based mapping for arm joint indices + pin_indices = [] + for jn in arm_jnames: + jid = mujoco.mj_name2id(model, mujoco.mjtObj.mjOBJ_JOINT, jn) + if jid >= 0: + pin_indices.append(model.jnt_qposadr[jid]) + else: + pin_indices = list(range(n_arm)) + if finger_joint_names: + for fjn in finger_joint_names: + for ji in range(model.njnt): + jname = mujoco.mj_id2name(model, mujoco.mjtObj.mjOBJ_JOINT, ji) or "" + if jname == fjn: + qi = model.jnt_qposadr[ji] + pin_indices.append(qi) + + # Build mapping from IK output (pinocchio reduced model) to MuJoCo qpos indices. + # When a URDF is used with model reduction, the IK output has only arm+finger + # joints in pinocchio order. We need to map those to MuJoCo's qpos order. + arm_jnames = cfg.get("arm_joints", []) if cfg else [] + pin_to_mj_map = None # None = direct mapping (qpos[:len(q)] = q) + if arm_jnames: + # Build ordered list: arm joints first, then finger joints + ordered_jnames = list(arm_jnames) + list(finger_joint_names) + mj_indices = [] + for jn in ordered_jnames: + jid = mujoco.mj_name2id(model, mujoco.mjtObj.mjOBJ_JOINT, jn) + if jid >= 0: + mj_indices.append(model.jnt_qposadr[jid]) + if mj_indices: + pin_to_mj_map = mj_indices + + # If joint_configs has more columns than n_arm, the extra column is a + # normalized gripper signal (raw UR: 0=open, 1=closed). Robotiq ctrl + # matches: 0=open, 255=closed → ctrl = raw * finger_max (no inversion). + finger_max = cfg.get("finger_max", 0.0) if cfg else 0.0 + has_gripper_ctrl = finger_max > 0.0 and model.nu > n_arm and joint_configs.shape[1] > n_arm + + def _apply_world_correction(pos: np.ndarray, mat: np.ndarray) -> tuple[np.ndarray, np.ndarray]: + """Map one MuJoCo world pose into the root-free Pinocchio world.""" + corrected_pos = world_correction[:3, :3] @ pos + world_correction[:3, 3] + corrected_mat = world_correction[:3, :3] @ mat + return corrected_pos.astype(np.float32), corrected_mat.astype(np.float32) + + for q in joint_configs: + if pin_to_mj_map and len(q) == len(pin_to_mj_map): + # Full arm+finger mapping from IK (e.g. Franka): use all joints. + data.qpos[:] = 0 + for i, mi in enumerate(pin_to_mj_map): + data.qpos[mi] = q[i] + else: + # For robots with a separate gripper ctrl signal (e.g. UR5e), + # only write arm joints — the 7th column is a raw gripper value, + # not a qpos DOF. For other robots the pinocchio output already + # includes finger joints in the trailing columns; write them all. + n_set = n_arm if has_gripper_ctrl else len(q) + data.qpos[:n_set] = q[:n_set] + mujoco.mj_forward(model, data) + + # For robots with equality constraints (e.g., Robotiq 4-bar linkage), + # step physics to let constraints resolve the passive linkage joints. + if model.neq > 0: + data.qvel[:] = 0 + data.ctrl[:] = 0 + # Raw UR gripper maps directly to Robotiq ctrl: 0=open, 255=closed. + if has_gripper_ctrl: + data.ctrl[-1] = float(q[n_arm]) * finger_max + gripper_ctrl_val = float(data.ctrl[-1]) if model.nu > 0 else 0.0 + saved = data.qpos[pin_indices].copy() + for _ in range(200): + mujoco.mj_step(model, data) + data.qpos[pin_indices] = saved + if model.nu > 0: + data.ctrl[-1] = gripper_ctrl_val # keep gripper ctrl during settling + data.qvel[:] = 0 + mujoco.mj_forward(model, data) + world_correction = get_mujoco_to_pinocchio_world_transform(model, data, robot_name) + + frame_transforms = [] + for gi in visual_geom_ids: + pos = data.geom_xpos[gi].copy() + mat = data.geom_xmat[gi].reshape(3, 3).copy() + frame_transforms.append(_apply_world_correction(pos, mat)) + all_transforms.append(frame_transforms) + + # Extract camera site pose + if has_camera_site: + cam_pos = data.site_xpos[camera_site_id].copy() + cam_mat = data.site_xmat[camera_site_id].reshape(3, 3).copy() + camera_poses.append(_apply_world_correction(cam_pos, cam_mat)) + elif camera_body_id >= 0: + cam_pos = data.xpos[camera_body_id].copy() + cam_mat = data.xmat[camera_body_id].reshape(3, 3).copy() + camera_poses.append(_apply_world_correction(cam_pos, cam_mat)) + + # Extract EE body pose + if ee_body_id >= 0: + ee_pos = data.xpos[ee_body_id].copy() + ee_mat = data.xmat[ee_body_id].reshape(3, 3).copy() + ee_poses.append(_apply_world_correction(ee_pos, ee_mat)) + + if robot_frames is not None: + for kind, name, frame_id in robot_frame_specs: + if kind == "body": + pos = data.xpos[frame_id].copy() + mat = data.xmat[frame_id].reshape(3, 3).copy() + else: + pos = data.site_xpos[frame_id].copy() + mat = data.site_xmat[frame_id].reshape(3, 3).copy() + robot_frames[f"{kind}:{name}"].append(_apply_world_correction(pos, mat)) + + return all_transforms, camera_poses, ee_poses, robot_frames diff --git a/cosmos_framework/data/vfm/action/urdf_visualizer/robot_scene_model.py b/cosmos_framework/data/vfm/action/urdf_visualizer/robot_scene_model.py new file mode 100644 index 0000000..0dfc5c7 --- /dev/null +++ b/cosmos_framework/data/vfm/action/urdf_visualizer/robot_scene_model.py @@ -0,0 +1,216 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Public robot-scene abstraction for the action viewer. + +`RobotSceneModel` is the viewer-facing contract for robot assets and +kinematics. It wraps the lower-level MuJoCo / Pinocchio helpers so callers do +not need to coordinate mesh loading, IK, frame extraction, or world-alignment +corrections themselves. +""" + +from __future__ import annotations + +import os +from collections.abc import Callable +from dataclasses import dataclass +from typing import Any + +import numpy as np + +from cosmos_framework.data.vfm.action.urdf_visualizer import urdf_loader + +MeshSpec = tuple[str, object, np.ndarray] +FramePose = tuple[np.ndarray, np.ndarray] +FrameSeries = dict[str, list[FramePose]] + + +@dataclass(frozen=True) +class RobotTrajectory: + """Render-ready robot geometry for one solved trajectory.""" + + mesh_transforms: list[list[FramePose]] + named_frames: FrameSeries + + +def get_robot_config(robot_name: str) -> dict[str, Any]: + """Return the static config for one supported robot.""" + cfg = urdf_loader.ROBOT_CONFIGS.get(robot_name) + if cfg is None: + raise ValueError(f"Unknown robot: {robot_name}. Available: {list(urdf_loader.ROBOT_CONFIGS)}") + return cfg + + +def get_urdf_path(robot_name: str) -> str | None: + """Return the preferred URDF path for one robot, if available.""" + return urdf_loader.get_urdf_path(robot_name) + + +def get_mjcf_path(robot_name: str) -> str: + """Return the canonical MJCF path for one robot.""" + return urdf_loader.get_mjcf_path(robot_name) + + +def get_mujoco_to_pinocchio_world_transform(model: Any, data: Any, robot_name: str | None = None) -> np.ndarray: + """Map MuJoCo world poses into the root-free Pinocchio world.""" + return urdf_loader.get_mujoco_to_pinocchio_world_transform(model, data, robot_name) + + +def get_visual_geom_ids(model: Any) -> list[int]: + """Return the MuJoCo visual geom order used by the viewer.""" + return urdf_loader._get_visual_geom_ids(model) + + +def get_ee_frame_candidates(robot_name: str | None = None) -> list[str]: + """Return the ordered end-effector frame candidates for one robot.""" + if robot_name is None: + return list(urdf_loader._EE_FRAME_CANDIDATES) + cfg = urdf_loader.ROBOT_CONFIGS.get(robot_name, {}) + ee_frame = cfg.get("ee_frame") + if ee_frame is None: + return list(urdf_loader._EE_FRAME_CANDIDATES) + return [str(ee_frame), *urdf_loader._EE_FRAME_CANDIDATES] + + +def get_robot_loaders() -> dict[str, Callable[[], tuple[list[MeshSpec], np.ndarray]]]: + """Return the low-level robot mesh loader registry.""" + return urdf_loader.get_robot_loaders() + + +def resolve_robot_name_from_mjcf(mjcf_path: str) -> str | None: + """Infer a configured robot name from an MJCF filename.""" + filename = os.path.basename(mjcf_path) + for robot_name, cfg in urdf_loader.ROBOT_CONFIGS.items(): + if filename == cfg.get("mjcf"): + return robot_name + return None + + +def _copy_mesh_specs(meshes: list[MeshSpec]) -> list[MeshSpec]: + """Copy mesh transforms while reusing immutable mesh geometry.""" + return [(name, mesh, transform.copy().astype(np.float32)) for name, mesh, transform in meshes] + + +def _transform_mesh_specs(meshes: list[MeshSpec], base_pose: np.ndarray | None) -> list[MeshSpec]: + """Apply one rigid transform to a list of world-space mesh poses.""" + copied = _copy_mesh_specs(meshes) + if base_pose is None: + return copied + transformed: list[MeshSpec] = [] + for name, mesh, transform in copied: + transformed.append((name, mesh, (base_pose @ transform).astype(np.float32))) # [4,4] + return transformed + + +def _remove_pose_base(poses_world: np.ndarray, base_pose: np.ndarray | None) -> np.ndarray: + """Map world-space wrist poses back into one arm-local base frame.""" + if base_pose is None: + return poses_world.astype(np.float32) + base_inv = np.linalg.inv(base_pose).astype(np.float32) # [4,4] + return np.einsum("ij,njk->nik", base_inv, poses_world).astype(np.float32) # [T,4,4] + + +def _apply_base_to_mesh_transforms( + transforms: list[list[FramePose]], + base_pose: np.ndarray | None, +) -> list[list[FramePose]]: + """Apply one rigid base pose to per-geom world transforms.""" + if base_pose is None: + return transforms + base_rot = base_pose[:3, :3].astype(np.float32) # [3,3] + base_pos = base_pose[:3, 3].astype(np.float32) # [3] + transformed: list[list[FramePose]] = [] + for frame_transforms in transforms: + transformed_frame: list[FramePose] = [] + for pos, rot in frame_transforms: + transformed_frame.append((base_rot @ pos + base_pos, base_rot @ rot)) + transformed.append(transformed_frame) + return transformed + + +def _apply_base_to_named_frames( + frames: FrameSeries, + base_pose: np.ndarray | None, +) -> FrameSeries: + """Apply one rigid base pose to named body/site frames.""" + if base_pose is None: + return frames + base_rot = base_pose[:3, :3].astype(np.float32) # [3,3] + base_pos = base_pose[:3, 3].astype(np.float32) # [3] + transformed: FrameSeries = {} + for frame_key, poses in frames.items(): + transformed[frame_key] = [(base_rot @ pos + base_pos, base_rot @ rot) for pos, rot in poses] + return transformed + + +class RobotSceneModel: + """Single public abstraction for robot meshes, IK/FK, and debug frames.""" + + def __init__(self, robot_name: str) -> None: + self.robot_name = robot_name + self._config = get_robot_config(robot_name) + self._mjcf_path = get_mjcf_path(robot_name) + self._home_meshes: list[MeshSpec] | None = None + + @property + def mjcf_path(self) -> str: + """Return the underlying MJCF path for this robot.""" + return self._mjcf_path + + @property + def ee_frame_name(self) -> str: + """Return the canonical IK / debug frame name for this robot.""" + return str(self._config.get("ee_frame", "ee_frame")) + + def get_home_meshes(self, base_pose: np.ndarray | None = None) -> list[MeshSpec]: + """Return home-pose meshes in the requested world frame.""" + if self._home_meshes is None: + loaders = get_robot_loaders() + loader = loaders.get(self.robot_name) + if loader is None: + raise ValueError(f"No robot loader registered for {self.robot_name}") + meshes, _ = loader() + self._home_meshes = _copy_mesh_specs(meshes) + return _transform_mesh_specs(self._home_meshes, base_pose) + + def solve_visual_trajectory( + self, + wrist_poses_world: np.ndarray | None, + gripper_openings: np.ndarray | None = None, + to_opencv: np.ndarray | None = None, + base_pose: np.ndarray | None = None, + ) -> RobotTrajectory | None: + """Solve IK for world-space wrist poses and return render-ready robot state.""" + if wrist_poses_world is None or len(wrist_poses_world) < 2: + return None + + local_wrist_poses = _remove_pose_base(wrist_poses_world, base_pose) # [T,4,4] + target_positions = local_wrist_poses[:, :3, 3].astype(np.float32) # [T,3] + target_rotations = local_wrist_poses[:, :3, :3].astype(np.float32) # [T,3,3] + if to_opencv is not None and not np.allclose(to_opencv, np.eye(3, dtype=np.float32)): + target_rotations = target_rotations @ to_opencv.T[None] # [T,3,3] + + from cosmos_framework.data.vfm.action.urdf_visualizer.ik_solver import ( + compute_fk_ee_poses, + compute_mujoco_geom_transforms, + solve_trajectory_ik, + ) + + joint_configs = solve_trajectory_ik( + self.mjcf_path, + target_positions, + gripper_openings=gripper_openings, + world_ee_orientations=target_rotations, + robot_name=self.robot_name, + ) + if joint_configs is None: + return None + + fk_pos, fk_rot = compute_fk_ee_poses(self.mjcf_path, joint_configs, robot_name=self.robot_name) + mesh_transforms, _, _, named_frames = compute_mujoco_geom_transforms(self.mjcf_path, joint_configs) + public_frames: FrameSeries = {} if named_frames is None else dict(named_frames) + public_frames[f"ik:{self.ee_frame_name}"] = list(zip(fk_pos, fk_rot, strict=True)) + + mesh_transforms = _apply_base_to_mesh_transforms(mesh_transforms, base_pose) + public_frames = _apply_base_to_named_frames(public_frames, base_pose) + return RobotTrajectory(mesh_transforms=mesh_transforms, named_frames=public_frames) diff --git a/cosmos_framework/data/vfm/action/urdf_visualizer/unified_action.py b/cosmos_framework/data/vfm/action/urdf_visualizer/unified_action.py new file mode 100644 index 0000000..2c3d76b --- /dev/null +++ b/cosmos_framework/data/vfm/action/urdf_visualizer/unified_action.py @@ -0,0 +1,535 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Canonical 57D action representation with explicit input formats. + +57D layout:: + + [ego(9) | R_wrist(9) | R_fingers(15) | L_wrist(9) | L_fingers(15)] + +Each 9D SE(3) slot is ``[pos(3) + rot6d(6)]``. +Each finger slot is 3D (position in wrist-local frame), 5 fingers × 3D = 15D. + +Any supported action format is converted to ``UnifiedAction(action_57d, mask)`` +before the viewer processes it. The mask explicitly declares which slots are valid. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from enum import Enum +from typing import Any + +import numpy as np +import torch + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.embodiment_c_fk import build_viewer_batch +from cosmos_framework.data.vfm.action.embodiment_c_spec import get_embodiment_c_embodiment_spec +from cosmos_framework.data.vfm.action.pose_utils import convert_rotation + +FINGER_NAMES = ("thumb", "index", "middle", "ring", "pinky") +ALL_FINGERS = (True, True, True, True, True) +NO_FINGERS = (False, False, False, False, False) + + +class ActionFormat(str, Enum): + """Explicit raw action layouts supported by the viewer pipeline.""" + + EGO_9D = "9d" + SINGLE_ARM_10D = "10d" + DUAL_ARM_20D = "20d" + UNIFIED_57D = "57d" + + @property + def expected_dim(self) -> int: + """Return the exact trailing dimension required by this format.""" + return { + ActionFormat.EGO_9D: 9, + ActionFormat.SINGLE_ARM_10D: 10, + ActionFormat.DUAL_ARM_20D: 20, + ActionFormat.UNIFIED_57D: 57, + }[self] + + +# ─── Data Structures ───────────────────────────────────────────────────────── + + +@dataclass +class Action57DMask: + """Per-component validity for the 57D layout. + + ``right_fingers`` / ``left_fingers`` are tuples of 5 bools + (thumb, index, middle, ring, pinky) — supports any combination from + 2-finger grippers to full 5-finger hands. + """ + + ego: bool = False + right_wrist: bool = False + right_fingers: tuple[bool, ...] = NO_FINGERS + left_wrist: bool = False + left_fingers: tuple[bool, ...] = NO_FINGERS + + +@dataclass +class UnifiedAction: + """Canonical 57D action for the viewer pipeline. + + ``action`` is always shape ``(T, 57)`` with invalid slots zero-padded. + ``gripper_right`` / ``gripper_left`` carry auxiliary scalar gripper data + for embodiments that don't map to finger positions (V-shape visualisation). + """ + + action: np.ndarray # (T, 57) + mask: Action57DMask + gripper_right: np.ndarray | None = None # (T,) scalar 0-1 + gripper_left: np.ndarray | None = None # (T,) scalar 0-1 + + +@dataclass +class SceneState: + """Render-ready world-space geometry reconstructed from ``UnifiedAction``. + + Contract: + - all SE(3) trajectories live in one shared ``scene_world`` frame + - fingertip positions are world-space if present + - gripper signals are scalar open/close values sampled at ``T+1`` frames + """ + + mask: Action57DMask = field(default_factory=Action57DMask) + # Absolute SE(3) trajectories — (T+1, 4, 4) + ego_poses: np.ndarray | None = None + right_poses: np.ndarray | None = None + left_poses: np.ndarray | None = None + # World-space fingertip positions — (T+1, 5, 3) + right_fingers: np.ndarray | None = None + left_fingers: np.ndarray | None = None + # Scalar gripper — (T+1,) + gripper_right: np.ndarray | None = None + gripper_left: np.ndarray | None = None + # Metadata + video: np.ndarray | None = None # (T+1, H, W, 3) uint8 + action_raw: np.ndarray | None = None # canonical 57D action tensor for display + T: int = 0 + # FK mesh animation: raw (T, nq) joint configs populated by datasets that + # perform EE conversion internally (e.g. robomind-ur). When set, the renderer + # uses these for FK mesh animation instead of running IK on right_poses. + joint_configs: np.ndarray | None = None + # AgiBot source-state FK mesh animation. When set, the renderer uses these + # native URDF link poses directly instead of re-solving IK from pose targets. + agibot_link_poses: dict[str, np.ndarray] | None = None + + +# ─── Converters ─────────────────────────────────────────────────────────────── + + +def to_unified_from_57d(action: np.ndarray) -> UnifiedAction: + """57D hand_pose → passthrough, all 5 slots valid.""" + return UnifiedAction( + action=action.astype(np.float32), + mask=Action57DMask( + ego=True, + right_wrist=True, + right_fingers=ALL_FINGERS, + left_wrist=True, + left_fingers=ALL_FINGERS, + ), + ) + + +def to_unified_from_10d(action: np.ndarray) -> UnifiedAction: + """10D single arm ``[pos(3)+rot6d(6)+grip(1)]`` → right wrist + gripper.""" + T = action.shape[0] + a = np.zeros((T, 57), dtype=np.float32) # [T,57] + a[:, 9:18] = action[:, :9] + return UnifiedAction( + action=a, + mask=Action57DMask(right_wrist=True), + gripper_right=action[:, 9].astype(np.float32), + ) + + +def to_unified_from_20d(action: np.ndarray) -> UnifiedAction: + """20D dual arm ``[left(10) | right(10)]`` → both wrists + both grippers. + + Data layout: ``[L_pos(3) + L_rot6d(6) + L_grip(1) | R_pos(3) + R_rot6d(6) + R_grip(1)]``. + Maps left arm → left wrist slot [33:42], right arm → right wrist slot [9:18]. + """ + T = action.shape[0] + a = np.zeros((T, 57), dtype=np.float32) # [T,57] + a[:, 33:42] = action[:, :9] # left arm → left wrist slot [33:42] + a[:, 9:18] = action[:, 10:19] # right arm → right wrist slot [9:18] + return UnifiedAction( + action=a, + mask=Action57DMask(right_wrist=True, left_wrist=True), + gripper_right=action[:, 19].astype(np.float32), # right arm gripper + gripper_left=action[:, 9].astype(np.float32), # left arm gripper + ) + + +def to_unified_from_9d(action: np.ndarray) -> UnifiedAction: + """9D camera/AV ``[pos(3)+rot6d(6)]`` → ego only.""" + T = action.shape[0] + a = np.zeros((T, 57), dtype=np.float32) # [T,57] + a[:, 0:9] = action[:, :9] + return UnifiedAction( + action=a, + mask=Action57DMask(ego=True), + ) + + +def _validate_action_shape(action: np.ndarray, action_format: ActionFormat) -> None: + """Raise when a raw action tensor does not match its declared format.""" + if action.ndim != 2: + raise ValueError(f"Expected a rank-2 action array, got shape {action.shape}") + actual_dim = int(action.shape[-1]) + expected_dim = action_format.expected_dim + if actual_dim != expected_dim: + raise ValueError(f"Action format {action_format.value} expects trailing dim {expected_dim}, got {actual_dim}") + + +def to_unified(action: np.ndarray, action_format: ActionFormat) -> UnifiedAction: + """Convert one explicit raw action format into ``UnifiedAction``.""" + _validate_action_shape(action, action_format) + if action_format is ActionFormat.UNIFIED_57D: + return to_unified_from_57d(action) + if action_format is ActionFormat.DUAL_ARM_20D: + return to_unified_from_20d(action) + if action_format is ActionFormat.EGO_9D: + return to_unified_from_9d(action) + if action_format is ActionFormat.SINGLE_ARM_10D: + return to_unified_from_10d(action) + raise ValueError(f"Unsupported action format: {action_format}") + + +def _poses_to_pose9d(poses: np.ndarray) -> np.ndarray: + """Convert ``(T,4,4)`` absolute transforms to ``(T,9)`` pos+rot6d rows.""" + + positions = poses[:, :3, 3].astype(np.float32) + rotations = convert_rotation(poses[:, :3, :3], input_format="matrix", output_format="rot6d") + rotations = np.asarray(rotations, dtype=np.float32) + return np.concatenate([positions, rotations], axis=-1).astype(np.float32, copy=False) + + +def to_unified_from_embodiment_c( + sample: dict[str, Any], + *, + embodiment_type: str, +) -> UnifiedAction: + """Convert one AgiBot sample to the canonical viewer representation.""" + + viewer_batch = build_viewer_batch(sample, embodiment_type) + embodiment_spec = get_embodiment_c_embodiment_spec(embodiment_type) + T = int(viewer_batch.head_camera_poses.shape[0]) + + action = np.zeros((T, 57), dtype=np.float32) + action[:, 0:9] = _poses_to_pose9d(viewer_batch.head_camera_poses) + action[:, 9:18] = _poses_to_pose9d(viewer_batch.right_wrist_poses) + action[:, 33:42] = _poses_to_pose9d(viewer_batch.left_wrist_poses) + + return UnifiedAction( + action=action, + mask=Action57DMask( + ego=True, + right_wrist=True, + left_wrist=True, + ), + gripper_right=viewer_batch.right_gripper if embodiment_spec.kind == "gripper" else None, + gripper_left=viewer_batch.left_gripper if embodiment_spec.kind == "gripper" else None, + ) + + +def to_unified_from_embodiment_c_fk_action(action: np.ndarray, kind: str = "gripper") -> UnifiedAction: + """Embodiment C FK-pose action → unified action. + + Relative input layout (gripper, 29D): + ``[head_delta(9), right_delta(9), right_gripper(1), left_delta(9), left_gripper(1)]`` + + These inputs already store ``rot6d`` SE(3) blocks and are copied directly + into the unified slots. + """ + T = action.shape[0] + a = np.zeros((T, 57), dtype=np.float32) + + if kind == "gripper" and action.shape[1] == 29: + a[:, 0:9] = action[:, 0:9] + a[:, 9:18] = action[:, 9:18] + a[:, 33:42] = action[:, 19:28] + return UnifiedAction( + action=a, + mask=Action57DMask(ego=True, right_wrist=True, left_wrist=True), + gripper_right=action[:, 18].astype(np.float32), + gripper_left=action[:, 28].astype(np.float32), + ) + + if kind == "gripper": + raise ValueError(f"Unsupported AgiBot gripper action dim {action.shape[1]}; expected 29.") + raise ValueError(f"Unsupported AgiBot kind {kind!r}; expected 'gripper'.") + + +# ─── Trajectory Reconstruction ──────────────────────────────────────────────── + + +def _pos_rot6d_to_mat(se3: np.ndarray) -> np.ndarray: + """Convert ``(N, 9)`` pos+rot6d to ``(N, 4, 4)`` SE(3) matrices.""" + N = se3.shape[0] + pos = se3[:, :3] + r6 = se3[:, 3:9] + + col0 = r6[:, :3].copy() + col0_norm = np.linalg.norm(col0, axis=-1, keepdims=True) + 1e-8 + col0 = col0 / col0_norm + + col1 = r6[:, 3:6] - np.sum(r6[:, 3:6] * col0, axis=-1, keepdims=True) * col0 + col1_norm = np.linalg.norm(col1, axis=-1, keepdims=True) + 1e-8 + col1 = col1 / col1_norm + + col2 = np.cross(col0, col1) + + mats = np.tile(np.eye(4, dtype=np.float32), (N, 1, 1)) + mats[:, :3, 0] = col0 + mats[:, :3, 1] = col1 + mats[:, :3, 2] = col2 + mats[:, :3, 3] = pos + return mats + + +def _chain_se3( + deltas: np.ndarray, + initial_pose: np.ndarray | None = None, + pose_convention: str = "backward_framewise", +) -> np.ndarray: + """Chain ``(T, 9)`` relative deltas into ``(T+1, 4, 4)`` absolute poses. + + For ``backward_framewise``: ``P_{t+1} = P_t @ delta_t``. + For ``absolute``: each row is already an absolute pose (no chaining). + """ + T = deltas.shape[0] + delta_mats = _pos_rot6d_to_mat(deltas) + + if initial_pose is None: + initial_pose = np.eye(4, dtype=np.float32) + else: + initial_pose = initial_pose.astype(np.float32) + + poses = np.empty((T + 1, 4, 4), dtype=np.float32) + poses[0] = initial_pose + + if pose_convention == "absolute": + poses[1:] = delta_mats + else: + for t in range(T): + poses[t + 1] = poses[t] @ delta_mats[t] + + return poses + + +def _extract_fingers(raw: np.ndarray) -> np.ndarray: + """``(T, 15)`` → ``(T+1, 5, 3)`` with first frame duplicated.""" + T = raw.shape[0] + fingers = raw.reshape(T, 5, 3).astype(np.float32) # [T,5,3] + return np.concatenate([fingers[:1], fingers], axis=0) + + +def _to_numpy_float32(value: object) -> np.ndarray: + """Convert a tensor-like value to a float32 NumPy array.""" + + if isinstance(value, torch.Tensor): + return value.detach().cpu().numpy().astype(np.float32) + return np.asarray(value, dtype=np.float32) + + +def _quat_xyzw_to_rotmat(q: np.ndarray) -> np.ndarray: + """Convert ``(N, 4)`` xyzw quaternions to ``(N, 3, 3)`` rotation matrices.""" + x, y, z, w = q[:, 0], q[:, 1], q[:, 2], q[:, 3] + R = np.zeros((len(q), 3, 3), dtype=np.float32) + R[:, 0, 0] = 1 - 2 * (y * y + z * z) + R[:, 0, 1] = 2 * (x * y - z * w) + R[:, 0, 2] = 2 * (x * z + y * w) + R[:, 1, 0] = 2 * (x * y + z * w) + R[:, 1, 1] = 1 - 2 * (x * x + z * z) + R[:, 1, 2] = 2 * (y * z - x * w) + R[:, 2, 0] = 2 * (x * z - y * w) + R[:, 2, 1] = 2 * (y * z + x * w) + R[:, 2, 2] = 1 - 2 * (x * x + y * y) + return R + + +def _build_absolute_from_overlay(sample: dict) -> dict[str, np.ndarray] | None: + """Build absolute world-frame poses from HandPoseDataset overlay data. + + Returns None if overlay keys are missing. + """ + raw_cam_pos = sample.get("raw_cam_position") + if raw_cam_pos is None: + return None + + cam_pos = _to_numpy_float32(raw_cam_pos) # [T+1,3] + cam_rot_q = _to_numpy_float32(sample["raw_cam_rotation"]) # [T+1,4] + right_3d = _to_numpy_float32(sample["raw_cam_right_3d"]) # [T+1,63] + left_3d = _to_numpy_float32(sample["raw_cam_left_3d"]) # [T+1,63] + right_rot = _to_numpy_float32(sample["raw_cam_right_rot"]) # [T+1,84] + left_rot = _to_numpy_float32(sample["raw_cam_left_rot"]) # [T+1,84] + + T1 = cam_pos.shape[0] + FTIP = [4, 8, 12, 16, 20] + + # Camera c2w (world frame) + cam_c2w = np.tile(np.eye(4, dtype=np.float32), (T1, 1, 1)) # [T+1,4,4] + cam_c2w[:, :3, 3] = cam_pos + cam_c2w[:, :3, :3] = _quat_xyzw_to_rotmat(cam_rot_q) + + def _wrist_world(pos_63, rot_84): + wrist_pos = pos_63[:, :3] + wrist_q = rot_84.reshape(T1, 21, 4)[:, 0] + wrist_cam = np.tile(np.eye(4, dtype=np.float32), (T1, 1, 1)) # [T+1,4,4] + wrist_cam[:, :3, 3] = wrist_pos + wrist_cam[:, :3, :3] = _quat_xyzw_to_rotmat(wrist_q) + return cam_c2w @ wrist_cam + + def _fingers_world(pos_63): + joints = pos_63.reshape(T1, 21, 3)[:, FTIP] + R = cam_c2w[:, :3, :3] + t = cam_c2w[:, :3, 3] + return np.einsum("tij,tfj->tfi", R, joints) + t[:, None, :] # [T+1,5,3] + + return { + "ego_poses": cam_c2w, + "right_wrist_poses": _wrist_world(right_3d, right_rot), + "left_wrist_poses": _wrist_world(left_3d, left_rot), + "right_fingers": _fingers_world(right_3d), + "left_fingers": _fingers_world(left_3d), + } + + +# ─── Scene State Builder ───────────────────────────────────────────────────── + + +def build_scene_state( + unified: UnifiedAction, + initial_pose: np.ndarray | None = None, + initial_pose_right: np.ndarray | None = None, + initial_pose_left: np.ndarray | None = None, + right_base_pose: np.ndarray | None = None, + left_base_pose: np.ndarray | None = None, + pose_convention: str = "backward_framewise", + sample: dict | None = None, +) -> SceneState: + """Reconstruct a canonical world-space ``SceneState`` from ``UnifiedAction``. + + Chains SE(3) deltas for valid mask slots. If ``sample`` contains overlay + data (HandPoseDataset raw camera/joint fields), overrides with absolute + world-frame poses. + + Args: + unified: Canonical 57D action with mask. + initial_pose: Default initial pose for all slots. + initial_pose_right: Override for right wrist (dual arm). + initial_pose_left: Override for left wrist (dual arm). + right_base_pose: Right-arm base pose that maps arm-local trajectories into ``scene_world``. + left_base_pose: Left-arm base pose that maps arm-local trajectories into ``scene_world``. + pose_convention: Pose convention for SE(3) chaining. + sample: Raw dataset sample (for overlay data). + """ + + def _apply_pose_base(poses: np.ndarray | None, base_pose: np.ndarray | None) -> np.ndarray | None: + if poses is None or base_pose is None: + return poses + return np.einsum("ij,njk->nik", base_pose, poses).astype(np.float32) # [T+1,4,4] + + def _fingers_local_to_world( + fingers_local: np.ndarray | None, + wrist_poses_world: np.ndarray | None, + ) -> np.ndarray | None: + if fingers_local is None: + return None + if wrist_poses_world is None: + raise ValueError("Finger trajectories require matching wrist poses to build world-space SceneState") + wrist_rot = wrist_poses_world[:, :3, :3].astype(np.float32) # [T+1,3,3] + wrist_pos = wrist_poses_world[:, :3, 3].astype(np.float32) # [T+1,3] + return np.einsum("tij,tfj->tfi", wrist_rot, fingers_local) + wrist_pos[:, None, :] # [T+1,5,3] + + mask = unified.mask + action = unified.action + state = SceneState(mask=mask) + + ip_default = initial_pose if initial_pose is not None else np.eye(4, dtype=np.float32) + ip_right = initial_pose_right if initial_pose_right is not None else ip_default + ip_left = initial_pose_left if initial_pose_left is not None else ip_default + + if mask.ego: + state.ego_poses = _chain_se3(action[:, 0:9], ip_default, pose_convention) + if mask.right_wrist: + state.right_poses = _chain_se3(action[:, 9:18], ip_right, pose_convention) + if any(mask.right_fingers): + state.right_fingers = _extract_fingers(action[:, 18:33]) + if mask.left_wrist: + state.left_poses = _chain_se3(action[:, 33:42], ip_left, pose_convention) + if any(mask.left_fingers): + state.left_fingers = _extract_fingers(action[:, 42:57]) + + if unified.gripper_right is not None: + g = unified.gripper_right + state.gripper_right = np.concatenate([[g[0]], g]).astype(np.float32, copy=False) # [T+1] + if unified.gripper_left is not None: + g = unified.gripper_left + state.gripper_left = np.concatenate([[g[0]], g]).astype(np.float32, copy=False) # [T+1] + + abs_data = _build_absolute_from_overlay(sample) if sample is not None else None + if abs_data is not None: + state.ego_poses = abs_data["ego_poses"] + state.right_poses = abs_data["right_wrist_poses"] + state.left_poses = abs_data["left_wrist_poses"] + state.right_fingers = abs_data["right_fingers"] + state.left_fingers = abs_data["left_fingers"] + log.info( + f"Overlay absolute mode | ego range: " + f"[{abs_data['ego_poses'][:, :3, 3].min():.3f}, " + f"{abs_data['ego_poses'][:, :3, 3].max():.3f}] | " + f"R wrist[0]: {abs_data['right_wrist_poses'][0, :3, 3]}" + ) + else: + state.right_poses = _apply_pose_base(state.right_poses, right_base_pose) + state.left_poses = _apply_pose_base(state.left_poses, left_base_pose) + state.right_fingers = _fingers_local_to_world(state.right_fingers, state.right_poses) + state.left_fingers = _fingers_local_to_world(state.left_fingers, state.left_poses) + + raw_agibot_link_poses = sample.get("agibot_link_poses") if sample is not None else None + if isinstance(raw_agibot_link_poses, dict): + state.agibot_link_poses = { + str(link_name): _to_numpy_float32(link_poses) for link_name, link_poses in raw_agibot_link_poses.items() + } # {name:[T+1,4,4]} + + state.action_raw = unified.action.astype(np.float32) + state.T = action.shape[0] + return state + + +# ─── Video Extraction ───────────────────────────────────────────────────────── + + +def get_video_from_sample(sample: dict) -> np.ndarray | None: + """Extract video frames from a dataset sample. + + Returns ``(T+1, H, W, 3)`` uint8 array, or None. + """ + video = sample.get("video") + if video is None: + return None + if isinstance(video, torch.Tensor): + video = video.numpy() + + if video.ndim == 4: + C, T_dim, H, W = video.shape + if C in (1, 3) and T_dim > 3: + video = np.transpose(video, (1, 2, 3, 0)) + elif video.shape[1] in (1, 3) and T_dim <= 3: + video = np.transpose(video, (0, 2, 3, 1)) + + if video.dtype in (np.float32, np.float64): + video = np.clip(video * 255, 0, 255).astype(np.uint8) + + if video.ndim == 4 and video.shape[-1] == 1: + video = np.repeat(video, 3, axis=-1) + + return video diff --git a/cosmos_framework/data/vfm/action/urdf_visualizer/unified_renderer.py b/cosmos_framework/data/vfm/action/urdf_visualizer/unified_renderer.py new file mode 100644 index 0000000..2662949 --- /dev/null +++ b/cosmos_framework/data/vfm/action/urdf_visualizer/unified_renderer.py @@ -0,0 +1,1019 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Mask-driven renderer for ``SceneState`` — the unified 57D viewer backend. + +Owns all viser scene handles. Draws only what the mask declares valid. +No action-format branching — everything goes through ``SceneState``. +""" + +from __future__ import annotations + +from typing import Any + +import numpy as np + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.embodiment_c_fk import DEFAULT_GRIPPER_LENGTH_M +from cosmos_framework.data.vfm.action.embodiment_c_spec import get_embodiment_c_kind +from cosmos_framework.data.vfm.action.urdf_visualizer.ik_solver import ( + compute_agibot_link_poses_batch_from_configs, + solve_agibot_trajectory_ik, +) +from cosmos_framework.data.vfm.action.urdf_visualizer.robot_scene_model import RobotSceneModel +from cosmos_framework.data.vfm.action.urdf_visualizer.unified_action import FINGER_NAMES, SceneState +from cosmos_framework.data.vfm.action.urdf_visualizer.urdf_loader import ( + get_embodiment_c_collision_geometries, +) + + +def _agibot_gripper_tip_positions( + pos: np.ndarray, + rot: np.ndarray, + opening: float, + max_finger_width: float, +) -> tuple[np.ndarray, np.ndarray]: + """Approximate AgiBot gripper tip sphere positions from an OpenCV wrist pose.""" + + half_w = float(opening) * max_finger_width / 2.0 + gripper_center = pos + DEFAULT_GRIPPER_LENGTH_M * rot[:, 2] # [3] + tip_l = gripper_center + rot @ np.array([half_w, 0.0, 0.0], dtype=np.float32) # [3] + tip_r = gripper_center + rot @ np.array([-half_w, 0.0, 0.0], dtype=np.float32) # [3] + return tip_l.astype(np.float32, copy=False), tip_r.astype(np.float32, copy=False) + + +def _get_entry_agibot_embodiment_type(entry: Any) -> str: + """Return the AgiBot robot embodiment declared by a viewer dataset entry.""" + + robot_embodiment_type = getattr(entry, "robot_embodiment_type", None) + if isinstance(robot_embodiment_type, str) and robot_embodiment_type: + return robot_embodiment_type + + dataset_kwargs = getattr(entry, "dataset_kwargs", {}) + if isinstance(dataset_kwargs, dict): + embodiment_type = dataset_kwargs.get("embodiment_type", "") + if isinstance(embodiment_type, str): + return embodiment_type + + return "" + + +def _get_wrist_to_opencv(to_opencv: object, wrist_name: str) -> np.ndarray | None: + """Resolve a native-to-OpenCV wrist rotation from renderer dataset metadata.""" + + if isinstance(to_opencv, dict): + rotation = to_opencv.get(wrist_name) + if isinstance(rotation, np.ndarray): + return rotation.astype(np.float32, copy=False) # [3,3] + return None + if isinstance(to_opencv, np.ndarray): + return to_opencv.astype(np.float32, copy=False) # [3,3] + return None + + +def _undo_wrist_to_opencv_for_ik( + poses: np.ndarray | None, + to_opencv: object, + wrist_name: str, +) -> np.ndarray | None: + """Convert viewer/OpenCV wrist rotations back to native frame for IK.""" + + if poses is None: + return None + wrist_to_opencv = _get_wrist_to_opencv(to_opencv, wrist_name) + if wrist_to_opencv is None or np.allclose(wrist_to_opencv, np.eye(3, dtype=np.float32)): + return poses + native_poses = poses.astype(np.float32, copy=True) # [T,4,4] + native_poses[:, :3, :3] = native_poses[:, :3, :3] @ wrist_to_opencv.T[None] # [T,3,3] + return native_poses + + +class UnifiedRenderer: + """Mask-driven 3D renderer for the unified 57D action viewer.""" + + # ── Palette ── + TIP_RADIUS = 0.0042 + FRUSTUM_SCALE = 0.10 + EGO_AXIS_LENGTH = 0.04 + EGO_AXIS_RADIUS = 0.002 + EGO_TRAJ_LENGTH_REFERENCE = 0.55 + EGO_TRAJ_SCALE_MIN = 0.35 + EGO_TRAJ_SCALE_MAX = 1.80 + EGO_TRAJ_LINE_WIDTH = 1.0 + EE_TRAJ_LINE_WIDTH = 1.5 + HAND_AXIS_LENGTH = 0.05 + HAND_AXIS_RADIUS = 0.002 + ROBOT_BODY_AXIS_LENGTH = 0.03 + ROBOT_BODY_AXIS_RADIUS = 0.0012 + ROBOT_SITE_AXIS_LENGTH = 0.022 + ROBOT_SITE_AXIS_RADIUS = 0.0009 + COLOR_EGO = (52, 152, 219) # blue + COLOR_EGO_TOP = (231, 76, 60) # red + COLOR_RIGHT = (243, 156, 18) # orange + COLOR_LEFT = (155, 89, 182) # purple + FINGER_COLORS = [ + (231, 76, 60), + (241, 196, 15), + (46, 204, 113), + (52, 152, 219), + (155, 89, 182), + ] + + @staticmethod + def _soften_color(color: tuple[int, int, int], mix: float = 0.35) -> tuple[int, int, int]: + """Blend a color toward white for less visually dominant trajectories.""" + base = np.asarray(color, dtype=np.float32) + soft = base * (1.0 - mix) + 255.0 * mix + rounded = soft.round() + return int(rounded[0]), int(rounded[1]), int(rounded[2]) + + def _p(self, path: str) -> str: + """Prepend the instance name_prefix to a scene path.""" + return f"{self._name_prefix}{path}" + + def __init__( + self, + server, + name_prefix: str = "", + palette: dict | None = None, + ): + import viser.transforms as vtf + + self.server = server + self.vtf = vtf + self.state: SceneState | None = None + self._entry: Any = None + self._axis_scale = 1.0 + self._name_prefix = name_prefix + + self._ego_frustum_scale_base = self.FRUSTUM_SCALE + self._ego_axis_length_base = self.EGO_AXIS_LENGTH + self._ego_axis_radius_base = self.EGO_AXIS_RADIUS + self._ego_frustum_fov = np.deg2rad(60.0) + self._ego_frustum_aspect = 4 / 3 + + # Per-instance color palette (allows GT and pred to use different colors) + pal = palette or {} + self.COLOR_EGO = pal.get("ego", type(self).COLOR_EGO) + self.COLOR_EGO_TOP = pal.get("ego_top", type(self).COLOR_EGO_TOP) + self.COLOR_RIGHT = pal.get("right", type(self).COLOR_RIGHT) + self.COLOR_LEFT = pal.get("left", type(self).COLOR_LEFT) + + # ── Ego ── + self.ego_frame = server.scene.add_frame( + self._p("/ego/frame"), axes_length=self.EGO_AXIS_LENGTH, axes_radius=self.EGO_AXIS_RADIUS + ) + self.ego_frustum = server.scene.add_line_segments( + self._p("/ego/frustum"), + points=self._make_ego_frustum_wireframe_points(self._ego_frustum_fov, self._ego_frustum_aspect), + colors=np.array(self.COLOR_EGO, dtype=np.uint8), + scale=self.FRUSTUM_SCALE, + line_width=2.0, + wxyz=(1.0, 0.0, 0.0, 0.0), + position=(0.0, 0.0, 0.0), + ) + self.ego_frustum_up = server.scene.add_line_segments( + self._p("/ego/frustum_up"), + points=self._make_ego_frustum_up_points(self._ego_frustum_fov, self._ego_frustum_aspect), + colors=np.array(self.COLOR_EGO_TOP, dtype=np.uint8), + line_width=3.0, + scale=self.FRUSTUM_SCALE, + wxyz=(1.0, 0.0, 0.0, 0.0), + position=(0.0, 0.0, 0.0), + ) + self.ego_traj = server.scene.add_spline_catmull_rom( + self._p("/ego/traj"), + positions=np.zeros([2, 3], dtype=np.float32), + color=self._soften_color(self.COLOR_EGO), + line_width=self.EGO_TRAJ_LINE_WIDTH, + ) + + # ── Right effector ── + self.right_frame = server.scene.add_frame( + self._p("/right/frame"), axes_length=self.HAND_AXIS_LENGTH, axes_radius=self.HAND_AXIS_RADIUS + ) + self.right_traj = server.scene.add_spline_catmull_rom( + self._p("/right/traj"), + positions=np.zeros([2, 3], dtype=np.float32), + color=self._soften_color(self.COLOR_RIGHT), + line_width=self.EE_TRAJ_LINE_WIDTH, + ) + self.right_ee = server.scene.add_point_cloud( + self._p("/right/point"), + points=np.zeros([1, 3], dtype=np.float32), + colors=np.array([self.COLOR_RIGHT], dtype=np.uint8), + point_size=0.015, + point_shape="circle", + ) + self.right_fingers = [ + server.scene.add_icosphere( + self._p(f"/right/finger_{FINGER_NAMES[i]}"), + radius=self.TIP_RADIUS, + color=self.FINGER_COLORS[i], + position=(0.0, 0.0, 0.0), + ) + for i in range(5) + ] + self.right_gripper_tips = [ + server.scene.add_icosphere( + self._p(f"/right/gripper_tip_{side}"), + radius=self.TIP_RADIUS, + color=self.FINGER_COLORS[i], + position=(0.0, 0.0, 0.0), + ) + for i, side in enumerate(("thumb", "index")) + ] + + # ── Left effector ── + self.left_frame = server.scene.add_frame( + self._p("/left/frame"), axes_length=self.HAND_AXIS_LENGTH, axes_radius=self.HAND_AXIS_RADIUS + ) + self.left_traj = server.scene.add_spline_catmull_rom( + self._p("/left/traj"), + positions=np.zeros([2, 3], dtype=np.float32), + color=self._soften_color(self.COLOR_LEFT), + line_width=self.EE_TRAJ_LINE_WIDTH, + ) + self.left_ee = server.scene.add_point_cloud( + self._p("/left/point"), + points=np.zeros([1, 3], dtype=np.float32), + colors=np.array([self.COLOR_LEFT], dtype=np.uint8), + point_size=0.015, + point_shape="circle", + ) + self.left_fingers = [ + server.scene.add_icosphere( + self._p(f"/left/finger_{FINGER_NAMES[i]}"), + radius=self.TIP_RADIUS, + color=self.FINGER_COLORS[i], + position=(0.0, 0.0, 0.0), + ) + for i in range(5) + ] + self.left_gripper_tips = [ + server.scene.add_icosphere( + self._p(f"/left/gripper_tip_{side}"), + radius=self.TIP_RADIUS, + color=self.FINGER_COLORS[i], + position=(0.0, 0.0, 0.0), + ) + for i, side in enumerate(("thumb", "index")) + ] + + # ── IK robot meshes ── + self.robot_right: list = [] + self.robot_left: list = [] + self._robot_frame_handles_right: dict[str, Any] = {} + self._robot_frame_handles_left: dict[str, Any] = {} + self._current_robot: Any | None = None + self._robot_scene_model: RobotSceneModel | None = None + self._ik_right: list | None = None + self._ik_left: list | None = None + self._robot_frames_right: dict[str, list[tuple[np.ndarray, np.ndarray]]] | None = None + self._robot_frames_left: dict[str, list[tuple[np.ndarray, np.ndarray]]] | None = None + self._robot_link_names: list[str] = [] + self._robot_local_transforms: list[np.ndarray] = [] + + # ── Video panel (set by viewer.py) ── + self._cam_handle: Any | None = None + + self.hide_all() + + # ─── Per-Episode ────────────────────────────────────────────────────────── + + def load( + self, + state: SceneState, + entry: Any, + to_opencv: np.ndarray | dict[str, np.ndarray] | None = None, + ): + """Load a new episode. Rebuild trajectories, robot meshes, and IK. + + Args: + state: Reconstructed ``SceneState`` with absolute poses. + entry: ``DatasetEntry`` with robot_name, max_finger_width, etc. + to_opencv: Optional native-to-OpenCV rotation. AgiBot can provide + side-specific ``{"left_wrist": R, "right_wrist": R}`` values. + """ + self.state = state + self._entry = entry + self._to_opencv = to_opencv if to_opencv is not None else np.eye(3, dtype=np.float32) + self._ego_frustum_fov = np.deg2rad(entry.camera_fov_deg) + self._ego_frustum_aspect = float(entry.camera_aspect) + self.ego_frustum.points = self._make_ego_frustum_wireframe_points( + self._ego_frustum_fov, + self._ego_frustum_aspect, + ) + self.ego_frustum_up.points = self._make_ego_frustum_up_points( + self._ego_frustum_fov, + self._ego_frustum_aspect, + ) + self._update_ego_visual_scale(state.ego_poses if state.mask.ego else None) + self.update_axis_scale(self._axis_scale) + self.hide_all() + + # ── Robot meshes + IK from canonical world-space SceneState ── + self._ik_right = None + self._ik_left = None + self._robot_frames_right = None + self._robot_frames_left = None + if entry.robot_name: + self._load_robot_and_ik(state, entry) + + # Rebuild trajectory splines from canonical world-space SceneState. + if state.mask.ego and state.ego_poses is not None: + self._rebuild_traj(self.ego_traj, state.ego_poses, self.COLOR_EGO) + if state.mask.right_wrist and state.right_poses is not None: + self._rebuild_traj(self.right_traj, state.right_poses, self.COLOR_RIGHT) + if state.mask.left_wrist and state.left_poses is not None: + self._rebuild_traj(self.left_traj, state.left_poses, self.COLOR_LEFT) + + def set_video_panel(self, panel_handle: Any | None) -> None: + """Attach the optional GUI image panel used for episode video.""" + self._cam_handle = panel_handle + + # ─── Per-Frame ──────────────────────────────────────────────────────────── + + def update(self, t: int, show: dict): + """Update all scene elements for time step ``t``. + + Args: + t: Frame index (0-based). + show: Visibility flags: ``frames``, ``traj``, ``fingertips``, ``ego``, ``robot``. + """ + state = self.state + if state is None: + return + mask = state.mask + + # ── Ego ── + self._update_ego(t, state.ego_poses, mask.ego and show.get("ego", False), show) + + # ── Right effector ── + self._update_effector( + t, + state.right_poses, + mask.right_wrist, + self.right_frame, + self.right_ee, + self.right_traj, + show, + ) + self._update_fingers( + t, + state.right_fingers, + mask.right_fingers, + self.right_fingers, + show, + ) + self._update_gripper( + t, + state.right_poses, + state.gripper_right, + mask.right_wrist, + mask.right_fingers, + self.right_gripper_tips, + show, + ) + + # ── Left effector ── + self._update_effector( + t, + state.left_poses, + mask.left_wrist, + self.left_frame, + self.left_ee, + self.left_traj, + show, + ) + self._update_fingers( + t, + state.left_fingers, + mask.left_fingers, + self.left_fingers, + show, + ) + self._update_gripper( + t, + state.left_poses, + state.gripper_left, + mask.left_wrist, + mask.left_fingers, + self.left_gripper_tips, + show, + ) + + # ── IK robot meshes ── + self._update_robot(t, show) + + # ── Video panel ── + if self._cam_handle is not None and state.video is not None and t < len(state.video): + self._cam_handle.image = state.video[t] + + # ─── Action Text ────────────────────────────────────────────────────────── + + def format_action_text(self, t: int) -> str: + """Return a formatted string showing 57D action values at step ``t``. + + Always shows the full 57D layout. Validity indicator (✓/·) in front of + each component based on the mask. + """ + state = self.state + if state is None or state.action_raw is None: + return "" + if t == 0: + return "*t=0: anchor pose (identity)*" + if (t - 1) >= len(state.action_raw): + return "" + + a = state.action_raw[t - 1] # always 57D (zero-padded) + mask = state.mask + + def _fmt(v): + return " ".join(f"{x:+.4f}" for x in v) + + def _v(active): + return "✓" if active else "·" + + gr = a[18:33].reshape(5, 3) + gl = a[42:57].reshape(5, 3) + + # Gripper auxiliary values (not in 57D vector) + grip_r_str = "" + grip_l_str = "" + if state.gripper_right is not None and t < len(state.gripper_right): + grip_r_str = f" ✓ gripper {state.gripper_right[t]:+.4f}" + if state.gripper_left is not None and t < len(state.gripper_left): + grip_l_str = f" ✓ gripper {state.gripper_left[t]:+.4f}" + + parts = [ + f"step {t - 1} → {t} (57D)", + "═" * 36, + f"{_v(mask.ego)} Ego pos [0:3] {_fmt(a[0:3])}", + f" {' ' * 1} rot [3:9] {_fmt(a[3:9])}", + "", + f"{_v(mask.right_wrist)} R wrist pos [9:12] {_fmt(a[9:12])}", + f" {' ' * 1} rot [12:18] {_fmt(a[12:18])}", + f" R fingers [18:33]", + ] + for i, name in enumerate(FINGER_NAMES): + parts.append(f" {_v(mask.right_fingers[i])} {name:7s} {_fmt(gr[i])}") + if grip_r_str: + parts.append(grip_r_str) + + parts += [ + "", + f"{_v(mask.left_wrist)} L wrist pos [33:36] {_fmt(a[33:36])}", + f" {' ' * 1} rot [36:42] {_fmt(a[36:42])}", + f" L fingers [42:57]", + ] + for i, name in enumerate(FINGER_NAMES): + parts.append(f" {_v(mask.left_fingers[i])} {name:7s} {_fmt(gl[i])}") + if grip_l_str: + parts.append(grip_l_str) + + return str("```\n" + "\n".join(parts) + "\n```") + + # ─── Private: Effector ──────────────────────────────────────────────────── + + @staticmethod + def _make_ego_frustum_wireframe_points(fov: float, aspect: float) -> np.ndarray: + """Build wireframe segments for the ego camera frustum.""" + half_height = float(np.tan(fov / 2.0)) + half_width = float(aspect) * half_height + top_left = np.array([-half_width, -half_height, 1.0], dtype=np.float32) + top_right = np.array([half_width, -half_height, 1.0], dtype=np.float32) + bottom_right = np.array([half_width, half_height, 1.0], dtype=np.float32) + bottom_left = np.array([-half_width, half_height, 1.0], dtype=np.float32) + origin = np.array([0.0, 0.0, 0.0], dtype=np.float32) + return np.array( + [ + [origin, top_left], + [origin, top_right], + [origin, bottom_right], + [origin, bottom_left], + [top_left, top_right], + [top_right, bottom_right], + [bottom_right, bottom_left], + [bottom_left, top_left], + ], + dtype=np.float32, + ) + + @staticmethod + def _make_ego_frustum_up_points(fov: float, aspect: float) -> np.ndarray: + """Build a red segment that marks the frustum's far-edge upright tick.""" + half_height = float(np.tan(fov / 2.0)) + _ = aspect + top_y = -half_height + return np.array( + [[[0.0, top_y, 1.0], [0.0, top_y * 1.18, 1.0]]], + dtype=np.float32, + ) + + def _update_ego(self, t: int, poses: np.ndarray | None, active: bool, show: dict) -> None: + if active and poses is not None and t < len(poses): + pos = poses[t, :3, 3] + rot = poses[t, :3, :3] + wxyz = self.vtf.SO3.from_matrix(rot).wxyz + self.ego_frame.position = pos + self.ego_frame.wxyz = wxyz + self.ego_frame.visible = show.get("frames", True) + self.ego_frustum.position = pos + self.ego_frustum.wxyz = wxyz + self.ego_frustum.visible = True + self.ego_frustum_up.position = pos + self.ego_frustum_up.wxyz = wxyz + self.ego_frustum_up.visible = True + self.ego_traj.visible = show.get("traj", True) + else: + self.ego_frame.visible = False + self.ego_frustum.visible = False + self.ego_frustum_up.visible = False + self.ego_traj.visible = False + + def _update_effector(self, t, poses, active, frame, ee, traj, show): + if active and poses is not None and t < len(poses): + pos = poses[t, :3, 3] + rot = poses[t, :3, :3] + frame.position = pos + frame.wxyz = self.vtf.SO3.from_matrix(rot).wxyz + frame.visible = show.get("frames", True) + ee.points = pos[None] + ee.visible = True + traj.visible = show.get("traj", True) + else: + frame.visible = False + ee.visible = False + traj.visible = False + + # ─── Private: Fingers ───────────────────────────────────────────────────── + + def _update_fingers(self, t, fingers, finger_mask, handles, show): + if fingers is None or t >= len(fingers): + for h in handles: + h.visible = False + return + if not show.get("fingertips", True): + for h in handles: + h.visible = False + return + + g = fingers[t] # (5, 3) + for fi, h in enumerate(handles): + if finger_mask[fi]: + h.position = g[fi].astype(np.float32) + h.visible = True + else: + h.visible = False + + # ─── Private: Gripper ───────────────────────────────────────────────────── + + def _update_gripper(self, t, poses, gripper, wrist_active, finger_mask, handle, show): + has_fingers = any(finger_mask) + if not wrist_active or has_fingers or gripper is None or poses is None or t >= len(gripper) or t >= len(poses): + for tip_handle in handle: + tip_handle.visible = False + return + if not show.get("fingertips", True): + for tip_handle in handle: + tip_handle.visible = False + return + pos = poses[t, :3, 3].astype(np.float32) + rot = poses[t, :3, :3].astype(np.float32) + g = float(gripper[t]) + mfw = getattr(self._entry, "max_finger_width", 0.05) + embodiment_type = _get_entry_agibot_embodiment_type(self._entry) + if ( + getattr(self._entry, "robot_name", "") == "embodiment_c" + and embodiment_type + and get_embodiment_c_kind(embodiment_type) == "gripper" + ): + tip_l, tip_r = _agibot_gripper_tip_positions(pos, rot, g, float(mfw)) + else: + half_w = g * mfw / 2.0 + finger_len = 0.06 + tip_l = pos + rot @ np.array([half_w, 0, finger_len], dtype=np.float32) + tip_r = pos + rot @ np.array([-half_w, 0, finger_len], dtype=np.float32) + for tip_handle, tip in zip(handle, (tip_l, tip_r), strict=True): + tip_handle.position = tip + tip_handle.visible = True + + # ─── Private: Trajectory ────────────────────────────────────────────────── + + def _rebuild_traj(self, traj_handle, poses, color): + """Rebuild a trajectory spline from absolute poses.""" + positions = poses[:, :3, 3].astype(np.float32) + if len(positions) < 2: + traj_handle.visible = False + return + line_width = self.EGO_TRAJ_LINE_WIDTH if traj_handle is self.ego_traj else self.EE_TRAJ_LINE_WIDTH + # Remove and recreate to avoid stale color issues in viser + name = traj_handle.name if hasattr(traj_handle, "name") else "/tmp/traj" + traj_handle.remove() + new_handle = self.server.scene.add_spline_catmull_rom( + name, + positions=positions, + color=self._soften_color(color), + line_width=line_width, + ) + # Update the reference — need to figure out which attribute to update + if traj_handle is self.ego_traj: + self.ego_traj = new_handle + elif traj_handle is self.right_traj: + self.right_traj = new_handle + elif traj_handle is self.left_traj: + self.left_traj = new_handle + + @staticmethod + def _trajectory_length(poses: np.ndarray | None) -> float: + """Compute the total path length of a pose trajectory.""" + if poses is None or len(poses) < 2: + return 0.0 + positions = poses[:, :3, 3].astype(np.float32) + deltas = np.diff(positions, axis=0) + return float(np.linalg.norm(deltas, axis=1).sum()) + + def _update_ego_visual_scale(self, poses: np.ndarray | None) -> None: + """Scale the ego camera frame/frustum from the episode trajectory length.""" + traj_length = self._trajectory_length(poses) + if traj_length <= 0.0: + traj_scale = 1.0 + else: + traj_scale = float( + np.clip( + traj_length / self.EGO_TRAJ_LENGTH_REFERENCE, + self.EGO_TRAJ_SCALE_MIN, + self.EGO_TRAJ_SCALE_MAX, + ) + ) + self._ego_frustum_scale_base = self.FRUSTUM_SCALE * traj_scale + self._ego_axis_length_base = self.EGO_AXIS_LENGTH * traj_scale + self._ego_axis_radius_base = self.EGO_AXIS_RADIUS * traj_scale + + # ─── Private: IK Robot ──────────────────────────────────────────────────── + + @classmethod + def _robot_frame_dims(cls, frame_key: str) -> tuple[float, float]: + """Return axis length/radius for one robot debug frame.""" + if frame_key.startswith("site:"): + return cls.ROBOT_SITE_AXIS_LENGTH, cls.ROBOT_SITE_AXIS_RADIUS + return cls.ROBOT_BODY_AXIS_LENGTH, cls.ROBOT_BODY_AXIS_RADIUS + + @staticmethod + def _clear_robot_frame_handles(handles: dict[str, Any]) -> None: + """Remove all robot debug frame handles in one arm.""" + for handle in handles.values(): + handle.remove() + handles.clear() + + def _robot_frame_selector_key(self, arm: str, frame_key: str) -> str: + """Return the GUI selector key for one robot debug frame.""" + if self._robot_frames_left is None and arm == "right": + return frame_key + return f"{arm}/{frame_key}" + + def get_robot_frame_selectors(self) -> list[tuple[str, str]]: + """Return selector keys and checkbox labels for available robot frames.""" + selectors = [] + for arm, frames in [("right", self._robot_frames_right), ("left", self._robot_frames_left)]: + if frames is None: + continue + for frame_key in sorted(frames): + selector_key = self._robot_frame_selector_key(arm, frame_key) + if self._robot_frames_left is None and arm == "right": + label = frame_key + else: + label = selector_key + selectors.append((selector_key, label)) + return selectors + + def _rebuild_robot_frame_handles( + self, + arm: str, + frames: dict[str, list[tuple[np.ndarray, np.ndarray]]] | None, + ) -> None: + """Recreate robot debug frame handles for one arm.""" + handles = self._robot_frame_handles_right if arm == "right" else self._robot_frame_handles_left + self._clear_robot_frame_handles(handles) + if frames is None: + return + for frame_key in sorted(frames): + kind, name = frame_key.split(":", 1) + axes_length, axes_radius = self._robot_frame_dims(frame_key) + handles[frame_key] = self.server.scene.add_frame( + self._p(f"/robot_{arm}_frames/{kind}/{name}"), + axes_length=axes_length * self._axis_scale, + axes_radius=axes_radius * self._axis_scale, + ) + log.info(f"Loaded {len(handles)} robot debug frames for {arm} arm") + + def _update_robot_debug_frames(self, t: int, show: dict) -> None: + """Update body/site coordinate frame overlays for both arms.""" + filters = show.get("robot_frame_filters", {}) + for arm, handles, frames in [ + ("right", self._robot_frame_handles_right, self._robot_frames_right), + ("left", self._robot_frame_handles_left, self._robot_frames_left), + ]: + for frame_key, handle in handles.items(): + poses = frames.get(frame_key) if frames is not None else None + selector_key = self._robot_frame_selector_key(arm, frame_key) + frame_enabled = bool(filters.get(selector_key, False)) + if poses is not None and t < len(poses) and frame_enabled: + pos, rot = poses[t] + handle.position = pos + handle.wxyz = self.vtf.SO3.from_matrix(rot).wxyz + handle.visible = True + else: + handle.visible = False + + def _set_agibot_mesh_animation_from_link_poses(self, link_poses: dict[str, np.ndarray], num_steps: int) -> None: + """Populate AgiBot mesh transforms from absolute URDF link poses.""" + + animated_transforms: list[list[tuple[np.ndarray, np.ndarray]]] = [] + for step_idx in range(num_steps): + frame_transforms: list[tuple[np.ndarray, np.ndarray]] = [] + for link_name, local_transform in zip( + self._robot_link_names, + self._robot_local_transforms, + strict=True, + ): + world_transform = link_poses[link_name][step_idx] @ local_transform # [4,4] + frame_transforms.append( + ( + world_transform[:3, 3].astype(np.float32, copy=False), # [3] + world_transform[:3, :3].astype(np.float32, copy=False), # [3,3] + ) + ) + animated_transforms.append(frame_transforms) + + self._ik_right = animated_transforms + self._ik_left = None + self._robot_frames_right = None + self._robot_frames_left = None + + def _load_robot_and_ik(self, state: SceneState, entry: Any): + """Load robot meshes and solve IK for the episode.""" + if entry.robot_name == "embodiment_c": + embodiment_type = _get_entry_agibot_embodiment_type(entry) + if not embodiment_type: + log.warning("AgiBot entry missing embodiment_type — skipping robot animation") + return + + robot_key = (entry.robot_name, embodiment_type, "urdf_collision") + if self._current_robot != robot_key: + for h in self.robot_right + self.robot_left: + h.remove() + self.robot_right = [] + self.robot_left = [] + self._clear_robot_frame_handles(self._robot_frame_handles_right) + self._clear_robot_frame_handles(self._robot_frame_handles_left) + self._robot_link_names = [] + self._robot_local_transforms = [] + + geometries = get_embodiment_c_collision_geometries() + for geometry in geometries: + handle = self.server.scene.add_mesh_trimesh( + self._p(f"/robot_right/{geometry.name}"), + mesh=geometry.mesh, + ) + handle.visible = False + self.robot_right.append(handle) + self._robot_link_names.append(geometry.link_name) + self._robot_local_transforms.append(geometry.local_transform) + + self._current_robot = robot_key + log.info(f"Loaded {len(geometries)} AgiBot URDF collision geometries") + + if state.agibot_link_poses: + missing_links = [ + link_name for link_name in self._robot_link_names if link_name not in state.agibot_link_poses + ] + if not missing_links: + num_steps = min( + int(state.agibot_link_poses[link_name].shape[0]) for link_name in self._robot_link_names + ) + if num_steps > 0: + self._set_agibot_mesh_animation_from_link_poses(state.agibot_link_poses, num_steps) + return + log.warning( + f"AgiBot direct FK link poses missing {len(missing_links)} robot links; falling back to IK." + ) + + if state.right_poses is None and state.left_poses is None and state.ego_poses is None: + log.warning("AgiBot viewer has no head/left/right pose targets — skipping robot animation") + return + + left_wrist_poses_ik = _undo_wrist_to_opencv_for_ik( + state.left_poses, + self._to_opencv, + "left_wrist", + ) # [T,4,4] | None + right_wrist_poses_ik = _undo_wrist_to_opencv_for_ik( + state.right_poses, + self._to_opencv, + "right_wrist", + ) # [T,4,4] | None + + joint_configs = solve_agibot_trajectory_ik( + head_camera_poses=state.ego_poses, + left_wrist_poses=left_wrist_poses_ik, + right_wrist_poses=right_wrist_poses_ik, + left_gripper_openings=state.gripper_left, + right_gripper_openings=state.gripper_right, + ) + if joint_configs is None: + log.warning("AgiBot IK failed — skipping robot animation") + return + + link_poses = compute_agibot_link_poses_batch_from_configs( + joint_configs, + self._robot_link_names, + ) + self._set_agibot_mesh_animation_from_link_poses(link_poses, int(joint_configs.shape[0])) + return + + # Determine robot config key (single vs dual) + is_dual = state.mask.left_wrist and entry.dual_base_left is not None + robot_key = (entry.robot_name, "dual") if is_dual else entry.robot_name + + if self._robot_scene_model is None or self._robot_scene_model.robot_name != entry.robot_name: + try: + self._robot_scene_model = RobotSceneModel(entry.robot_name) + except Exception as e: + log.warning(f"RobotSceneModel unavailable for {entry.robot_name}: {e}") + self._robot_scene_model = None + return + + # Only reload meshes if robot changed + if self._current_robot != robot_key: + for h in self.robot_right + self.robot_left: + h.remove() + self.robot_right = [] + self.robot_left = [] + self._clear_robot_frame_handles(self._robot_frame_handles_right) + self._clear_robot_frame_handles(self._robot_frame_handles_left) + + if self._robot_scene_model is None: + self._current_robot = robot_key + return + + right_meshes = self._robot_scene_model.get_home_meshes(entry.dual_base_right if is_dual else None) + for name, mesh, transform in right_meshes: + h = self.server.scene.add_mesh_trimesh( + self._p(f"/robot_right/{name}"), + mesh=mesh, + ) + h.position = transform[:3, 3] + h.wxyz = self.vtf.SO3.from_matrix(transform[:3, :3]).wxyz + self.robot_right.append(h) + + if is_dual: + left_meshes = self._robot_scene_model.get_home_meshes(entry.dual_base_left) + for name, mesh, transform in left_meshes: + h = self.server.scene.add_mesh_trimesh( + self._p(f"/robot_left/{name}"), + mesh=mesh, + ) + h.position = transform[:3, 3] + h.wxyz = self.vtf.SO3.from_matrix(transform[:3, :3]).wxyz + self.robot_left.append(h) + + self._current_robot = robot_key + log.info(f"Loaded {len(right_meshes)} meshes for {entry.robot_name}") + + if self._robot_scene_model is None: + return + + # Joint-position datasets (e.g. robomind-ur): bypass IK, use FK directly + if state.joint_configs is not None: + from cosmos_framework.data.vfm.action.urdf_visualizer.ik_solver import compute_mujoco_geom_transforms + from cosmos_framework.data.vfm.action.urdf_visualizer.robot_scene_model import get_mjcf_path + + try: + mjcf_path = get_mjcf_path(entry.robot_name) + transforms, _, _fk_ee_poses, robot_frames = compute_mujoco_geom_transforms( + mjcf_path, state.joint_configs + ) + self._ik_right = transforms + self._robot_frames_right = robot_frames + self._rebuild_robot_frame_handles("right", robot_frames) + log.info(f"FK geom transforms computed for {len(transforms)} frames ({entry.robot_name})") + except Exception as e: + log.warning(f"FK failed for {entry.robot_name}: {e}") + import traceback + + traceback.print_exc() + return + + # Right arm IK + if state.right_poses is not None: + try: + right_result = self._robot_scene_model.solve_visual_trajectory( + state.right_poses, + gripper_openings=state.gripper_right, + to_opencv=self._to_opencv, + base_pose=entry.dual_base_right if is_dual else None, + ) + if right_result is not None: + self._ik_right = right_result.mesh_transforms + self._robot_frames_right = right_result.named_frames + self._rebuild_robot_frame_handles("right", self._robot_frames_right) + else: + self._ik_right = None + self._robot_frames_right = None + self._rebuild_robot_frame_handles("right", None) + except Exception as e: + log.warning(f"IK failed (right): {e}") + self._ik_right = None + self._robot_frames_right = None + self._rebuild_robot_frame_handles("right", None) + else: + self._ik_right = None + self._robot_frames_right = None + self._rebuild_robot_frame_handles("right", None) + + # Left arm IK (dual only) + if is_dual and state.left_poses is not None: + try: + left_result = self._robot_scene_model.solve_visual_trajectory( + state.left_poses, + gripper_openings=state.gripper_left, + to_opencv=self._to_opencv, + base_pose=entry.dual_base_left, + ) + if left_result is not None: + self._ik_left = left_result.mesh_transforms + self._robot_frames_left = left_result.named_frames + self._rebuild_robot_frame_handles("left", self._robot_frames_left) + else: + self._ik_left = None + self._robot_frames_left = None + self._rebuild_robot_frame_handles("left", None) + except Exception as e: + log.warning(f"IK failed (left): {e}") + self._ik_left = None + self._robot_frames_left = None + self._rebuild_robot_frame_handles("left", None) + else: + self._ik_left = None + self._robot_frames_left = None + self._rebuild_robot_frame_handles("left", None) + + def _update_robot(self, t: int, show: dict): + vis = show.get("robot", True) + for handles, ik in [(self.robot_right, self._ik_right), (self.robot_left, self._ik_left)]: + for idx, h in enumerate(handles): + if vis and ik is not None and t < len(ik) and idx < len(ik[t]): + p, m = ik[t][idx] + h.position = p + h.wxyz = self.vtf.SO3.from_matrix(m).wxyz + h.visible = True + else: + h.visible = False + self._update_robot_debug_frames(t, show) + + # ─── Visibility ─────────────────────────────────────────────────────────── + + def hide_all(self): + """Hide every scene element.""" + for attr in [ + self.ego_frame, + self.ego_frustum, + self.ego_frustum_up, + self.ego_traj, + self.right_frame, + self.right_ee, + self.right_traj, + self.left_frame, + self.left_ee, + self.left_traj, + ]: + attr.visible = False + for h in self.right_fingers + self.left_fingers: + h.visible = False + for h in self.right_gripper_tips + self.left_gripper_tips: + h.visible = False + for h in self.robot_right + self.robot_left: + h.visible = False + for handle in list(self._robot_frame_handles_right.values()) + list(self._robot_frame_handles_left.values()): + handle.visible = False + + def update_axis_scale(self, scale: float): + """Update coordinate frame axis size and effector point size.""" + self._axis_scale = scale + s = scale + self.ego_frame.axes_length = self._ego_axis_length_base * s + self.ego_frame.axes_radius = self._ego_axis_radius_base * s + self.ego_frustum.scale = self._ego_frustum_scale_base * s + self.ego_frustum_up.scale = self._ego_frustum_scale_base * s + for frame in (self.right_frame, self.left_frame): + frame.axes_length = self.HAND_AXIS_LENGTH * s + frame.axes_radius = self.HAND_AXIS_RADIUS * s + for ee in (self.right_ee, self.left_ee): + ee.point_size = 0.015 * s + for handles in (self._robot_frame_handles_right, self._robot_frame_handles_left): + for frame_key, handle in handles.items(): + axes_length, axes_radius = self._robot_frame_dims(frame_key) + handle.axes_length = axes_length * s + handle.axes_radius = axes_radius * s diff --git a/cosmos_framework/data/vfm/action/urdf_visualizer/ur5e_robotiq_2f85.xml b/cosmos_framework/data/vfm/action/urdf_visualizer/ur5e_robotiq_2f85.xml new file mode 100644 index 0000000..9f47762 --- /dev/null +++ b/cosmos_framework/data/vfm/action/urdf_visualizer/ur5e_robotiq_2f85.xml @@ -0,0 +1,326 @@ + + + + + + + diff --git a/cosmos_framework/data/vfm/action/urdf_visualizer/urdf_loader.py b/cosmos_framework/data/vfm/action/urdf_visualizer/urdf_loader.py new file mode 100644 index 0000000..9723f85 --- /dev/null +++ b/cosmos_framework/data/vfm/action/urdf_visualizer/urdf_loader.py @@ -0,0 +1,1107 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Robot mesh loading from mujoco_menagerie. + +Downloads robot MJCF/URDF/mesh assets on first use, caches in ~/.cache/mujoco_menagerie/. +Each loader returns (meshes, ee_home_pose) where: + - meshes: list of (name, trimesh.Trimesh, 4x4_transform) + - ee_home_pose: (4, 4) float32 — EE pose at home configuration +""" + +from __future__ import annotations + +import os +import xml.etree.ElementTree as ET +from dataclasses import dataclass +from functools import lru_cache +from pathlib import Path +from typing import Any + +import numpy as np +from scipy.spatial.transform import Rotation as R + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.embodiment_c_spec import get_embodiment_c_urdf_path + +_MENAGERIE_REPO = "https://github.com/google-deepmind/mujoco_menagerie" +_ROBOT_CACHE_DIR = Path("/tmp") / "mujoco_menagerie" +_AGIBOT_GEAR_LUSTRE_URDF_ROOT = Path("") +_AGIBOT_GEAR_LOCAL_SAMPLE_URDF_ROOT = Path(__file__).resolve().parents[6] / "dev/embodiment_c_samples/g1_urdf" +_FALLBACK_MESH_COLORS = np.asarray( + [ + [52, 152, 219, 255], + [243, 156, 18, 255], + [155, 89, 182, 255], + [46, 204, 113, 255], + [231, 76, 60, 255], + [26, 188, 156, 255], + [241, 196, 15, 255], + [52, 73, 94, 255], + [127, 140, 141, 255], + [211, 84, 0, 255], + [39, 174, 96, 255], + [41, 128, 185, 255], + ], + dtype=np.uint8, +) +_UNSPECIFIED_RGBA_VALUES = ( + np.asarray([0.5, 0.5, 0.5, 1.0], dtype=np.float32), + np.asarray([1.0, 1.0, 1.0, 1.0], dtype=np.float32), + # Some autogenerated URDF/MJCF assets use pink as a placeholder/default + # material. Treat it as unspecified so meshes do not all render pink. + np.asarray([1.0, 0.5, 1.0, 1.0], dtype=np.float32), +) +_UNSPECIFIED_MATERIAL_NAMES = {"", "default", "default_material"} + + +@dataclass(frozen=True) +class AgibotGeometry: + """One AgiBot collision primitive attached to a URDF link.""" + + name: str + link_name: str + mesh: Any + local_transform: np.ndarray + + +def _fallback_mesh_color(name: str, index: int = 0) -> np.ndarray: + """Return a stable fallback RGBA color for a mesh without material color.""" + + key = name or f"mesh_{index}" + palette_index = sum((char_idx + 1) * ord(char) for char_idx, char in enumerate(key)) + return _FALLBACK_MESH_COLORS[(palette_index + index) % len(_FALLBACK_MESH_COLORS)].copy() + + +def _is_unspecified_rgba(rgba: np.ndarray) -> bool: + """Return whether RGBA is an uninformative default color.""" + + return any(np.allclose(rgba, default_rgba, atol=1e-4) for default_rgba in _UNSPECIFIED_RGBA_VALUES) + + +def _apply_mesh_color(mesh: Any, rgba: np.ndarray) -> None: + """Apply one RGBA color to all mesh faces for viser/trimesh rendering.""" + + face_count = int(len(getattr(mesh, "faces", []))) + if face_count > 0: + mesh.visual.face_colors = np.tile(np.asarray(rgba, dtype=np.uint8), (face_count, 1)) + + +# ── Robot-specific configuration ────────────────────────────────────────────── + +# EE frame name candidates (tried in order by IK solver) +# --------------------------------------------------------------------------- +# End-effector frame references per dataset +# --------------------------------------------------------------------------- +# Each dataset records EE poses at a specific URDF link. The IK solver must +# target the same frame (via ``ee_frame`` or the ``_EE_FRAME_CANDIDATES`` +# fallback list). Below we document, for every robot: +# (a) which link the dataset records at, with upstream source reference, +# (b) what that link physically represents (flange, gripper body, or TCP), +# (c) any orientation transforms applied during recording. +# +# ── Google Robot (fractal / RT-1) ────────────────────────────────────────── +# Recorded link: ``link_gripper_tcp`` +# The TFDS observation field is ``base_pose_tool_reached`` (shape (7,)). +# Source: https://www.tensorflow.org/datasets/catalog/fractal20220817_data +# Meaning: TCP — a calibrated tool-center-point 164 mm past the wrist +# (``link_gripper``), roughly at the fingertip. The name contains "tool" +# and SimplerEnv's real→sim calibration targets the same link: +# ``self.ee_link_name = "link_gripper_tcp"`` +# Source: https://github.com/simpler-env/ManiSkill2_real2sim/blob/ef7a4d4/mani_skill2_real2sim/agents/configs/google_robot/defaults.py#L141 +# Orientation: recorded as-is from the URDF FK (quaternion). No extra +# rotation applied. +# Re-referenced to: ``link_gripper`` (gripper body / last actuated link) +# via ``_TCP_TO_FLANGE`` in the dataset class (−164 mm along local z). +# See ``fractal.py`` for the full transform and rationale. +# Our config: ``"ee_frame": "link_gripper"`` (explicit). +# Important MJCF note: MuJoCo Menagerie places ``base_link`` at +# ``pos="0 0 0.06205"`` under ``worldbody``. Pinocchio's +# ``buildModelFromMJCF()`` omits this global root transform, so IK / dataset +# poses live in a root-free world that is 62.05 mm lower than raw MuJoCo +# body/site poses. We therefore strip the same root transform from MuJoCo +# mesh/body/site outputs before rendering or comparing them. +# +# ── WidowX (bridge) ─────────────────────────────────────────────────────── +# Recorded link: ``ee_gripper_link`` +# The bridge data collection uses the Interbotix SDK's Modern Robotics FK +# (``FKinSpace(M, Slist, q)``). The M matrix for wx250s terminates at +# ``ee_gripper_link`` — confirmed by the Interbotix source comment and +# numerically: ``wx250s.M[0,3] = 0.458325`` matches ``ee_gripper_link`` +# position exactly (vs ``gripper_link`` at 0.3648). +# Source: https://github.com/Interbotix/interbotix_ros_toolboxes/blob/main/interbotix_xs_toolbox/interbotix_xs_modules/src/interbotix_xs_modules/mr_descriptions.py +# Line 1: "Note that the end-effector is positioned at '/ee_gripper_link'" +# SimplerEnv confirms: ``self.ee_link_name = "ee_gripper_link"`` +# Source: https://github.com/simpler-env/ManiSkill2_real2sim/blob/ef7a4d4/mani_skill2_real2sim/agents/configs/widowx/defaults.py#L87 +# Meaning: Interbotix-defined "end-effector" reference point, 93.6 mm past +# the wrist (``gripper_link``), 27.6 mm past the finger pivot +# (``fingers_link``). This is roughly the grasp center between the +# finger pads — *not* the fingertip. +# Orientation: the bridge data collection applies ``DEFAULT_ROTATION`` to +# the FK orientation before recording (in ``transform2state``): +# ``euler = rotationMatrixToEulerAngles(rot.dot(default_rotation.T))`` +# where ``DEFAULT_ROTATION = [[0,0,1],[0,1,0],[-1,0,0]]``. +# Source: https://github.com/rail-berkeley/bridge_data_robot/blob/main/widowx_envs/widowx_controller/src/widowx_controller/widowx_controller.py#L44 +# Re-referenced to: ``gripper_link`` (wrist rotate body / last actuated link) +# via ``_TCP_TO_FLANGE`` in the dataset class (−93.6 mm along local x). +# See ``bridge_orig_lerobot_dataset.py`` for the full transform and rationale. +# Our config: ``"ee_frame": "gripper_link"`` (explicit). +# +# ── Franka + Robotiq 2F-85 (DROID) ──────────────────────────────────────── +# Recorded link: ``panda_link8`` (= Franka flange) +# DROID's ``get_robot_state()`` calls Polymetis FK: +# ``pos, quat = self._robot.robot_model.forward_kinematics(joint_positions)`` +# Source: https://github.com/droid-dataset/droid/blob/main/droid/franka/robot.py +# Polymetis is configured with ``ee_link_name: panda_link8``: +# Source: https://github.com/facebookresearch/fairo/blob/main/polymetis/polymetis/conf/robot_model/franka_panda.yaml +# Meaning: flange — the bare mounting plate at the end of the Franka arm, +# *before* the Robotiq gripper. +# MJCF body: ``panda_hand`` in the composite model (pos 0 0 0.107 +# above ``link7``, identity quat). This is at the panda_link8 +# (flange) position with link7 orientation, matching the URDF. +# ``link7`` is panda_link7 (wrist), *not* the flange. +# Orientation: Polymetis reports FK orientation at ``panda_link8`` frame +# as-is. No extra rotation applied. +# Current viewer convention: all Franka-backed datasets reuse this DROID +# composite model, so ``franka_panda`` is the only supported Franka robot +# config in this module. +# Our config: ``"ee_frame": "panda_hand"`` (explicit). +# --------------------------------------------------------------------------- + +_EE_FRAME_CANDIDATES = [ + "link_gripper_tcp", # SimplerEnv Google Robot (calibrated TCP at fingertip) + "link_gripper", # Google Robot (Menagerie / fallback) + "ee_gripper_link", # SimplerEnv WidowX (end of finger chain) + "gripper_link", # WidowX (MuJoCo body name) + "wx250s/gripper_link", # WidowX Menagerie (prefixed body name) + "gripper", # Google Robot (Menagerie site) + "hand", # Franka Panda (standard gripper) + "attachment", # Franka + Robotiq + "attachment_site", # UR5e (mujoco_menagerie) + "wx250s/gripper_link", # WidowX 250S (Menagerie) + "ee_link", # generic + "tool0", # generic industrial +] + +# Robot-specific configs: menagerie name, MJCF filename, joint info, finger range. +ROBOT_CONFIGS = { + "google_robot": { + "menagerie": "google_robot", + "mjcf": "robot.xml", + # EE: link_gripper (gripper body / last actuated link). + # The Menagerie model has link_gripper directly (no TCP frame). + # The dataset applies _TCP_TO_FLANGE to shift from the recorded + # link_gripper_tcp to link_gripper, so poses already target this frame. + # MuJoCo Menagerie also adds a global worldbody -> base_link offset + # (+62.05 mm in z). Pinocchio's MJCF importer omits that root transform, + # so the viewer removes it from MuJoCo outputs to keep meshes / body + # frames aligned with dataset poses and IK outputs. + "ee_frame": "link_gripper", + "pinocchio_removed_root_body": "base_link", + "arm_joints": [ + "joint_torso", + "joint_shoulder", + "joint_bicep", + "joint_elbow", + "joint_forearm", + "joint_wrist", + "joint_gripper", + ], + "n_arm_joints": 7, + "finger_joint_names": ["joint_finger_left", "joint_finger_right"], + "finger_min": 0.01, + "finger_max": 1.30, + "finger_close_is_max": True, + "camera_body": None, + }, + "franka_panda": { + "menagerie": "droid_franka_robotiq", + "mjcf": "panda_updated_robotiq_2f85.xml", + # EE: panda_link8 (flange) — the bare mounting plate, *not* the + # gripper TCP. MJCF body: ``panda_hand`` in the composite model + # (pos 0 0 0.107 above link7, identity quat). + + "ee_frame": "panda_hand", + "arm_joints": [ + "joint1", + "joint2", + "joint3", + "joint4", + "joint5", + "joint6", + "joint7", + ], + "n_arm_joints": 7, + "finger_min": 0.0, + "finger_max": 0.8, + "finger_close_is_max": True, + # All Robotiq 2F-85 chain joints (some share names with bodies in + # pinocchio MJCF parse — must keep all to avoid buildReducedModel error) + "finger_joint_names": [ + "right_driver_joint", + "left_driver_joint", + "left_spring_link_joint", + "left_follower", + "right_spring_link_joint", + "right_follower_joint", + ], + "camera_body": None, + }, + "widowx": { + "menagerie": "trossen_wx250s", + "mjcf": "wx250s.xml", + # EE: gripper_link (wrist rotate body / last actuated link). + # The Menagerie model has gripper_link directly. + # The dataset applies _TCP_TO_FLANGE to shift from the recorded + # ee_gripper_link to gripper_link, so poses already target this frame. + "ee_frame": "gripper_link", + "arm_joints": [ + "waist", + "shoulder", + "elbow", + "forearm_roll", + "wrist_angle", + "wrist_rotate", + ], + "n_arm_joints": 6, + "finger_min": 0.015, + "finger_max": 0.037, + "finger_close_is_max": False, + "finger_joint_names": ["left_finger", "right_finger"], + "camera_body": None, + }, + "ur5e": { + "menagerie": "ur5e_robotiq", + "mjcf": "ur5e_robotiq_2f85.xml", + # EE: attachment_site — the UR5e flange site where the Robotiq mounts. + # joint_configs[:, 6] raw UR gripper maps directly to Robotiq ctrl: + # ctrl = raw * 255 (0=open, 255=closed). + "ee_frame": "attachment_site", + "n_arm_joints": 6, + "finger_min": 0.0, + "finger_max": 255.0, + "finger_close_is_max": False, + "camera_body": None, + }, + "embodiment_c": { + # No menagerie — uses local AgiBot G1/omnipicker URDF assets. + # The dataset targets the gripper base/flange, not the TCP/center link. + "menagerie": None, + "ee_frame": "gripper_r_base_link", + "arm_joints": [f"idx{11 + i:02d}_right_arm_joint{i}" for i in range(1, 8)], + "n_arm_joints": 7, + "finger_min": 0.0, + "finger_max": 1.0, + "finger_close_is_max": False, + "camera_body": "head_pitch_link", + }, +} + + +def get_urdf_path(robot_name: str) -> str | None: + """Get URDF path for a robot if available, else None (falls back to MJCF in caller).""" + cfg = ROBOT_CONFIGS.get(robot_name) + if cfg is None: + return None + + if robot_name == "embodiment_c": + return str(get_embodiment_c_urdf_path()) + + menagerie_name = cfg.get("menagerie") + if menagerie_name is None: + return None + mjcf_dir = _ensure_robot_assets(menagerie_name) + + urdf_filename = cfg.get("urdf") + if urdf_filename: + urdf_path = mjcf_dir / urdf_filename + if urdf_path.exists(): + return str(urdf_path) + + urdfs = list(mjcf_dir.glob("*.urdf")) + if urdfs: + return str(urdfs[0]) + + return None + + +def get_mjcf_path(robot_name: str) -> str: + """Get MJCF path for a robot, downloading from mujoco_menagerie if needed.""" + cfg = ROBOT_CONFIGS.get(robot_name) + if cfg is None: + raise ValueError(f"Unknown robot: {robot_name}. Available: {list(ROBOT_CONFIGS.keys())}") + + menagerie_name = cfg.get("menagerie") + if menagerie_name is None: + raise FileNotFoundError(f"Robot {robot_name!r} does not have a MuJoCo MJCF asset.") + mjcf_filename = cfg.get("mjcf", "robot.xml") + + mjcf_dir = _ensure_robot_assets(menagerie_name) + mjcf_path = mjcf_dir / mjcf_filename + if mjcf_path.exists(): + return str(mjcf_path) + + fallback = mjcf_dir / "robot.xml" + if fallback.exists(): + return str(fallback) + + raise FileNotFoundError(f"MJCF not found: {mjcf_path} (also tried robot.xml)") + + +# ── Asset download ─────────────────────────────────────────────────────────── + + +def _ensure_robot_assets(robot_name: str) -> Path: + """Ensure robot MJCF and mesh assets are available locally. + + Downloads from mujoco_menagerie GitHub repo if not found. + For composite robots (``droid_franka_robotiq``), downloads from both + MuJoCo Menagerie and MuJoCo Playground and merges assets. + + Assets are cached in ~/.cache/mujoco_menagerie//. + + Returns: + Path to the cached MJCF directory (caller picks the right .xml). + """ + cached_dir = _ROBOT_CACHE_DIR / robot_name + if cached_dir.exists(): + return cached_dir + + # Composite: Franka arm + Robotiq 2F-85 gripper (DROID setup) + if robot_name == "droid_franka_robotiq": + return _build_droid_franka_robotiq() + + # Composite: UR5e arm + Robotiq 2F-85 gripper (RoboMIND UR setup) + if robot_name == "ur5e_robotiq": + return _build_ur5e_robotiq() + + log.info(f"Downloading {robot_name} assets from mujoco_menagerie...") + _download_menagerie_model(robot_name) + return cached_dir + + +def _download_menagerie_model(model_name: str) -> Path: + """Download a single model from MuJoCo Menagerie. Returns cached path.""" + cached_dir = _ROBOT_CACHE_DIR / model_name + if cached_dir.exists(): + return cached_dir + + import shutil + import subprocess + import tempfile + + _ROBOT_CACHE_DIR.mkdir(parents=True, exist_ok=True) + with tempfile.TemporaryDirectory() as tmpdir: + subprocess.run( + ["git", "clone", "--depth=1", "--filter=blob:none", "--sparse", _MENAGERIE_REPO, tmpdir], + check=True, + capture_output=True, + ) + subprocess.run( + ["git", "sparse-checkout", "set", model_name], + cwd=tmpdir, + check=True, + capture_output=True, + ) + src = Path(tmpdir) / model_name + dst = _ROBOT_CACHE_DIR / model_name + if dst.exists(): + shutil.rmtree(dst) + shutil.copytree(src, dst) + + log.info(f"Cached {model_name} assets at {dst}") + return dst + + +def _build_droid_franka_robotiq() -> Path: + """Prepare composite Franka + Robotiq 2F-85 MJCF for DROID visualization. + + The hand-tuned composite XML is committed in the repo alongside this + module (``droid_franka_robotiq_2f85.xml``). This function downloads the + mesh assets from MuJoCo Menagerie (``franka_emika_panda`` + + ``robotiq_2f85_v4``) and copies them together with the XML into a cache + directory that MuJoCo can load from. + """ + import shutil + + log.info("Preparing composite Franka + Robotiq 2F-85 (committed XML + Menagerie assets)...") + + franka_dir = _download_menagerie_model("franka_emika_panda") + robotiq_dir = _download_menagerie_model("robotiq_2f85_v4") + + dst = _ROBOT_CACHE_DIR / "droid_franka_robotiq" + dst.mkdir(parents=True, exist_ok=True) + assets_dir = dst / "assets" + assets_dir.mkdir(exist_ok=True) + + for f in (franka_dir / "assets").iterdir(): + shutil.copy2(f, assets_dir / f.name) + for f in (robotiq_dir / "assets").iterdir(): + shutil.copy2(f, assets_dir / f.name) + + committed_xml = Path(__file__).parent / "droid_franka_robotiq_2f85.xml" + out_path = dst / "panda_updated_robotiq_2f85.xml" + shutil.copy2(committed_xml, out_path) + + log.info(f"Prepared composite Franka + Robotiq 2F-85 at {dst}") + return dst + + +def _build_ur5e_robotiq() -> Path: + """Prepare composite UR5e + Robotiq 2F-85 MJCF for RoboMIND UR visualization. + + The hand-tuned composite XML is committed in the repo alongside this + module (``ur5e_robotiq_2f85.xml``). This function downloads the mesh + assets from MuJoCo Menagerie (``universal_robots_ur5e`` + + ``robotiq_2f85_v4``) and copies them together with the XML into a cache + directory that MuJoCo can load from. + """ + import shutil + + log.info("Preparing composite UR5e + Robotiq 2F-85 (committed XML + Menagerie assets)...") + + ur5e_dir = _download_menagerie_model("universal_robots_ur5e") + robotiq_dir = _download_menagerie_model("robotiq_2f85_v4") + + dst = _ROBOT_CACHE_DIR / "ur5e_robotiq" + dst.mkdir(parents=True, exist_ok=True) + assets_dir = dst / "assets" + assets_dir.mkdir(exist_ok=True) + + for f in (ur5e_dir / "assets").iterdir(): + shutil.copy2(f, assets_dir / f.name) + for f in (robotiq_dir / "assets").iterdir(): + shutil.copy2(f, assets_dir / f.name) + + committed_xml = Path(__file__).parent / "ur5e_robotiq_2f85.xml" + shutil.copy2(committed_xml, dst / "ur5e_robotiq_2f85.xml") + + log.info(f"Prepared composite UR5e + Robotiq 2F-85 at {dst}") + return dst + + +# ── Robot loaders ──────────────────────────────────────────────────────────── + + +def _load_google_robot() -> tuple[list, np.ndarray]: + """Load Google Robot from MuJoCo Menagerie. + + Uses the official google_robot model from + https://github.com/google-deepmind/mujoco_menagerie/tree/main/google_robot + which has visual OBJ meshes for accurate rendering. + """ + import mujoco + + cfg = ROBOT_CONFIGS["google_robot"] + mjcf_dir = _ensure_robot_assets(cfg["menagerie"]) + mjcf_path = mjcf_dir / cfg["mjcf"] + model = mujoco.MjModel.from_xml_path(str(mjcf_path)) + data = mujoco.MjData(model) + mujoco.mj_forward(model, data) + + meshes = _extract_mujoco_meshes(model, data) + world_correction = get_mujoco_to_pinocchio_world_transform(model, data, "google_robot") + if not np.allclose(world_correction, np.eye(4, dtype=np.float32)): + meshes = _apply_world_transform_to_meshes(meshes, world_correction) + + # Find EE pose at link_gripper + ee_pose = np.eye(4, dtype=np.float32) + for i in range(model.nbody): + name = mujoco.mj_id2name(model, mujoco.mjtObj.mjOBJ_BODY, i) or "" + if name == "link_gripper": + ee_pose[:3, 3] = data.xpos[i].astype(np.float32) + ee_pose[:3, :3] = data.xmat[i].reshape(3, 3).astype(np.float32) + break + ee_pose = (world_correction @ ee_pose).astype(np.float32) + + log.info(f"Google Robot (Menagerie) loaded: {len(meshes)} meshes, EE pos={ee_pose[:3, 3]}") + return meshes, ee_pose + + +def _load_franka_panda() -> tuple[list, np.ndarray]: + """Load Franka Panda + Robotiq 2F-85 gripper (DROID variant). + + Uses the composite model built from MuJoCo Menagerie + (``franka_emika_panda`` arm + ``robotiq_2f85_v4`` gripper). + """ + import mujoco + + mjcf_dir = _ensure_robot_assets("droid_franka_robotiq") + mjcf_path = mjcf_dir / "panda_updated_robotiq_2f85.xml" + model = mujoco.MjModel.from_xml_path(str(mjcf_path)) + data = mujoco.MjData(model) + + # Franka home configuration + home_qpos = np.array([0.0, -0.78, 0.0, -2.36, 0.0, 1.57, 0.78]) + data.qpos[: len(home_qpos)] = home_qpos + mujoco.mj_forward(model, data) + + meshes = _extract_mujoco_meshes(model, data) + + # EE pose: use the "gripper" site (on the Robotiq base), or fall + # back to "base_mount" / "attachment" body if site isn't found. + ee_pose = np.eye(4, dtype=np.float32) + for si in range(model.nsite): + name = mujoco.mj_id2name(model, mujoco.mjtObj.mjOBJ_SITE, si) or "" + if name == "gripper": + ee_pose[:3, 3] = data.site_xpos[si].astype(np.float32) + ee_pose[:3, :3] = data.site_xmat[si].reshape(3, 3).astype(np.float32) + break + else: + # Fallback to body search + for i in range(model.nbody): + name = mujoco.mj_id2name(model, mujoco.mjtObj.mjOBJ_BODY, i) or "" + if name in ("base_mount", "attachment"): + ee_pose[:3, 3] = data.xpos[i].astype(np.float32) + ee_pose[:3, :3] = data.xmat[i].reshape(3, 3).astype(np.float32) + break + + log.info(f"Franka Panda + Robotiq loaded: {len(meshes)} meshes, EE pos={ee_pose[:3, 3]}") + return meshes, ee_pose + + +def _load_widowx() -> tuple[list, np.ndarray]: + """Load WidowX 250S from MuJoCo Menagerie (trossen_wx250s). + + Uses the official Trossen Robotics WX250S model from + https://github.com/google-deepmind/mujoco_menagerie/tree/main/trossen_wx250s + which has visual OBJ meshes for accurate rendering. + """ + import mujoco + + cfg = ROBOT_CONFIGS["widowx"] + mjcf_dir = _ensure_robot_assets(cfg["menagerie"]) + mjcf_path = mjcf_dir / cfg["mjcf"] + model = mujoco.MjModel.from_xml_path(str(mjcf_path)) + data = mujoco.MjData(model) + mujoco.mj_forward(model, data) + + meshes = _extract_mujoco_meshes(model, data) + + ee_pose = np.eye(4, dtype=np.float32) + for i in range(model.nbody): + name = mujoco.mj_id2name(model, mujoco.mjtObj.mjOBJ_BODY, i) or "" + if name == "gripper_link": + ee_pose[:3, 3] = data.xpos[i].astype(np.float32) + ee_pose[:3, :3] = data.xmat[i].reshape(3, 3).astype(np.float32) + break + + log.info(f"WidowX 250S (Menagerie) loaded: {len(meshes)} meshes, EE pos={ee_pose[:3, 3]}") + return meshes, ee_pose + + +def _load_ur5e() -> tuple[list, np.ndarray]: + """Load UR5e + Robotiq 2F-85 composite from MuJoCo Menagerie.""" + import mujoco + + mjcf_dir = _ensure_robot_assets("ur5e_robotiq") + mjcf_path = mjcf_dir / "ur5e_robotiq_2f85.xml" + model = mujoco.MjModel.from_xml_path(str(mjcf_path)) + data = mujoco.MjData(model) + + # UR5e home: -90, -90, 90, -90, -90, 0 (degrees → radians); gripper open + home_qpos = np.array([-1.5708, -1.5708, 1.5708, -1.5708, -1.5708, 0.0]) + data.qpos[: len(home_qpos)] = home_qpos + mujoco.mj_forward(model, data) + + meshes = _extract_mujoco_meshes(model, data) + + # EE pose: use robotiq_base body (the flange-to-gripper attachment point) + ee_pose = np.eye(4, dtype=np.float32) + for i in range(model.nbody): + name = mujoco.mj_id2name(model, mujoco.mjtObj.mjOBJ_BODY, i) or "" + if name == "robotiq_base": + ee_pose[:3, 3] = data.xpos[i].astype(np.float32) + ee_pose[:3, :3] = data.xmat[i].reshape(3, 3).astype(np.float32) + break + + log.info(f"UR5e+Robotiq loaded: {len(meshes)} meshes, EE pos={ee_pose[:3, 3]}") + return meshes, ee_pose + + +def _load_embodiment_c() -> tuple[list, np.ndarray]: + """Load Embodiment C G1/omnipicker robot meshes from the local URDF.""" + + from cosmos_framework.data.vfm.action.embodiment_c_fk import compute_link_poses_batch + + poses_batch = compute_link_poses_batch(np.zeros((1, 32), dtype=np.float32), "embodiment_c_gripper") + poses = {link_name: link_poses[0] for link_name, link_poses in poses_batch.items()} + geometries = get_embodiment_c_collision_geometries() + meshes = [] + for geometry in geometries: + link_pose = poses.get(geometry.link_name) + if link_pose is None: + continue + world_transform = link_pose @ geometry.local_transform + meshes.append((geometry.name, geometry.mesh.copy(), world_transform.astype(np.float32))) + + ee_pose = poses.get("gripper_r_base_link", np.eye(4, dtype=np.float32)).astype(np.float32) + log.info(f"Embodiment C (G1 omnipicker geometry) loaded: {len(meshes)} meshes, EE pos={ee_pose[:3, 3]}") + return meshes, ee_pose + + +def _parse_urdf_origin(origin_element) -> np.ndarray: + transform = np.eye(4, dtype=np.float32) + if origin_element is None: + return transform + + xyz_text = origin_element.attrib.get("xyz", "0 0 0") + rpy_text = origin_element.attrib.get("rpy", "0 0 0") + xyz = np.asarray([float(value) for value in xyz_text.split()], dtype=np.float32) + rpy = np.asarray([float(value) for value in rpy_text.split()], dtype=np.float32) + transform[:3, :3] = R.from_euler("xyz", rpy).as_matrix().astype(np.float32) + transform[:3, 3] = xyz + return transform + + +def _parse_urdf_color(color_element: ET.Element | None) -> np.ndarray | None: + if color_element is None: + return None + rgba_text = color_element.attrib.get("rgba") + if not rgba_text: + return None + rgba = np.asarray([float(value) for value in rgba_text.split()], dtype=np.float32) + if rgba.shape != (4,): + return None + return (np.clip(rgba, 0.0, 1.0) * 255.0).astype(np.uint8) + + +def _parse_urdf_material_colors(root: ET.Element) -> dict[str, np.ndarray]: + material_colors: dict[str, np.ndarray] = {} + for material_element in root.findall("material"): + material_name = material_element.attrib.get("name", "") + color = _parse_urdf_color(material_element.find("color")) + if material_name and color is not None: + material_colors[material_name] = color + return material_colors + + +def _resolve_urdf_visual_color(parent_element: ET.Element, material_colors: dict[str, np.ndarray]) -> np.ndarray | None: + material_element = parent_element.find("material") + if material_element is None: + return None + + material_name = material_element.attrib.get("name", "") + color = _parse_urdf_color(material_element.find("color")) + if color is None and material_name: + color = material_colors.get(material_name) + if color is None: + return None + + color_unit = color.astype(np.float32) / 255.0 + if material_name.lower() in _UNSPECIFIED_MATERIAL_NAMES or _is_unspecified_rgba(color_unit): + return None + return color + + +def _agibot_urdf_asset_roots() -> list[Path]: + """Return candidate roots that can satisfy package://genie_robot_description assets.""" + + roots: list[Path] = [] + env_urdf_root = os.environ.get("AGIBOT_GEAR_URDF_ROOT") + if env_urdf_root: + roots.append(Path(env_urdf_root)) + env_mesh_root = os.environ.get("AGIBOT_GEAR_MESH_ROOT") + if env_mesh_root: + roots.append(Path(env_mesh_root)) + roots.extend( + [ + get_embodiment_c_urdf_path().parent, + _AGIBOT_GEAR_LUSTRE_URDF_ROOT, + _AGIBOT_GEAR_LOCAL_SAMPLE_URDF_ROOT, + ] + ) + + unique_roots: list[Path] = [] + seen: set[Path] = set() + for root in roots: + resolved = root.expanduser() + if resolved in seen: + continue + seen.add(resolved) + unique_roots.append(resolved) + return unique_roots + + +def _resolve_agibot_mesh_path(filename: str) -> Path: + if filename.startswith("package://genie_robot_description/"): + rel = filename.removeprefix("package://genie_robot_description/") + candidates: list[Path] = [] + for root in _agibot_urdf_asset_roots(): + candidates.append(root / rel) + if rel.startswith("meshes/"): + candidates.append(root / rel.removeprefix("meshes/")) + for candidate in candidates: + if candidate.is_file(): + return candidate + return candidates[0] + path = Path(filename) + if path.is_absolute(): + return path + return get_embodiment_c_urdf_path().parent / path + + +def _build_urdf_geometry_mesh(geometry_element: ET.Element | None, warn_missing_mesh: bool = False) -> Any | None: + import trimesh + + if geometry_element is None: + return None + + mesh_element = geometry_element.find("mesh") + if mesh_element is not None: + filename = mesh_element.attrib.get("filename") + if not filename: + return None + mesh_path = _resolve_agibot_mesh_path(filename) + if not mesh_path.is_file(): + if warn_missing_mesh: + log.warning(f"AgiBot mesh file missing: {mesh_path}") + return None + try: + mesh = trimesh.load(mesh_path, force="mesh") + if hasattr(mesh, "geometry"): + mesh = trimesh.util.concatenate(tuple(mesh.geometry.values())) + except Exception as exc: + log.warning(f"Failed to load AgiBot mesh {mesh_path}: {exc}") + return None + scale_text = mesh_element.attrib.get("scale") + if scale_text: + mesh.apply_scale([float(value) for value in scale_text.split()]) + return mesh + + box = geometry_element.find("box") + if box is not None: + size = np.asarray([float(value) for value in box.attrib["size"].split()], dtype=np.float32) + return trimesh.creation.box(extents=size) + + sphere = geometry_element.find("sphere") + if sphere is not None: + radius = float(sphere.attrib["radius"]) + return trimesh.creation.icosphere(radius=radius) + + cylinder = geometry_element.find("cylinder") + if cylinder is not None: + radius = float(cylinder.attrib["radius"]) + length = float(cylinder.attrib["length"]) + return trimesh.creation.cylinder(radius=radius, height=length) + + return None + + +@lru_cache(maxsize=1) +def _load_embodiment_c_collision_geometries() -> tuple[AgibotGeometry, ...]: + root = ET.parse(get_embodiment_c_urdf_path()).getroot() + material_colors = _parse_urdf_material_colors(root) + geometries: list[AgibotGeometry] = [] + + for link_element in root.findall("link"): + link_name = link_element.attrib["name"] + visual_elements = link_element.findall("visual") + collision_elements = link_element.findall("collision") + # Prefer visual meshes when the real asset files are available. Collision + # primitives remain the fallback for environments without AgiBot mesh assets. + for elements, warn_missing_meshes in ((visual_elements, False), (collision_elements, False)): + loaded_before = len(geometries) + for geometry_idx, parent_element in enumerate(elements): + geometry_element = parent_element.find("geometry") + mesh = _build_urdf_geometry_mesh(geometry_element, warn_missing_mesh=warn_missing_meshes) + if mesh is None: + continue + geometry_name = f"{link_name}_geometry_{geometry_idx}" + face_color = _resolve_urdf_visual_color(parent_element, material_colors) + if face_color is None: + face_color = _fallback_mesh_color(geometry_name, geometry_idx) + _apply_mesh_color(mesh, face_color) + local_transform = _parse_urdf_origin(parent_element.find("origin")) + geometries.append( + AgibotGeometry( + name=geometry_name, + link_name=link_name, + mesh=mesh, + local_transform=local_transform, + ) + ) + if len(geometries) > loaded_before: + break + + return tuple(geometries) + + +def get_embodiment_c_collision_geometries() -> list[AgibotGeometry]: + """Return cached AgiBot primitive collision geometries from the committed G1 URDF.""" + + return list(_load_embodiment_c_collision_geometries()) + + +# ── Helpers ────────────────────────────────────────────────────────────────── + + +def _get_visual_geom_ids(model) -> list[int]: + """Get visual geom IDs from a MuJoCo model. + + Strategy: + 1. Google Robot uses *_v suffix for visual meshes → use those + 2. Other robots (Franka, WidowX) use geom_group=2 for visual → use those + 3. Fallback: all mesh geoms + + Includes both mesh geoms and primitive geoms (box, cylinder, sphere) in group 2. + """ + import mujoco + + VISUAL_TYPES = { + mujoco.mjtGeom.mjGEOM_MESH, + mujoco.mjtGeom.mjGEOM_BOX, + mujoco.mjtGeom.mjGEOM_CYLINDER, + mujoco.mjtGeom.mjGEOM_SPHERE, + mujoco.mjtGeom.mjGEOM_CAPSULE, + mujoco.mjtGeom.mjGEOM_ELLIPSOID, + } + + v_ids = [] + group2_ids = [] + all_ids = [] + for gi in range(model.ngeom): + gtype = model.geom_type[gi] + if gtype not in VISUAL_TYPES: + continue + all_ids.append(gi) + if gtype == mujoco.mjtGeom.mjGEOM_MESH: + mesh_id = model.geom_dataid[gi] + mesh_name = mujoco.mj_id2name(model, mujoco.mjtObj.mjOBJ_MESH, mesh_id) or "" + if mesh_name.endswith("_v"): + v_ids.append(gi) + if model.geom_group[gi] == 2: + group2_ids.append(gi) + + # Priority: _v suffix > group 2 > all + if v_ids: + return v_ids + if group2_ids: + return group2_ids + return all_ids + + +def _resolve_geom_color(model, gi: int, geom_name: str) -> np.ndarray: + """Resolve the effective RGBA color for a MuJoCo geom. + + Priority: + 1. If geom has a material with a diffuse texture → sample average texture color + 2. If geom has a material with non-default rgba → use mat_rgba + 3. If geom_rgba is non-default → use geom_rgba + 4. Fallback: deterministic per-geom color + + Returns (4,) uint8 RGBA. + """ + + rgba = model.geom_rgba[gi].copy() # (4,) float [0,1] + mat_id = model.geom_matid[gi] + + if mat_id >= 0: + # Check if material has a diffuse texture + tex_id = model.mat_texid[mat_id][1] # index 1 = diffuse texture + if tex_id >= 0: + # Sample average color from the texture + tex_adr = model.tex_adr[tex_id] + tex_w = model.tex_width[tex_id] + tex_h = model.tex_height[tex_id] + n_pixels = tex_w * tex_h + tex_data = model.tex_data[tex_adr : tex_adr + n_pixels * 3].reshape(n_pixels, 3) + avg_rgb = tex_data.mean(axis=0) / 255.0 + # Multiply by mat_rgba (which acts as a tint) + mat_rgba = model.mat_rgba[mat_id] + final = np.array( + [ + avg_rgb[0] * mat_rgba[0], + avg_rgb[1] * mat_rgba[1], + avg_rgb[2] * mat_rgba[2], + mat_rgba[3], + ] + ) + return (np.clip(final, 0, 1) * 255).astype(np.uint8) + + # No texture — use material rgba if it is informative. + mat_rgba = model.mat_rgba[mat_id].copy() + if not _is_unspecified_rgba(mat_rgba): + return (np.clip(mat_rgba, 0, 1) * 255).astype(np.uint8) + + if not _is_unspecified_rgba(rgba): + return (np.clip(rgba, 0, 1) * 255).astype(np.uint8) + + return _fallback_mesh_color(geom_name, gi) + + +def _extract_mujoco_meshes(model, data) -> list: + """Extract visual geom meshes from a MuJoCo model as trimesh objects. + + Uses the same visual geom filtering as ``ik_solver._get_visual_geom_ids`` + to ensure mesh ordering matches IK transform output. + + For geoms with primitive types (box, cylinder, etc.), generates the + corresponding trimesh primitive. + + Applies per-geom RGBA color from MuJoCo materials when available. + + Returns list of (geom_name, trimesh.Trimesh, 4x4_transform). + """ + import mujoco + import trimesh + + # Use the same visual filter as IK solver + visual_geom_ids = _get_visual_geom_ids(model) + + meshes = [] + for gi in visual_geom_ids: + gtype = model.geom_type[gi] + pos = data.geom_xpos[gi].copy() + rot = data.geom_xmat[gi].reshape(3, 3).copy() + + geom_name = mujoco.mj_id2name(model, mujoco.mjtObj.mjOBJ_GEOM, gi) or f"geom_{gi}" + face_color = _resolve_geom_color(model, gi, geom_name) + + mesh = None + if gtype == mujoco.mjtGeom.mjGEOM_MESH: + mesh_id = model.geom_dataid[gi] + vert_start = model.mesh_vertadr[mesh_id] + vert_count = model.mesh_vertnum[mesh_id] + face_start = model.mesh_faceadr[mesh_id] + face_count = model.mesh_facenum[mesh_id] + verts = model.mesh_vert[vert_start : vert_start + vert_count].copy() + faces = model.mesh_face[face_start : face_start + face_count].copy() + mesh = trimesh.Trimesh(vertices=verts, faces=faces) + _apply_mesh_color(mesh, face_color) + elif gtype == mujoco.mjtGeom.mjGEOM_BOX: + size = model.geom_size[gi].copy() + mesh = trimesh.creation.box(extents=size * 2) + _apply_mesh_color(mesh, face_color) + elif gtype == mujoco.mjtGeom.mjGEOM_SPHERE: + mesh = trimesh.creation.icosphere(radius=model.geom_size[gi][0]) + _apply_mesh_color(mesh, face_color) + elif gtype == mujoco.mjtGeom.mjGEOM_CYLINDER: + mesh = trimesh.creation.cylinder(radius=model.geom_size[gi][0], height=model.geom_size[gi][1] * 2) + _apply_mesh_color(mesh, face_color) + elif gtype == mujoco.mjtGeom.mjGEOM_CAPSULE: + mesh = trimesh.creation.capsule(radius=model.geom_size[gi][0], height=model.geom_size[gi][1] * 2) + _apply_mesh_color(mesh, face_color) + + if mesh is not None: + transform = np.eye(4) + transform[:3, :3] = rot + transform[:3, 3] = pos + meshes.append((geom_name, mesh, transform)) + return meshes + + +def get_mujoco_to_pinocchio_world_transform(model, data, robot_name: str | None = None) -> np.ndarray: + """Return a correction that maps MuJoCo world poses into Pinocchio world. + + For some MJCFs, MuJoCo includes a fixed worldbody -> root-body transform + that Pinocchio's ``buildModelFromMJCF()`` omits. When that happens, the IK + solver and dataset poses live in a root-free world, while raw MuJoCo + body/site poses are globally shifted. This helper returns the inverse root + transform so callers can strip that offset from MuJoCo-derived meshes, + body frames, and sites before comparing them against Pinocchio / dataset + poses. + """ + import mujoco + + cfg = ROBOT_CONFIGS.get(robot_name, {}) if robot_name else {} + root_body_name = cfg.get("pinocchio_removed_root_body") + if not root_body_name: + return np.eye(4, dtype=np.float32) + + root_body_id = mujoco.mj_name2id(model, mujoco.mjtObj.mjOBJ_BODY, root_body_name) + if root_body_id < 0: + log.warning(f"Configured root body '{root_body_name}' not found in MuJoCo model") + return np.eye(4, dtype=np.float32) + + root_pose = np.eye(4, dtype=np.float32) + root_pose[:3, 3] = data.xpos[root_body_id].astype(np.float32) + root_pose[:3, :3] = data.xmat[root_body_id].reshape(3, 3).astype(np.float32) + return np.linalg.inv(root_pose).astype(np.float32) + + +def _apply_world_transform_to_meshes( + meshes: list[tuple[str, object, np.ndarray]], + world_transform: np.ndarray, +) -> list[tuple[str, object, np.ndarray]]: + """Left-multiply a world-space transform into each mesh pose.""" + transformed_meshes = [] + for name, mesh, transform in meshes: + transformed_meshes.append((name, mesh, (world_transform @ transform).astype(np.float32))) + return transformed_meshes + + +# ── Public API ─────────────────────────────────────────────────────────────── + + +def get_robot_loaders() -> dict[str, callable]: + """Get the robot loader registry. + + Maps robot_name → loader function that returns (meshes, ee_home_pose). + """ + return { + "google_robot": _load_google_robot, + "franka_panda": _load_franka_panda, + "widowx": _load_widowx, + "ur5e": _load_ur5e, + "embodiment_c": _load_embodiment_c, + } + + +def extract_gripper_openings(unified_57d: np.ndarray, robot_name: str = "google_robot") -> np.ndarray: + """Extract gripper opening fractions from unified action grasp state. + + Uses fingertip spread (f0-f1 distance) to invert the FK and recover + the scalar gripper opening at each timestep. + + Args: + unified_57d: (T, 57) unified action. + robot_name: Robot identifier for FK lookup table. + + Returns: + (T+1,) array of gripper openings in [0, 1]. + """ + T = unified_57d.shape[0] + grasp = unified_57d[:, 18:33].reshape(T, 5, 3) + all_grasp = np.concatenate([grasp[0:1], grasp], axis=0) + + # Build monotonic inverse lookup from FK (robot-specific) + _gs = np.linspace(0, 1, 10001).astype(np.float32) + try: + if robot_name == "franka_panda": + from cosmos_framework.data.vfm.action.robot_descriptions.franka import franka_fingertip_fk + + _tips = franka_fingertip_fk(_gs) + elif robot_name == "widowx": + from cosmos_framework.data.vfm.action.robot_descriptions.widowx import widowx_fingertip_fk + + _tips = widowx_fingertip_fk(_gs) + elif robot_name == "ur5e": + from cosmos_framework.data.vfm.action.robot_descriptions.umi import _WSG50_MAX_WIDTH, umi_fingertip_fk + + _tips = umi_fingertip_fk(_gs * _WSG50_MAX_WIDTH) + else: + from cosmos_framework.data.vfm.action.robot_descriptions.google_robot import ( + google_robot_fingertip_fk_vectorized, + ) + + _tips = google_robot_fingertip_fk_vectorized(_gs) + _spreads = np.linalg.norm(_tips[:, 0] - _tips[:, 1], axis=1) + except ImportError: + # Fallback: linear approximation + _spreads = _gs * 0.145 + _min_idx = int(np.argmin(_spreads)) + mono_gs = _gs[_min_idx:] + mono_spreads = _spreads[_min_idx:] + + openings = np.zeros(T + 1, dtype=np.float32) + for t in range(T + 1): + f0, f1 = all_grasp[t, 0], all_grasp[t, 1] + spread = np.linalg.norm(f0 - f1) + if spread <= mono_spreads[0]: + openings[t] = 0.0 + elif spread >= mono_spreads[-1]: + openings[t] = 1.0 + else: + openings[t] = float(np.interp(spread, mono_spreads, mono_gs)) + return openings diff --git a/cosmos_framework/data/vfm/action/urdf_visualizer/viewer.py b/cosmos_framework/data/vfm/action/urdf_visualizer/viewer.py new file mode 100755 index 0000000..170852f --- /dev/null +++ b/cosmos_framework/data/vfm/action/urdf_visualizer/viewer.py @@ -0,0 +1,1039 @@ +#!/usr/bin/env python +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Interactive 3D viewer for robot action datasets. + +Uses the unified 57D action representation: every dataset declares one explicit +raw ``ActionFormat`` (9D/10D/20D/57D), which is converted to +``UnifiedAction(action_57d, mask)`` before rendering. + +**57D layout**: ``[ego(9) | R_wrist(9) | R_fingers(15) | L_wrist(9) | L_fingers(15)]`` + +Dependencies:: + + pip install viser mujoco pin + +Usage: + # Use each dataset's declared raw action format: + uv run python cosmos_framework/data/vfm/action/urdf_visualizer/viewer.py --share + + # Override the raw action format explicitly: + uv run python cosmos_framework/data/vfm/action/urdf_visualizer/viewer.py --action-format 57d --share +""" + +from __future__ import annotations + +import argparse +import importlib +import os +import random +import sys +import time as _time +from dataclasses import dataclass, field +from functools import lru_cache +from pathlib import Path +from typing import Any, cast + +import numpy as np +import torch + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.action.urdf_visualizer.unified_action import ActionFormat + +_REPO_ROOT = str(Path(__file__).resolve().parents[6]) +if _REPO_ROOT not in sys.path: + sys.path.insert(0, _REPO_ROOT) + +# ── Dataset Registry ────────────────────────────────────────────────────────── + + +@dataclass +class DatasetEntry: + """Metadata for a dataset available in the viewer.""" + + name: str + robot_name: str + max_finger_width: float + fps: int + pose_convention: str = "backward_framewise" + camera_fov_deg: float = 60.0 + camera_aspect: float = 4 / 3 + dataset_class: str = "" + dataset_kwargs: dict[str, Any] = field(default_factory=dict) + action_format: ActionFormat = ActionFormat.SINGLE_ARM_10D + dual_base_left: np.ndarray | None = None + dual_base_right: np.ndarray | None = None + robot_embodiment_type: str | None = None + to_unified_fn: str | None = None # "module.path:function" for custom action conversion + to_opencv: np.ndarray | dict[str, np.ndarray] | None = None # native EE → OpenCV rotation + + +def _lazycfg_to_entry( + cfg: Any, + *, + robot_name: str = "", + max_finger_width: float = 0.0, + fps: int = 10, + action_format: ActionFormat = ActionFormat.SINGLE_ARM_10D, + camera_fov_deg: float = 60.0, + camera_aspect: float = 4 / 3, + dual_base_left: np.ndarray | None = None, + dual_base_right: np.ndarray | None = None, + robot_embodiment_type: str | None = None, + to_unified_fn: str | None = None, + to_opencv: np.ndarray | dict[str, np.ndarray] | None = None, + viewer_overrides: dict[str, Any] | None = None, +) -> DatasetEntry: + """Build a viewer dataset entry from a v1p2 ``LazyCall(dataset_entry)`` config.""" + + from omegaconf import OmegaConf + + from cosmos_framework.utils.lazy_config.registry import convert_target_to_string + + ds_cfg = cfg.dataset + target = ds_cfg._target_ + dataset_class = target if isinstance(target, str) else convert_target_to_string(target) + dataset_kwargs = { + key: value for key, value in OmegaConf.to_container(ds_cfg, resolve=False).items() if key != "_target_" + } + dataset_kwargs["action_normalization"] = None + if viewer_overrides is not None: + dataset_kwargs.update(viewer_overrides) + + pose_convention = str(dataset_kwargs.get("pose_convention", "backward_framewise")) + cfg_dict = OmegaConf.to_container(cfg, resolve=False) + dataset_name = str(cfg_dict.get("name", "unknown")) if isinstance(cfg_dict, dict) else "unknown" + return DatasetEntry( + name=dataset_name, + robot_name=robot_name, + max_finger_width=max_finger_width, + fps=fps, + pose_convention=pose_convention, + camera_fov_deg=camera_fov_deg, + camera_aspect=camera_aspect, + dataset_class=dataset_class, + dataset_kwargs=dataset_kwargs, + action_format=action_format, + dual_base_left=dual_base_left, + dual_base_right=dual_base_right, + robot_embodiment_type=robot_embodiment_type, + to_unified_fn=to_unified_fn, + to_opencv=to_opencv, + ) + + +def _build_datasets() -> dict[str, DatasetEntry]: + """Build the viewer dataset registry from ``action_datasets_v1p2``.""" + + from cosmos_framework.configs.base.experiment.action.midtrain_config.action_datasets_v1p2 import ( + DATASET_EMBODIMENT_C_GRIPPER_480, + DATASET_EMBODIMENT_C_GRIPPER_EXT_480, + DATASET_AGIBOTWORLD_BETA_480, + DATASET_AV_480, + DATASET_BRIDGE_480, + DATASET_CAMERA_480, + DATASET_DROID_480, + DATASET_FRACTAL_256, + DATASET_HAND_POSE_480, + DATASET_ROBOMIND_FRANKA_480, + DATASET_ROBOMIND_FRANKA_DUAL_480, + DATASET_ROBOMIND_UR_480, + DATASET_UMI_256, + ) + + raw_action_override = {"action_normalization": None} + credential_override = {"credential_path": "credentials/gcp_training.secret"} + agibot_unifier = ( + "cosmos_framework.data.vfm.action.urdf_visualizer.unified_action:to_unified_from_embodiment_c_fk_action" + ) + + from cosmos_framework.data.vfm.action.embodiment_c_fk import AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST + from cosmos_framework.data.vfm.action.bridge_orig_lerobot_dataset import _BRIDGE_TO_OPENCV + from cosmos_framework.data.vfm.action.droid_lerobot_dataset import _DROID_TO_OPENCV + from cosmos_framework.data.vfm.action.fractal import _GOOGLE_ROBOT_TO_OPENCV + from cosmos_framework.data.vfm.action.robomind_franka_dataset import _ROBOMIND_FRANKA_TO_OPENCV + + _FRANKA_TO_OPENCV = _ROBOMIND_FRANKA_TO_OPENCV[:3, :3] + + return { + "fractal": _lazycfg_to_entry( + DATASET_FRACTAL_256, + robot_name="google_robot", + max_finger_width=0.05, + fps=3, + action_format=ActionFormat.SINGLE_ARM_10D, + camera_fov_deg=69.0, + camera_aspect=320 / 256, + to_opencv=_GOOGLE_ROBOT_TO_OPENCV, + viewer_overrides=raw_action_override, + ), + "droid": _lazycfg_to_entry( + DATASET_DROID_480, + robot_name="franka_panda", + max_finger_width=0.08, + fps=15, + action_format=ActionFormat.SINGLE_ARM_10D, + to_opencv=_DROID_TO_OPENCV, + viewer_overrides=raw_action_override, + ), + "bridge": _lazycfg_to_entry( + DATASET_BRIDGE_480, + robot_name="widowx", + max_finger_width=0.06, + fps=5, + action_format=ActionFormat.SINGLE_ARM_10D, + to_opencv=_BRIDGE_TO_OPENCV, + viewer_overrides=raw_action_override, + ), + "robomind_franka": _lazycfg_to_entry( + DATASET_ROBOMIND_FRANKA_480, + robot_name="franka_panda", + max_finger_width=0.08, + fps=10, + action_format=ActionFormat.SINGLE_ARM_10D, + to_opencv=_FRANKA_TO_OPENCV, + viewer_overrides=raw_action_override, + ), + "robomind_franka_dual": _lazycfg_to_entry( + DATASET_ROBOMIND_FRANKA_DUAL_480, + robot_name="franka_panda", + max_finger_width=0.08, + fps=10, + action_format=ActionFormat.DUAL_ARM_20D, + dual_base_left=np.array( + [[1, 0, 0, 0.0], [0, 1, 0, 0.3], [0, 0, 1, 0.0], [0, 0, 0, 1.0]], + dtype=np.float32, + ), + dual_base_right=np.array( + [[1, 0, 0, 0.0], [0, 1, 0, -0.3], [0, 0, 1, 0.0], [0, 0, 0, 1.0]], + dtype=np.float32, + ), + to_opencv=_FRANKA_TO_OPENCV, + viewer_overrides=raw_action_override, + ), + "robomind_ur": _lazycfg_to_entry( + DATASET_ROBOMIND_UR_480, + robot_name="ur5e", + max_finger_width=0.085, + fps=10, + action_format=ActionFormat.SINGLE_ARM_10D, + to_opencv=np.eye(3, dtype=np.float32), + viewer_overrides=raw_action_override, + ), + "umi": _lazycfg_to_entry( + DATASET_UMI_256, + robot_name="", + max_finger_width=0.08, + fps=10, + action_format=ActionFormat.SINGLE_ARM_10D, + viewer_overrides={**raw_action_override, "normalizer_dir": ""}, + ), + "camera": _lazycfg_to_entry( + DATASET_CAMERA_480, + robot_name="", + max_finger_width=0.0, + fps=10, + action_format=ActionFormat.EGO_9D, + viewer_overrides={**raw_action_override, **credential_override}, + ), + "av": _lazycfg_to_entry( + DATASET_AV_480, + robot_name="", + max_finger_width=0.0, + fps=10, + action_format=ActionFormat.EGO_9D, + viewer_overrides={**raw_action_override, **credential_override}, + ), + "hand_pose": _lazycfg_to_entry( + DATASET_HAND_POSE_480, + robot_name="", + max_finger_width=0.0, + fps=15, + action_format=ActionFormat.UNIFIED_57D, + viewer_overrides={**raw_action_override, "return_overlay_data": True}, + ), + "embodiment_c_gripper": _lazycfg_to_entry( + DATASET_EMBODIMENT_C_GRIPPER_480, + robot_name="embodiment_c", + max_finger_width=0.12, + fps=10, + camera_fov_deg=69.0, + camera_aspect=640 / 480, + viewer_overrides={**raw_action_override, "return_agibot_link_poses": True}, + robot_embodiment_type="embodiment_c_gripper", + to_unified_fn=agibot_unifier, + to_opencv=AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST, + ), + "embodiment_c_gripper_ext": _lazycfg_to_entry( + DATASET_EMBODIMENT_C_GRIPPER_EXT_480, + robot_name="embodiment_c", + max_finger_width=0.12, + fps=10, + camera_fov_deg=69.0, + camera_aspect=640 / 480, + viewer_overrides={**raw_action_override, "return_agibot_link_poses": True}, + robot_embodiment_type="embodiment_c_gripper_ext", + to_unified_fn=agibot_unifier, + to_opencv=AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST, + ), + "agibotworld_beta": _lazycfg_to_entry( + DATASET_AGIBOTWORLD_BETA_480, + robot_name="embodiment_c", + max_finger_width=0.12, + fps=10, + camera_fov_deg=69.0, + camera_aspect=640 / 480, + viewer_overrides={**raw_action_override, "return_agibot_link_poses": True}, + robot_embodiment_type="embodiment_c_gripper", + to_unified_fn=agibot_unifier, + to_opencv=AGIBOT_GEAR_GRIPPER_TO_OPENCV_BY_WRIST, + ), + } + + +DATASETS: dict[str, DatasetEntry] = {} + + +# ── Dataset Creation ────────────────────────────────────────────────────────── + + +def _create_dataset(entry: DatasetEntry, chunk_length: int): + """Instantiate a dataset class for the given entry.""" + import importlib + import inspect + + module_path, class_name = entry.dataset_class.rsplit(".", 1) + mod = importlib.import_module(module_path) + cls = getattr(mod, class_name) + + kwargs = dict(entry.dataset_kwargs) + kwargs["chunk_length"] = chunk_length + kwargs["split"] = "full" + kwargs["mode"] = "policy" + kwargs["enable_fast_init"] = True + + # UMI: factory function + if callable(cls) and not inspect.isclass(cls): + _OMEGACONF_BLOCKLIST = {"chunk_length", "split", "action_normalization", "enable_fast_init"} + kwargs = {k: v for k, v in kwargs.items() if k not in _OMEGACONF_BLOCKLIST} + kwargs["eager_load"] = True + return cls(**kwargs) + + sig = inspect.signature(cls.__init__) + valid_params = set(sig.parameters.keys()) - {"self"} + has_var_keyword = any(p.kind == inspect.Parameter.VAR_KEYWORD for p in sig.parameters.values()) + if not has_var_keyword: + kwargs = {k: v for k, v in kwargs.items() if k in valid_params} + + dataset = cls(**kwargs) + + if hasattr(dataset, "_register_sources"): + dataset._register_sources() + if hasattr(dataset, "__len__") and len(dataset) == 0: + raise RuntimeError(f"{entry.name}: registered sources but found no valid samples") + + from torch.utils.data import IterableDataset as _IterableBase + + if isinstance(dataset, _IterableBase): + dataset = _IterableToMapDataset(dataset) + + return dataset + + +def _resolve_action_format( + entry: DatasetEntry, + action_format_override: ActionFormat | None, +) -> ActionFormat: + """Return the explicit raw action format for one dataset load.""" + return action_format_override if action_format_override is not None else entry.action_format + + +@lru_cache(maxsize=None) +def _load_symbol(target: str): + """Import and cache one ``module:symbol`` reference.""" + + module_name, symbol_name = target.split(":", maxsplit=1) + module = importlib.import_module(module_name) + return getattr(module, symbol_name) + + +def _format_sample_text(value: Any, max_chars: int | None = None) -> str: + """Format optional sample text for the viewer info panel.""" + + if value is None: + return "" + text = value if isinstance(value, str) else str(value) + if not text: + return "" + if max_chars is None: + return text + return text[:max_chars] + + +def _build_viewer_idle_action_spec(action_format: ActionFormat) -> Any: + """Build a fallback idle-frame spec from the viewer-declared action format.""" + + from cosmos_framework.data.vfm.action.action_spec import Gripper, Pos, Rot, build_action_spec + + if action_format is ActionFormat.EGO_9D: + return build_action_spec(Pos(prefix="ego"), Rot("rot6d", prefix="ego")) + if action_format is ActionFormat.SINGLE_ARM_10D: + return build_action_spec(Pos(), Rot("rot6d"), Gripper()) + if action_format is ActionFormat.DUAL_ARM_20D: + return build_action_spec( + Pos(prefix="left"), + Rot("rot6d", prefix="left"), + Gripper(prefix="left"), + Pos(prefix="right"), + Rot("rot6d", prefix="right"), + Gripper(prefix="right"), + ) + if action_format is ActionFormat.UNIFIED_57D: + return build_action_spec( + Pos(prefix="ego"), + Rot("rot6d", prefix="ego"), + Pos(prefix="right_wrist"), + Rot("rot6d", prefix="right_wrist"), + Pos(dim=15, prefix="right_fingers"), + Pos(prefix="left_wrist"), + Rot("rot6d", prefix="left_wrist"), + Pos(dim=15, prefix="left_fingers"), + ) + raise ValueError(f"Unsupported action format for idle-frame detection: {action_format}") + + +def _compute_viewer_idle_frames( + action: Any, + dataset: Any, + action_format: ActionFormat, +) -> torch.Tensor | None: + """Compute idle frames for a viewer sample when the dataset did not provide them.""" + + action_spec = getattr(dataset, "action_spec", None) + compute_idle_frames_method = getattr(dataset, "_compute_idle_frames", None) + if action_spec is not None and compute_idle_frames_method is not None: + return compute_idle_frames_method(action) + + from cosmos_framework.data.vfm.action.pose_utils import compute_idle_frames + + spec = _build_viewer_idle_action_spec(action_format) + try: + idle_frames = compute_idle_frames(action, spec) + except (TypeError, ValueError) as error: + log.warning(f"Viewer idle-frame detection skipped for {action_format.value}: {error}") + return None + return torch.tensor(idle_frames, dtype=torch.long) # [] + + +@lru_cache(maxsize=1) +def _get_viewer_idle_frames_augmentor() -> Any: + """Return the caption augmentor used by the viewer idle-frame path.""" + + from cosmos_framework.data.vfm.augmentors.idle_frames_text_info import IdleFramesTextInfo + + return IdleFramesTextInfo( + input_keys=["ai_caption", "idle_frames", "action"], + output_keys=["ai_caption"], + args={ + "caption_key": "ai_caption", + "idle_frames_key": "idle_frames", + "action_key": "action", + "dropout_rate": 0.0, + "enabled": True, + }, + ) + + +def _enable_viewer_idle_frames(sample: dict[str, Any], dataset: Any, action_format: ActionFormat) -> dict[str, Any]: + """Populate idle-frame metadata and append text in the direct viewer data path.""" + + updated_sample = sample + idle_frames = updated_sample.get("idle_frames") + action = updated_sample.get("action") + if idle_frames is None and action is not None: + idle_frames = _compute_viewer_idle_frames(action, dataset, action_format) + if idle_frames is not None: + updated_sample = dict(updated_sample) + updated_sample["idle_frames"] = idle_frames + + if idle_frames is None: + return updated_sample + + updated_sample = dict(updated_sample) + caption = updated_sample.get("ai_caption") + if isinstance(caption, dict): + updated_sample["ai_caption"] = dict(caption) + + augmented_sample = _get_viewer_idle_frames_augmentor()(updated_sample) + return updated_sample if augmented_sample is None else augmented_sample + + +class _IterableToMapDataset: + """Wraps an IterableDataset into a random-access dataset with lazy loading.""" + + _MAX_CACHE = 200 + + def __init__(self, iterable_dataset, max_samples: int | None = None): + self._iter = iter(iterable_dataset) + self._samples: list[dict] = [] + self._exhausted = False + self._max = max_samples or self._MAX_CACHE + self._ds_name = iterable_dataset.__class__.__name__ + log.info(f"Lazy wrapper: {self._ds_name} (max {self._max})") + + def _fetch_up_to(self, idx: int) -> bool: + while len(self._samples) <= idx and not self._exhausted and len(self._samples) < self._max: + try: + self._samples.append(next(self._iter)) + log.info(f"{self._ds_name}: fetched sample {len(self._samples) - 1}") + except StopIteration: + self._exhausted = True + log.info(f"{self._ds_name}: exhausted at {len(self._samples)}") + return idx < len(self._samples) + + def __len__(self): + return max(len(self._samples), 1) + + def __getitem__(self, idx): + if self._fetch_up_to(idx): + return self._samples[idx] + if self._samples: + return self._samples[idx % len(self._samples)] + raise IndexError(f"{self._ds_name}: no samples available") + + +# ── Viewer ──────────────────────────────────────────────────────────────────── + + +def _collect_scene_points(state) -> np.ndarray: + """Collect all visible trajectory positions for camera fitting.""" + points: list[np.ndarray] = [] + for poses in (state.ego_poses, state.right_poses, state.left_poses): + if poses is not None and len(poses) > 0: + points.append(poses[:, :3, 3].astype(np.float32)) + if not points: + return np.zeros((1, 3), dtype=np.float32) + return np.concatenate(points, axis=0) + + +def _get_observation_up_direction(state, view_forward: np.ndarray) -> np.ndarray: + """Estimate a stable viewer up-direction from the observation camera poses.""" + if state.ego_poses is None or len(state.ego_poses) == 0: + return np.array([0.0, 0.0, 1.0], dtype=np.float32) + + # Ego poses are camera-to-world transforms in OpenCV camera convention, + # where image up corresponds to the negative camera Y axis. + camera_up = -state.ego_poses[:, :3, 1].astype(np.float32) + reference = camera_up[0] + aligned_up = camera_up.copy() + for idx in range(1, len(aligned_up)): + if float(np.dot(aligned_up[idx], reference)) < 0.0: + aligned_up[idx] *= -1.0 + + up_direction = aligned_up.mean(axis=0) + up_direction -= view_forward * float(np.dot(up_direction, view_forward)) + up_norm = float(np.linalg.norm(up_direction)) + if up_norm < 1e-6: + return np.array([0.0, 0.0, 1.0], dtype=np.float32) + return up_direction / up_norm + + +def _get_observation_forward_direction(state) -> np.ndarray | None: + """Estimate a stable viewer forward direction from the observation camera poses.""" + if state.ego_poses is None or len(state.ego_poses) == 0: + return None + + # Ego poses are camera-to-world transforms in OpenCV camera convention, + # where the optical axis points along the positive camera Z axis. + camera_forward = state.ego_poses[:, :3, 2].astype(np.float32) + reference = camera_forward[0] + aligned_forward = camera_forward.copy() + for idx in range(1, len(aligned_forward)): + if float(np.dot(aligned_forward[idx], reference)) < 0.0: + aligned_forward[idx] *= -1.0 + + forward_direction = aligned_forward.mean(axis=0) + forward_norm = float(np.linalg.norm(forward_direction)) + if forward_norm < 1e-6: + return None + return forward_direction / forward_norm + + +def _reset_camera_to_trajectory(client, state, camera_fov_deg: float) -> None: + """Frame one client's viewport using the current trajectory extent.""" + points = _collect_scene_points(state) + center = points.mean(axis=0) + extent = points - center[None, :] + radius = float(np.linalg.norm(extent, axis=1).max()) if len(points) > 0 else 0.0 + radius = max(radius, 0.15) + + fov_rad = float(np.deg2rad(camera_fov_deg)) + fit_distance = radius / max(np.tan(fov_rad / 2.0), 0.35) + view_forward = _get_observation_forward_direction(state) + if view_forward is None: + view_dir = np.array([1.0, -1.0, 0.7], dtype=np.float32) + view_dir /= np.linalg.norm(view_dir) + camera_position = center + view_dir * max(fit_distance * 1.35, 0.5) + view_forward = center - camera_position + view_forward /= np.linalg.norm(view_forward) + else: + camera_position = center - view_forward * max(fit_distance * 1.35, 0.5) + view_forward = center - camera_position + view_forward /= np.linalg.norm(view_forward) + up_direction = _get_observation_up_direction(state, view_forward) + camera = client.camera + + # Camera state arrives asynchronously from the browser. Wait briefly so we can + # update only this client's viewport instead of broadcasting a global reset target. + deadline = _time.time() + 1.0 + while getattr(camera._state, "update_timestamp", 0.0) == 0.0 and _time.time() < deadline: + _time.sleep(0.01) + if getattr(camera._state, "update_timestamp", 0.0) == 0.0: + return + + camera.fov = fov_rad + camera.up_direction = tuple(up_direction.tolist()) + camera.look_at = tuple(center.tolist()) + # Setting position also translates look_at, so restore the target afterwards. + camera.position = tuple(camera_position.tolist()) + camera.look_at = tuple(center.tolist()) + client.flush() + + +def launch_viewer( + port: int = 8013, + share: bool = False, + chunk_length: int = 16, + action_format_override: ActionFormat | None = None, +) -> None: + """Launch the interactive dataset viewer.""" + global DATASETS + import threading as _threading + + import viser + + from cosmos_framework.data.vfm.action.urdf_visualizer.unified_action import ( + build_scene_state, + get_video_from_sample, + to_unified, + ) + from cosmos_framework.data.vfm.action.urdf_visualizer.unified_renderer import UnifiedRenderer + + server = viser.ViserServer(host="0.0.0.0", port=port) + if not DATASETS: + DATASETS = _build_datasets() + datasets = DATASETS + dataset_cache: dict[str, Any] = {} + dataset_locks: dict[str, Any] = {} + dataset_cache_lock = _threading.Lock() + sessions_lock = _threading.Lock() + + def _get_dataset_lock(cache_key: str) -> Any: + """Return the per-dataset lock for a cache key.""" + with dataset_cache_lock: + lock = dataset_locks.get(cache_key) + if lock is None: + lock = _threading.Lock() + dataset_locks[cache_key] = lock + return lock + + @dataclass + class ViewerSession: + client: Any + renderer: Any + time_slider: Any + speed_slider: Any + play_button: Any + action_text: Any + show: dict[str, bool | float] + load_lock: Any = field(default_factory=_threading.Lock) + playing: bool = False + last_frame_time: float = 0.0 + + sessions: dict[int, ViewerSession] = {} + + @server.on_client_connect + def _(client) -> None: + client.scene.reset() + client.scene.set_up_direction("+z") + client.gui.reset() + + renderer = UnifiedRenderer(client) + + with client.gui.add_folder("Dataset"): + ds_dropdown = client.gui.add_dropdown( + "Dataset", options=list(datasets.keys()), initial_value=list(datasets.keys())[0] + ) + ep_input = client.gui.add_number("Episode", initial_value=0, min=0, step=1) + random_button = client.gui.add_button("🎲 Random episode") + status_text = client.gui.add_markdown("*Ready*") + info_text = client.gui.add_markdown("") + + with client.gui.add_folder("Display", expand_by_default=False): + show_robot = client.gui.add_checkbox("Show robot mesh", initial_value=True) + show_frames = client.gui.add_checkbox("Show wrist frames", initial_value=True) + show_traj = client.gui.add_checkbox("Show trajectory", initial_value=True) + show_fingertips = client.gui.add_checkbox("Show fingertips", initial_value=True) + show_ego = client.gui.add_checkbox("Show ego camera", initial_value=True) + axis_scale = client.gui.add_slider("Axis scale", min=0.1, max=20.0, step=0.1, initial_value=1.0) + + robot_frame_toggle_handles: dict[str, Any] = {} + robot_frame_toggle_folder = client.gui.add_folder("Robot Frame Toggles", expand_by_default=False) + with robot_frame_toggle_folder: + robot_frame_toggle_status = client.gui.add_markdown("*Load an episode to choose robot frame coordinates.*") + + with client.gui.add_folder("Playback"): + time_slider = client.gui.add_slider("Time", min=0, max=1, step=1, initial_value=0) + play_button = client.gui.add_button("▶ Play") + speed_slider = client.gui.add_slider("Speed (fps)", min=1, max=30, step=1, initial_value=3) + + cam_panel = client.gui.add_image(np.zeros((64, 64, 3), dtype=np.uint8)) + renderer.set_video_panel(cam_panel) + + with client.gui.add_folder("Action (57D)"): + action_text = client.gui.add_markdown("*No episode loaded*") + + show = { + "frames": True, + "traj": True, + "fingertips": True, + "ego": True, + "robot": True, + "robot_frame_filters": {}, + "axis_scale": 1.0, + } + session = ViewerSession( + client=client, + renderer=renderer, + time_slider=time_slider, + speed_slider=speed_slider, + play_button=play_button, + action_text=action_text, + show=show, + ) + + def _update_action_text(t: int) -> None: + """Update the 57D action display for one client.""" + txt = renderer.format_action_text(t) + action_text.content = txt if txt else "*No data*" + + def _clear_robot_frame_toggles() -> None: + """Remove the dynamic per-frame toggle controls.""" + for handle in robot_frame_toggle_handles.values(): + handle.remove() + robot_frame_toggle_handles.clear() + + def _rebuild_robot_frame_toggles() -> None: + """Rebuild the GUI toggles for the currently loaded robot frames.""" + _clear_robot_frame_toggles() + selectors = renderer.get_robot_frame_selectors() + if not selectors: + show["robot_frame_filters"] = {} + robot_frame_toggle_status.content = "*No robot frame coordinates available for this episode.*" + return + + prev_filters = cast(dict[str, bool], show.get("robot_frame_filters", {})) + show["robot_frame_filters"] = { + selector_key: prev_filters.get(selector_key, False) for selector_key, _ in selectors + } + robot_frame_toggle_status.content = "*Choose which robot frame coordinates to show.*" + + with robot_frame_toggle_folder: + for selector_key, label in selectors: + checkbox = client.gui.add_checkbox( + label, + initial_value=cast(dict[str, bool], show["robot_frame_filters"])[selector_key], + ) + robot_frame_toggle_handles[selector_key] = checkbox + + @checkbox.on_update + def _(_, selector_key: str = selector_key, checkbox: Any = checkbox) -> None: + cast(dict[str, bool], show["robot_frame_filters"])[selector_key] = bool(checkbox.value) + renderer.update(time_slider.value, show) + + def do_load_episode() -> None: + t_start = _time.time() + ds_name = ds_dropdown.value + entry = datasets[ds_name] + effective_action_format = _resolve_action_format(entry, action_format_override) + ep_idx = max(int(ep_input.value), 0) + cache_key = ds_name + + status_text.content = f"⏳ Loading {ds_name} episode {ep_idx}..." + + try: + with _get_dataset_lock(cache_key): + dataset: Any + with dataset_cache_lock: + dataset = cast(Any, dataset_cache.get(cache_key)) + if dataset is None: + status_text.content = f"⏳ Creating {ds_name} dataset..." + dataset = _create_dataset(entry, chunk_length) + with dataset_cache_lock: + dataset_cache[cache_key] = dataset + to_opencv = entry.to_opencv if entry.to_opencv is not None else np.eye(3, dtype=np.float32) + + n_total = len(dataset) + if ep_idx >= n_total: + if isinstance(dataset, _IterableToMapDataset) and not dataset._exhausted: + pass + else: + ep_idx = n_total - 1 + ep_input.value = ep_idx + + sample: Any = _enable_viewer_idle_frames(dataset[ep_idx], dataset, effective_action_format) + + action_tensor = sample["action"] + action_raw = ( + action_tensor.numpy() if isinstance(action_tensor, torch.Tensor) else np.asarray(action_tensor) + ) + + uses_dual_initial_pose = effective_action_format is ActionFormat.DUAL_ARM_20D + + initial_pose_t = sample.get("initial_pose") + if initial_pose_t is None: + initial_pose = np.eye(4, dtype=np.float32) + elif isinstance(initial_pose_t, torch.Tensor): + initial_pose = initial_pose_t.numpy().astype(np.float32) + else: + initial_pose = np.asarray(initial_pose_t, dtype=np.float32) + + initial_pose_right_t = sample.get("initial_pose_right") + initial_pose_left_t = sample.get("initial_pose_left") + initial_pose_right = None + initial_pose_left = None + if initial_pose_right_t is not None: + initial_pose_right = ( + initial_pose_right_t.numpy().astype(np.float32) + if isinstance(initial_pose_right_t, torch.Tensor) + else np.asarray(initial_pose_right_t, dtype=np.float32) + ) + if initial_pose_left_t is not None: + initial_pose_left = ( + initial_pose_left_t.numpy().astype(np.float32) + if isinstance(initial_pose_left_t, torch.Tensor) + else np.asarray(initial_pose_left_t, dtype=np.float32) + ) + if uses_dual_initial_pose: + if initial_pose_left is None: + initial_pose_left = initial_pose + + if entry.to_unified_fn: + converter = _load_symbol(entry.to_unified_fn) + import inspect as _inspect + + params = _inspect.signature(converter).parameters + embodiment_type = entry.robot_embodiment_type or str( + entry.dataset_kwargs.get("embodiment_type", "") + ) + if "embodiment_type" in params: + unified = converter(sample, embodiment_type=embodiment_type) + elif "kind" in params: + unified = converter(action_raw, kind="gripper") + else: + unified = converter(action_raw) + raw_action_label = "custom" + else: + unified = to_unified(action_raw, action_format=effective_action_format) + raw_action_label = effective_action_format.value + state = build_scene_state( + unified, + initial_pose=initial_pose, + initial_pose_right=initial_pose_right, + initial_pose_left=initial_pose_left, + right_base_pose=entry.dual_base_right, + left_base_pose=entry.dual_base_left, + pose_convention=entry.pose_convention, + sample=sample, + ) + state.video = get_video_from_sample(sample) + + # Inject FK joint configs when the dataset provides them (e.g. UR). + jc = sample.get("joint_configs") + if jc is not None: + state.joint_configs = ( + jc.numpy().astype(np.float32) + if isinstance(jc, torch.Tensor) + else np.asarray(jc, dtype=np.float32) + ) + status_text.content = "⏳ Loading robot animation..." + renderer.load(state, entry, to_opencv=to_opencv) + _rebuild_robot_frame_toggles() + _reset_camera_to_trajectory(client, state, entry.camera_fov_deg) + + T = state.T + time_slider.max = max(T, 1) + time_slider.value = 0 + + ai_caption_text = _format_sample_text(sample.get("ai_caption", ""), max_chars=160) + debug_caption_text = _format_sample_text(sample.get("debug_caption", "")) + t_total = _time.time() - t_start + info_text.content = ( + f"**{ds_name.upper()}** — Episode {ep_idx}\n\n" + + (f"Task: {ai_caption_text}\n\n" if ai_caption_text else "") + + (f"Debug: {debug_caption_text}\n\n" if debug_caption_text else "") + + ( + f"Steps: {T} | Raw: {raw_action_label} ({action_raw.shape[-1]}D) → 57D | " + f"Robot: {entry.robot_name or '—'} | FPS: {entry.fps}" + ) + ) + status_text.content = f"✅ Loaded in {t_total:.1f}s" + log.info(f"Loaded {ds_name} ep {ep_idx}: {ai_caption_text[:60]}, {T} steps, {t_total:.1f}s") + + renderer.update(0, show) + renderer.update_axis_scale(axis_scale.value) + _update_action_text(0) + session.last_frame_time = _time.time() + + except Exception as e: + status_text.content = f"❌ Load failed: {e}" + log.error(f"Load failed for client {client.client_id}: {e}") + import traceback + + traceback.print_exc() + + def _do_load_threaded() -> None: + if not session.load_lock.acquire(blocking=False): + return + + def _run() -> None: + try: + do_load_episode() + finally: + session.load_lock.release() + + _threading.Thread(target=_run, daemon=True).start() + + @ds_dropdown.on_update + def _(_) -> None: + _do_load_threaded() + + @ep_input.on_update + def _(_) -> None: + _do_load_threaded() + + @random_button.on_click + def _(_) -> None: + ds_name = ds_dropdown.value + cache_key = ds_name + with _get_dataset_lock(cache_key): + with dataset_cache_lock: + ds = dataset_cache.get(cache_key) + if ds is None: + ep_input.value = 0 + elif isinstance(ds, _IterableToMapDataset): + ep_input.value = len(ds._samples) + else: + ep_input.value = random.randint(0, max(len(ds) - 1, 0)) + _do_load_threaded() + + @time_slider.on_update + def _(_) -> None: + renderer.update(time_slider.value, show) + _update_action_text(time_slider.value) + + @show_robot.on_update + def _(_) -> None: + show["robot"] = show_robot.value + renderer.update(time_slider.value, show) + + @show_frames.on_update + def _(_) -> None: + show["frames"] = show_frames.value + renderer.update(time_slider.value, show) + + @show_traj.on_update + def _(_) -> None: + show["traj"] = show_traj.value + renderer.update(time_slider.value, show) + + @show_fingertips.on_update + def _(_) -> None: + show["fingertips"] = show_fingertips.value + renderer.update(time_slider.value, show) + + @show_ego.on_update + def _(_) -> None: + show["ego"] = show_ego.value + renderer.update(time_slider.value, show) + + @axis_scale.on_update + def _(_) -> None: + show["axis_scale"] = axis_scale.value + renderer.update_axis_scale(axis_scale.value) + + @play_button.on_click + def _(_) -> None: + session.playing = not session.playing + session.last_frame_time = _time.time() + play_button.label = "⏸ Pause" if session.playing else "▶ Play" + + with sessions_lock: + sessions[client.client_id] = session + _do_load_threaded() + + @server.on_client_disconnect + def _(client) -> None: + with sessions_lock: + sessions.pop(client.client_id, None) + + # ── Share URL ── + log.info(f"✅ Viewer ready at http://0.0.0.0:{port}") + if share: + share_url = server.request_share_url() + if share_url: + log.info(f"🌐 Share URL: {share_url}") + try: + with open(os.path.expanduser("~/share_url.txt"), "w") as f: + f.write(share_url + "\n") + except Exception: + pass + + # ── Main Loop ── + try: + while True: + now = _time.time() + with sessions_lock: + active_sessions = list(sessions.values()) + for session in active_sessions: + renderer = session.renderer + if not session.playing or renderer.state is None: + continue + frame_period = 1.0 / max(float(session.speed_slider.value), 1.0) + if now - session.last_frame_time < frame_period: + continue + t = session.time_slider.value + t = (t + 1) % max(renderer.state.T, 1) + session.time_slider.value = t + renderer.update(t, session.show) + txt = renderer.format_action_text(t) + session.action_text.content = txt if txt else "*No data*" + session.last_frame_time = now + _time.sleep(0.02) + except KeyboardInterrupt: + log.info("Shutting down.") + + +def main(): + parser = argparse.ArgumentParser(description="Action dataset viewer (unified 57D)") + parser.add_argument("--port", type=int, default=8013) + parser.add_argument("--share", action="store_true") + parser.add_argument("--chunk-length", type=int, default=16) + parser.add_argument( + "--action-format", + choices=[fmt.value for fmt in ActionFormat], + default=None, + help="Optional override for the dataset-declared raw action format", + ) + args = parser.parse_args() + launch_viewer( + port=args.port, + share=args.share, + chunk_length=args.chunk_length, + action_format_override=ActionFormat(args.action_format) if args.action_format is not None else None, + ) + + +if __name__ == "__main__": + main() diff --git a/cosmos_framework/data/vfm/action/viewpoint_utils.py b/cosmos_framework/data/vfm/action/viewpoint_utils.py new file mode 100644 index 0000000..f083e3d --- /dev/null +++ b/cosmos_framework/data/vfm/action/viewpoint_utils.py @@ -0,0 +1,114 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Viewpoint type definitions and caption augmentor for Action datasets. + +Provides a ``Viewpoint`` type alias for camera perspective labels and a +``ViewpointTextInfo`` augmentor that appends a human-readable viewpoint +description to the caption string. +""" + +from __future__ import annotations + +from typing import Literal + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log + +Viewpoint = Literal["ego_view", "third_person_view", "wrist_view", "concat_view"] + +DEFAULT_VIEWPOINT_TEMPLATES: dict[str, str] = { + "ego_view": "This video is captured from a first-person perspective looking at the scene.", + "third_person_view": "This video is captured from a third-person perspective looking towards the agent from the front.", + "wrist_view": "This video is captured from a wrist-mounted camera.", + "concat_view": "This video contains concatenated views from multiple camera perspectives.", +} + + +class ViewpointTextInfo(Augmentor): + """Augmentor that appends viewpoint type description to captions. + + Reads a viewpoint label from ``data_dict[viewpoint_key]`` and appends + the corresponding template sentence to the caption. Designed to run + after the raw ``ai_caption`` is set but before duration/FPS metadata + is appended. + + Args: + input_keys: Input keys (kept for API compatibility). + output_keys: Output keys (kept for API compatibility). + args: Configuration arguments: + - caption_key (str): Key for caption in data_dict. Default: ``"ai_caption"`` + - viewpoint_key (str): Key for viewpoint label. Default: ``"viewpoint"`` + - templates (dict): Override mapping from viewpoint to sentence. + Default: :data:`DEFAULT_VIEWPOINT_TEMPLATES` + - separator (str): Separator between caption and metadata. Default: ``". "`` + - enabled (bool): Whether augmentation is enabled. Default: ``True`` + """ + + def __init__( + self, + input_keys: list | None = None, + output_keys: list | None = None, + args: dict | None = None, + ) -> None: + super().__init__(input_keys or [], output_keys or [], args) + + self.caption_key: str = args.get("caption_key", "ai_caption") if args else "ai_caption" + self.viewpoint_key: str = args.get("viewpoint_key", "viewpoint") if args else "viewpoint" + self.templates: dict[str, str] = ( + args.get("templates", DEFAULT_VIEWPOINT_TEMPLATES) if args else DEFAULT_VIEWPOINT_TEMPLATES + ) + self.default_separator: str = args.get("separator", ". ") if args else ". " + self.enabled: bool = args.get("enabled", True) if args else True + + def __call__(self, data_dict: dict) -> dict | None: + """Append viewpoint description to the caption. + + If the sample provides an ``"additional_view_description"`` key (a + free-form string describing the concatenated camera layout), it is + appended after the generic ``concat_view`` template. This allows each + dataset to supply its own description of which cameras are tiled and + how. + + Args: + data_dict: Sample dictionary containing caption and viewpoint. + + Returns: + The mutated *data_dict*, or the original unchanged if the + viewpoint key is missing or unrecognized. + """ + if not self.enabled: + return data_dict + + viewpoint = data_dict.get(self.viewpoint_key) + if viewpoint is None: + raise ValueError( + f"ViewpointTextInfo: missing key {self.viewpoint_key!r} in data_dict. " + f"All action datasets must provide a viewpoint label." + ) + + # Append dataset-specific concat_view details after the base template. + additional_view_description = data_dict.pop("additional_view_description", None) + template = self.templates.get(viewpoint) + + if template is None: + log.warning( + f"ViewpointTextInfo: unrecognized viewpoint {viewpoint!r}. " + f"Known viewpoints: {sorted(self.templates.keys())}. Skipping.", + rank0_only=False, + ) + return data_dict + + if additional_view_description: + separator = " " if template.endswith(".") else self.default_separator + template = template + separator + additional_view_description.rstrip() + + caption = data_dict.get(self.caption_key) + if not isinstance(caption, str) or caption == "": + return data_dict + + caption = caption.rstrip() + separator = " " if caption.endswith(".") else self.default_separator + data_dict[self.caption_key] = caption + separator + template + + return data_dict diff --git a/cosmos_framework/data/vfm/action_scripts/__init__.py b/cosmos_framework/data/vfm/action_scripts/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/data/vfm/action_scripts/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/data/vfm/action_scripts/memprofile.py b/cosmos_framework/data/vfm/action_scripts/memprofile.py new file mode 100644 index 0000000..f00338b --- /dev/null +++ b/cosmos_framework/data/vfm/action_scripts/memprofile.py @@ -0,0 +1,254 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Lightweight CPU memory-profiling helpers. + +Only depends on ``os``, ``sys``, and ``psutil`` so it can be imported safely +from dataset modules without pulling in heavy dependencies. + +Enable per-stage logging by setting the ``MEMORY_PROFILE`` env var:: + + MEMORY_PROFILE=1 torchrun ... +""" + +import contextlib +import gc +import logging +import os +import sys +from collections.abc import Callable, Iterator + +import psutil + +_log = logging.getLogger(__name__) + + +def memprofile_enabled() -> bool: + """Return ``True`` when the ``MEMORY_PROFILE`` env var is truthy.""" + return os.environ.get("MEMORY_PROFILE", "").strip() not in ("", "0", "false") + + +def fmt_mb(mb: float) -> str: + """Format a MiB value as a human-readable string (MiB or GiB).""" + if mb >= 1024: + return f"{mb / 1024:.2f} GiB" + return f"{mb:.1f} MiB" + + +@contextlib.contextmanager +def rss_tracker( + label: str, + *, + enabled: bool | None = None, + extras_fn: Callable[[], list[str]] | None = None, + after_fn: Callable[[], None] | None = None, +) -> Iterator[None]: + """Track RSS delta across a block. No-op when profiling is disabled. + + When *enabled* is ``False`` (or ``None`` and ``MEMORY_PROFILE`` is unset) + the context manager yields immediately with zero overhead -- no + ``gc.collect()`` and no ``psutil`` calls. + + Args: + label: Human-readable description included in the log line. + enabled: Explicit toggle. When ``None``, falls back to + ``memprofile_enabled()`` (i.e. the ``MEMORY_PROFILE`` env var). + extras_fn: Optional callback invoked *after* the measured block. + Each returned string is logged as a supplementary detail line. + after_fn: Optional side-effect callback invoked after logging. + Use for actions that should only run when profiling is active + (e.g. detailed worker memory breakdowns). + """ + if enabled is None: + enabled = memprofile_enabled() + if not enabled: + yield + return + gc.collect() + rss_before = get_rss_mb() + yield + gc.collect() + rss_after = get_rss_mb() + _log.debug( + "[MEMPROFILE] %s | RSS: %s (delta: +%s)", + label, + fmt_mb(rss_after), + fmt_mb(rss_after - rss_before), + ) + if extras_fn is not None: + for line in extras_fn(): + _log.debug("[MEMPROFILE] %s", line) + if after_fn is not None: + after_fn() + + +def get_rss_mb() -> float: + """Return the current process RSS in MiB.""" + return psutil.Process(os.getpid()).memory_info().rss / (1024 * 1024) + + +def get_process_tree_rss_mb() -> float: + """Return RSS of the current process + all children in MiB.""" + proc = psutil.Process(os.getpid()) + total = proc.memory_info().rss + for child in proc.children(recursive=True): + try: + total += child.memory_info().rss + except (psutil.NoSuchProcess, psutil.AccessDenied): + pass + return total / (1024 * 1024) + + +def get_worker_memory_breakdown() -> list[tuple[int, float]]: + """Return a list of ``(pid, rss_mib)`` for each child process.""" + proc = psutil.Process(os.getpid()) + result: list[tuple[int, float]] = [] + for child in proc.children(recursive=True): + try: + rss_mb = child.memory_info().rss / (1024 * 1024) + result.append((child.pid, rss_mb)) + except (psutil.NoSuchProcess, psutil.AccessDenied): + pass + return result + + +def get_worker_memory_detailed() -> list[dict[str, float]]: + """Return RSS, USS (Unique Set Size), and PSS for each child process. + + USS is the memory *unique* to a process -- not shared with any other. + It directly measures CoW-duplicated pages plus worker-only allocations. + + PSS counts shared pages proportionally (shared_page / num_sharers). + + Returns list of dicts with keys: ``pid``, ``rss``, ``uss``, ``pss`` (all in MiB). + Falls back to RSS-only if ``memory_full_info()`` is unavailable. + """ + proc = psutil.Process(os.getpid()) + result: list[dict[str, float]] = [] + for child in proc.children(recursive=True): + try: + full = child.memory_full_info() + result.append( + { + "pid": float(child.pid), + "rss": full.rss / (1024 * 1024), + "uss": full.uss / (1024 * 1024), + "pss": full.pss / (1024 * 1024), + } + ) + except (psutil.NoSuchProcess, psutil.AccessDenied, AttributeError): + try: + rss_mb = child.memory_info().rss / (1024 * 1024) + result.append( + { + "pid": float(child.pid), + "rss": rss_mb, + "uss": -1.0, + "pss": -1.0, + } + ) + except (psutil.NoSuchProcess, psutil.AccessDenied): + pass + return result + + +def get_uss_mb() -> float: + """Return USS (Unique Set Size) of the current process in MiB. + + Falls back to RSS if ``memory_full_info()`` is unavailable. + """ + proc = psutil.Process(os.getpid()) + try: + return proc.memory_full_info().uss / (1024 * 1024) + except (AttributeError, psutil.AccessDenied): + return proc.memory_info().rss / (1024 * 1024) + + +def log_worker_memory_breakdown(dataset: object) -> None: + """Log a detailed memory breakdown from inside a dataloader worker. + + Designed to be called periodically from ``__getitem__`` when + ``MEMORY_PROFILE=1``. Inspects the dataset's internal state to + report how many ``LeRobotDataset`` instances are loaded, HuggingFace + Arrow table sizes, and the LeRobot ``VideoDecoderCache`` size. + + Args: + dataset: A ``BaseActionLeRobotDataset`` instance (or compatible). + """ + import gc + import logging + + pid = os.getpid() + rss = get_rss_mb() + uss = get_uss_mb() + logger = logging.getLogger(f"memprofile.worker.{pid}") + + logger.warning(f"[WORKER {pid}] RSS={fmt_mb(rss)} USS={fmt_mb(uss)}") + + # --- LeRobotDataset instances --- + datasets_list = getattr(dataset, "_datasets", []) + loaded_count = sum(1 for ds in datasets_list if ds is not None) + total_count = len(datasets_list) + logger.warning(f"[WORKER {pid}] LeRobotDataset: {loaded_count}/{total_count} loaded") + + total_arrow_bytes = 0 + total_hf_rows = 0 + for i, ds in enumerate(datasets_list): + if ds is None: + continue + hf_ds = getattr(ds, "hf_dataset", None) + if hf_ds is None: + logger.warning(f"[WORKER {pid}] ds[{i}]: hf_dataset not yet loaded") + continue + + num_rows = len(hf_ds) + total_hf_rows += num_rows + + arrow_bytes = 0 + data_table = getattr(hf_ds, "_data", None) + if data_table is not None and hasattr(data_table, "nbytes"): + arrow_bytes = data_table.nbytes + total_arrow_bytes += arrow_bytes + + logger.warning(f"[WORKER {pid}] ds[{i}]: rows={num_rows}, arrow={fmt_mb(arrow_bytes / (1024 * 1024))}") + + if loaded_count > 0: + logger.warning( + f"[WORKER {pid}] Total HF rows={total_hf_rows}, total arrow={fmt_mb(total_arrow_bytes / (1024 * 1024))}" + ) + + # --- VideoDecoderCache --- + try: + from lerobot.datasets.video_utils import _default_decoder_cache + + cache_size = _default_decoder_cache.size() + logger.warning(f"[WORKER {pid}] VideoDecoderCache entries: {cache_size}") + except Exception: + pass + + # --- GC stats --- + gc_counts = gc.get_count() + all_objects = len(gc.get_objects()) + logger.warning(f"[WORKER {pid}] GC counts={gc_counts}, tracked objects={all_objects}") + + +def deep_size(obj: object, seen: set | None = None) -> int: + """Approximate deep memory size in bytes for nested Python containers. + + Recursively walks ``dict``, ``list``, ``tuple``, ``set``, and ``frozenset``. + Does **not** follow arbitrary object attributes. + """ + if seen is None: + seen = set() + obj_id = id(obj) + if obj_id in seen: + return 0 + seen.add(obj_id) + size = sys.getsizeof(obj) + if isinstance(obj, dict): + for k, v in obj.items(): + size += deep_size(k, seen) + deep_size(v, seen) + elif isinstance(obj, (list, tuple, set, frozenset)): + for item in obj: + size += deep_size(item, seen) + return size diff --git a/cosmos_framework/data/vfm/augmentor_provider.py b/cosmos_framework/data/vfm/augmentor_provider.py new file mode 100644 index 0000000..2ace65c --- /dev/null +++ b/cosmos_framework/data/vfm/augmentor_provider.py @@ -0,0 +1,1418 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Optional + +import cosmos_framework.data.imaginaire.webdataset.augmentors.image.cropping as cropping +import cosmos_framework.data.imaginaire.webdataset.augmentors.image.normalize as normalize +import cosmos_framework.data.imaginaire.webdataset.augmentors.image.padding as padding +import cosmos_framework.data.imaginaire.webdataset.augmentors.image.resize as resize +import cosmos_framework.data.vfm.augmentors.append_fps_frames_for_image as append_fps_frames_for_image +import cosmos_framework.data.vfm.augmentors.audio_caption as audio_caption +import cosmos_framework.data.vfm.augmentors.caption_filter as caption_filter +import cosmos_framework.data.vfm.augmentors.duration_fps_text_timestamps as duration_fps_text_timestamps +import cosmos_framework.data.vfm.augmentors.image_resolution_filter as image_resolution_filter +import cosmos_framework.data.vfm.augmentors.merge_datadict as merge_datadict +import cosmos_framework.data.vfm.augmentors.resolution_text_info as resolution_text_info +import cosmos_framework.data.vfm.augmentors.sound_sequence_plan as sound_sequence_plan +import cosmos_framework.data.vfm.augmentors.text_tokenizer as text_tokenizer +import cosmos_framework.data.vfm.augmentors.text_transforms_for_image as text_transforms_for_image +import cosmos_framework.data.vfm.augmentors.text_transforms_for_video as text_transforms_for_video +import cosmos_framework.data.vfm.augmentors.video_parsing as video_parsing +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils.lazy_config import LazyDict +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.augmentors import sequence_plan +from cosmos_framework.data.vfm.utils import IMAGE_RES_SIZE_INFO, VIDEO_RES_SIZE_INFO + +AUGMENTOR_OPTIONS = {} + +CAMERA_MOVEMENT_PHRASES = [ + # Panning + "camera pan", + "camera pans", + "camera slowly pan", + "camera slowly pans", + "camera quickly pans", + "camera fast pans", + "panning shot", + "panning camera", + "slow pan", + "quick pan", + "fast pan", + "pan across", + "pan around", + "pan shot", + "panoramic shot", + # Tracking / Dolly + "camera moves", + "camera slowly moves", + "camera quickly moves", + "moving camera", + "tracking shot", + "tracking camera", + "dolly shot", + "dolly in", + "dolly out", + "camera follows", + "camera tracks", + "tracking movement", + # Sweeps / Rotations + "sweeping camera", + "camera sweep", + "rotating camera", + "camera rotation", + "camera rotates", + "camera circles around", + # Tilts + "camera tilt", + "camera tilts", + "camera slowly tilts", + "tilting camera", + "tilt up", + "tilt down", + # Zooms + "camera zoom", + "camera zooms", + "zooming camera", + "zoom in", + "zoom out", + # Handheld / Shake + "handheld camera", + "handheld shot", + "shaky camera", + "camera shake", + "shaky shot", + "handheld movement", +] + + +def augmentor_register(key): + log.info(f"registering {key}...") + + def decorator(func): + AUGMENTOR_OPTIONS[key] = func + return func + + return decorator + + +def get_video_text_transform( + caption_type: str, + embedding_type: Optional[str] = "t5_xxl", + long_caption_ratio: int = 7, + medium_caption_ratio: int = 2, + short_caption_ratio: int = 1, + user_caption_ratio: int = 90, + num_video_frames: int = -1, +): + del num_video_frames + if caption_type == "vila_caption": + video_text_transform = L(text_transforms_for_video.TextTransformForVideo)( + input_keys=[], + args={ + "captions_key": "metas", + "embeddings_key": embedding_type, + "caption_windows_key": "windows", + "caption_type": "vila_caption", + "embedding_caption_type": "vila_caption", + "t5_tokens": {"num": 512}, + "is_mask_all_ones": True, + }, + ) + elif caption_type == "t2w_qwen2p5_7b": + log.info( + f"caption_type: {caption_type}, long_caption_ratio: {long_caption_ratio}, medium_caption_ratio: {medium_caption_ratio}, short_caption_ratio: {short_caption_ratio}, user_caption_ratio: {user_caption_ratio}" + ) + video_text_transform = L(text_transforms_for_video.TextTransformForVideo)( + input_keys=[], + args={ + "captions_key": "metas", + "embeddings_key": embedding_type, + "caption_windows_key": "t2w_windows", + "caption_type": "qwen2p5_7b_caption", + "embedding_caption_type": "t2w_qwen2p5_7b", + "t5_tokens": {"num": 512}, + "is_mask_all_ones": True, + "caption_probs": { + "long": long_caption_ratio, + "medium": medium_caption_ratio, + "short": short_caption_ratio, + "user": user_caption_ratio, + }, + }, + ) + elif caption_type == "i2w_qwen2p5_7b_later_frames": + video_text_transform = L(text_transforms_for_video.TextTransformForVideo)( + input_keys=[], + args={ + "captions_key": "metas", + "embeddings_key": embedding_type, + "caption_windows_key": "i2w_windows_later_frames", + "caption_type": "qwen2p5_7b_caption", + "embedding_caption_type": "i2w_qwen2p5_7b_later_frames", + "t5_tokens": {"num": 512}, + "is_mask_all_ones": True, + "caption_probs": { + "long": long_caption_ratio, + "medium": medium_caption_ratio, + "short": short_caption_ratio, + "user": user_caption_ratio, + }, + }, + ) + else: + raise ValueError(f"Unsupported caption type ({caption_type}) for video data") + + return video_text_transform + + +@augmentor_register("video_basic_augmentor_v1") +def get_video_augmentor_v1( + resolution: str, + caption_type: str = "vila_caption", + embedding_type: str = "t5_xxl", + min_fps: int = 10, + max_fps: int = 60, + long_caption_ratio: int = 7, + medium_caption_ratio: int = 2, + short_caption_ratio: int = 1, + user_caption_ratio: int = 90, +): + """Video augmentor V1. It relies on a separate video decoder to decode videos of required number of frames. + Augmentors here will resize the video, add reflection padding, and extract captions and embeddings. + + Supported caption_type include vila_caption. + Supported embedding_type include t5_xxl. + """ + assert caption_type == "vila_caption", f"Unsupported caption type ({caption_type}) for video data" + assert embedding_type == "t5_xxl", f"Unsupported embeddings type ({embedding_type}) for video data" + video_text_transform = get_video_text_transform( + caption_type=caption_type, + embedding_type=embedding_type, + long_caption_ratio=long_caption_ratio, + medium_caption_ratio=medium_caption_ratio, + short_caption_ratio=short_caption_ratio, + user_caption_ratio=user_caption_ratio, + ) + + return { + "merge_datadict": L(merge_datadict.DataDictMerger)( + input_keys=["video"], + output_keys=[ + "video", + "fps", + "num_frames", + "chunk_index", + "frame_start", + "frame_end", + "n_orig_video_frames", + "num_multiplier", # Add frame skipping multiplier for duration/FPS calculations + ], + ), + "resize_largest_side_aspect_ratio_preserving": L(resize.ResizeLargestSideAspectPreserving)( + input_keys=["video"], + args={"size": VIDEO_RES_SIZE_INFO[resolution]}, + ), + "reflection_padding": L(padding.ReflectionPadding)( + input_keys=["video"], + args={"size": VIDEO_RES_SIZE_INFO[resolution]}, + ), + "text_transform": video_text_transform, + } + + +@augmentor_register("video_basic_augmentor_v2") +def get_video_augmentor_v2( + resolution: str, + caption_type: str = "t2w_qwen2p5_7b", + embedding_type: Optional[str] = "t5_xxl", + min_fps: int = 10, + max_fps: int = 60, + long_caption_ratio: int = 7, + medium_caption_ratio: int = 2, + short_caption_ratio: int = 1, + user_caption_ratio: int = 90, + num_video_frames: int = -1, + use_native_fps: bool = True, + use_original_fps: bool = False, + use_random_consecutive_frames: bool = False, + append_duration_fps_timestamps: bool = False, + append_resolution_info: bool = False, + use_dynamic_fps: bool = False, + low_fps_bias: float = 0.5, + dataset_resolution_type: str = "all", # Unused here; resolution check is only in VideoParsingWithFullFrames (v3) + **_kwargs, # absorbs tokenizer_config, cfg_dropout_rate, caption_config, etc. passed by generic callers +): + """ + num_video_frames: -1 means use all frames, otherwise use the number of frames specified. + + Video augmentor V2. It works with a naive video decoder ("video_naive_bytes") that does nothing. + Augmentors here include: + - a basic video decoder that fetches frames within a window and delegates further subsampling or duplication to the modeling code to produce videos with the required number of frames. + - resize the video + - add reflection padding + - extract captions and embeddings. + + When use_random_consecutive_frames is True, the augmentor will sample random consecutive frames, preserving the original fps. + + Supported caption_type include t2w_qwen2p5_7b and i2w_qwen2p5_7b_later_frames. + Supported embedding_type include t5_xxl and umt5_xxl. + """ + video_text_transform = get_video_text_transform( + caption_type=caption_type, + embedding_type=embedding_type, + long_caption_ratio=long_caption_ratio, + medium_caption_ratio=medium_caption_ratio, + short_caption_ratio=short_caption_ratio, + user_caption_ratio=user_caption_ratio, + ) + if caption_type == "t2w_qwen2p5_7b": + key_for_caption = "t2w_windows" + elif caption_type == "i2w_qwen2p5_7b_later_frames": + key_for_caption = "i2w_windows_later_frames" + else: + f"Unsupported caption type ({caption_type}) for video data" + if embedding_type is not None: + assert embedding_type in ( + "t5_xxl", + "umt5_xxl", + ), f"Unsupported embeddings type ({embedding_type}) for video data" + + return { + "video_parsing": L(video_parsing.VideoParsing)( + input_keys=["metas", "video"], + args={ + "key_for_caption": key_for_caption, + "min_duration": 4.0, + "min_fps": min_fps, + "max_fps": max_fps, + "video_decode_num_threads": 4, + "num_video_frames": num_video_frames, + "use_native_fps": use_native_fps, + "use_original_fps": use_original_fps, + "use_random_consecutive_frames": use_random_consecutive_frames, + "use_dynamic_fps": use_dynamic_fps, + "low_fps_bias": low_fps_bias, + }, + ), + "merge_datadict": L(merge_datadict.DataDictMerger)( + input_keys=["video"], + output_keys=[ + "video", + "fps", + "num_frames", + "chunk_index", + "frame_start", + "frame_end", + "n_orig_video_frames", + "num_multiplier", # Add frame skipping multiplier for duration/FPS calculations + "conditioning_fps", # Add conditioning FPS for RoPE modulation + ], + ), + "resize_largest_side_aspect_ratio_preserving": L(resize.ResizeLargestSideAspectPreserving)( + input_keys=["video"], + args={"size": VIDEO_RES_SIZE_INFO[resolution]}, + ), + "reflection_padding": L(padding.ReflectionPadding)( + input_keys=["video"], + args={"size": VIDEO_RES_SIZE_INFO[resolution]}, + ), + "text_transform": video_text_transform, + # Duration/FPS timestamp augmentor - appends metadata like "The video is 2.5 seconds long and is of 24 FPS." + # To customize the template or separator, add them to the args dict below: + # "template": "Custom format: {duration:.2f} seconds at {fps:.0f} FPS" + # Must include "{duration} seconds" and "{fps} FPS" in the template for the visualization callback + # "separator": " - " # Used when caption doesn't end with '.' + "duration_fps_timestamps": L(duration_fps_text_timestamps.DurationFPSTextTimeStamps)( + input_keys=["ai_caption", "video", "conditioning_fps"], + output_keys=[ + "ai_caption", + "conditioning_duration", + "duration_fps_template", + ], # Add duration and template as output keys + args={ + "caption_key": "ai_caption", + "video_key": "video", + "fps_key": "conditioning_fps", + "enabled": append_duration_fps_timestamps, + }, + ), + # Resolution info augmentor - appends metadata like "This video is 480x854." + # Reads final_height/final_width from CropToMultiple (required). + "resolution_info": L(resolution_text_info.ResolutionTextInfo)( + input_keys=["ai_caption", "video", "image_size"], + output_keys=[ + "ai_caption", + ], + args={ + "caption_key": "ai_caption", + "video_key": "video", + "enabled": append_resolution_info, + }, + ), + } + + +@augmentor_register("video_basic_augmentor_v2_with_tokenization") +def get_video_augmentor_v2_with_tokenization( + resolution: str, + caption_type: str = "t2w_qwen2p5_7b", + embedding_type: Optional[str] = "t5_xxl", + min_fps: int = 10, + max_fps: int = 60, + long_caption_ratio: int = 7, + medium_caption_ratio: int = 2, + short_caption_ratio: int = 1, + user_caption_ratio: int = 90, + num_video_frames: int = -1, + use_native_fps: bool = True, + use_original_fps: bool = False, + use_random_consecutive_frames: bool = False, + tokenizer_config: Optional[LazyDict] = None, + cfg_dropout_rate: float = 0.0, + append_duration_fps_timestamps: bool = False, + append_resolution_info: bool = False, + use_dynamic_fps: bool = False, + low_fps_bias: float = 0.5, + caption_config: dict | None = None, + use_system_prompt: bool = False, + dataset_resolution_type: str = "all", # Unused here; resolution check is only in VideoParsingWithFullFrames (v3) +): + """ + num_video_frames: -1 means use all frames, otherwise use the number of frames specified. + + Video augmentor V2. It works with a naive video decoder ("video_naive_bytes") that does nothing. + Augmentors here include: + - a basic video decoder that fetches frames within a window and delegates further subsampling or duplication to the modeling code to produce videos with the required number of frames. + - resize the video + - add reflection padding + - extract captions and embeddings. + + When use_random_consecutive_frames is True, the augmentor will sample random consecutive frames, preserving the original fps. + + Supported caption_type include t2w_qwen2p5_7b and i2w_qwen2p5_7b_later_frames. + Supported embedding_type include t5_xxl and umt5_xxl. + """ + video_text_transform = get_video_text_transform( + caption_type=caption_type, + embedding_type=embedding_type, + long_caption_ratio=long_caption_ratio, + medium_caption_ratio=medium_caption_ratio, + short_caption_ratio=short_caption_ratio, + user_caption_ratio=user_caption_ratio, + ) + if caption_type == "t2w_qwen2p5_7b": + key_for_caption = "t2w_windows" + elif caption_type == "i2w_qwen2p5_7b_later_frames": + key_for_caption = "i2w_windows_later_frames" + else: + f"Unsupported caption type ({caption_type}) for video data" + if embedding_type is not None: + assert embedding_type in ( + "t5_xxl", + "umt5_xxl", + ), f"Unsupported embeddings type ({embedding_type}) for video data" + + return { + "video_parsing": L(video_parsing.VideoParsing)( + input_keys=["metas", "video"], + args={ + "key_for_caption": key_for_caption, + "min_duration": 4.0, + "min_fps": min_fps, + "max_fps": max_fps, + "video_decode_num_threads": 4, + "num_video_frames": num_video_frames, + "use_native_fps": use_native_fps, + "use_original_fps": use_original_fps, + "use_random_consecutive_frames": use_random_consecutive_frames, + "use_dynamic_fps": use_dynamic_fps, + "low_fps_bias": low_fps_bias, + }, + ), + "merge_datadict": L(merge_datadict.DataDictMerger)( + input_keys=["video"], + output_keys=[ + "video", + "fps", + "num_frames", + "chunk_index", + "frame_start", + "frame_end", + "n_orig_video_frames", + "num_multiplier", # Add frame skipping multiplier for duration/FPS calculations + "conditioning_fps", # Add conditioning FPS for RoPE modulation + ], + ), + "resize_largest_side_aspect_ratio_preserving": L(resize.ResizeLargestSideAspectPreserving)( + input_keys=["video"], + args={"size": VIDEO_RES_SIZE_INFO[resolution]}, + ), + "reflection_padding": L(padding.ReflectionPadding)( + input_keys=["video"], + args={"size": VIDEO_RES_SIZE_INFO[resolution]}, + ), + "text_transform": video_text_transform, + # Duration/FPS timestamp augmentor - appends metadata like "The video is 2.5 seconds long and is of 24 FPS." + # To customize the template or separator, add them to the args dict below: + # "template": "Custom format: {duration:.2f} seconds at {fps:.0f} FPS" + # Must include "{duration} seconds" and "{fps} FPS" in the template for the visualization callback + # "separator": " - " # Used when caption doesn't end with '.' + "duration_fps_timestamps": L(duration_fps_text_timestamps.DurationFPSTextTimeStamps)( + input_keys=["ai_caption", "video", "conditioning_fps"], + output_keys=[ + "ai_caption", + "conditioning_duration", + "duration_fps_template", + ], # Add duration and template as output keys + args={ + "caption_key": "ai_caption", + "video_key": "video", + "fps_key": "conditioning_fps", + "enabled": append_duration_fps_timestamps, + }, + ), + # Resolution info augmentor - appends metadata like "This video is 480x854." + # Reads final_height/final_width from CropToMultiple (required). + "resolution_info": L(resolution_text_info.ResolutionTextInfo)( + input_keys=["ai_caption", "video", "image_size"], + output_keys=[ + "ai_caption", + ], + args={ + "caption_key": "ai_caption", + "video_key": "video", + "enabled": append_resolution_info, + }, + ), + "text_tokenization": L(text_tokenizer.TextTokenizerTransform)( + input_keys=["ai_caption"], + output_keys=["text_token_ids"], + args={ + "tokenizer_config": tokenizer_config, + "cfg_dropout_rate": cfg_dropout_rate, + "use_system_prompt": use_system_prompt, + }, + ), + } + + +@augmentor_register("video_basic_augmentor_v3") +def get_video_augmentor_v3( + resolution: str, + caption_config: dict | None = None, + tokenizer_config: Optional[LazyDict] = None, + cfg_dropout_rate: float = 0.0, + append_duration_fps_timestamps: bool = False, + append_resolution_info: bool = False, + use_dynamic_fps: bool = False, + max_stride: int = 3, + min_stride: int = 1, + use_system_prompt: bool = False, + resize_on_read: bool = False, + dataset_resolution_type: str = "all", + max_num_frames: int = 1000, + **kwargs, +): + """Build a video augmentation pipeline with parsing, resizing, captioning, and tokenization. + + Args: + resolution: Target resolution key (e.g., "480p"). Looked up in VIDEO_RES_SIZE_INFO + to determine the resize target size. + caption_config: Caption sampling configuration mapping caption field names to + selection ratios. Example:: + + { + "caption_short": {"ratio": 0.1, "use_for": "all"}, + "caption_rewrite_descriptive": {"ratio": 0.2, "use_for": "all"}, + "caption_dense": {"ratio": 0.3, "use_for": "all"}, + } + + tokenizer_config: Lazy config for the text tokenizer. Passed to TextTokenizerTransform + to convert captions into token IDs. + cfg_dropout_rate: Probability of dropping the caption (replacing with empty string) + for classifier-free guidance training. + append_duration_fps_timestamps: If True, appends a duration/FPS metadata string + (e.g., "The video is 2.5 seconds long and is of 24 FPS.") to the caption. + append_resolution_info: If True, appends a resolution metadata string + (e.g., "This video is 480x854.") to the caption. + use_dynamic_fps: If True, enables dynamic FPS sampling during video parsing, + allowing variable stride to simulate different frame rates. + max_stride: Maximum temporal stride for frame sampling during video parsing. + use_system_prompt: If True, prepends a system prompt to the tokenized caption + that instructs the model it is a video generation assistant. + resize_on_read: If True, resizes video frames during decoding rather than as a + separate augmentation step, reducing peak CPU memory usage. + **kwargs: Additional keyword arguments. + + Returns: + dict: Ordered dictionary of augmentation stage name to LazyCall config. + + Augmentors include: + - a basic video decoder that fetches frames within a window and delegates further + subsampling or duplication to the modeling code to produce videos with the + required number of frames. + - resize the video + - add reflection padding + - extract captions and embeddings. + + Supported caption_type include t2w_qwen2p5_7b and i2w_qwen2p5_7b_later_frames. + Supported embedding_type include t5_xxl and umt5_xxl. + """ + + conditioning_config = kwargs.get("conditioning_config", None) + uniform_conditioning = kwargs.get("uniform_conditioning", False) + temporal_compression_factor = kwargs.get("temporal_compression_factor", 4) + + print("Running video_basic_augmentor_v3...") + augmentors = { + "video_parsing": L(video_parsing.VideoParsingWithFullFrames)( + input_keys=["metas", "video"], + args={ + "video_decode_num_threads": 4, + "max_num_frames": max_num_frames, + "use_dynamic_fps": use_dynamic_fps, + "max_stride": max_stride, + "min_stride": min_stride, + "seek_mode": "exact", # Change to "approximate"? + "dataset_resolution_type": dataset_resolution_type, + }, + ), + "merge_datadict": L(merge_datadict.DataDictMerger)( + input_keys=["video"], + output_keys=[ + "video", + "fps", + "num_frames", + "frame_start", + "frame_end", + "n_orig_video_frames", + "conditioning_fps", # Add conditioning FPS for RoPE modulation + ], + ), + } + if conditioning_config is not None or uniform_conditioning: + augmentors["sequence_plan"] = L(sequence_plan.SequencePlanAugmentor)( + input_keys=["video"], + args={ + "conditioning_config": conditioning_config, + "uniform_conditioning": uniform_conditioning, + "temporal_compression_factor": temporal_compression_factor, + }, + ) + augmentors.update( + { + "resize_largest_side_aspect_ratio_preserving": L(resize.ResizeLargestSideAspectPreserving)( + input_keys=["video"], + args={"size": VIDEO_RES_SIZE_INFO[resolution]}, + ), + "reflection_padding": L(padding.ReflectionPadding)( + input_keys=["video"], + args={"size": VIDEO_RES_SIZE_INFO[resolution]}, + ), + "text_transform": L(text_transforms_for_video.TextTransformForVideoWithFullFrames)( + input_keys=["metas", "ai_caption", "sequence_plan"], + args={ + "caption_config": caption_config, + "caption_prefix": kwargs.get("caption_prefix", None), + }, + ), + # Duration/FPS timestamp augmentor - appends metadata like "The video is 2.5 seconds long and is of 24 FPS." + # To customize the template or separator, add them to the args dict below: + # "template": "Custom format: {duration:.2f} seconds at {fps:.0f} FPS" + # Must include "{duration} seconds" and "{fps} FPS" in the template for the visualization callback + # "separator": " - " # Used when caption doesn't end with '.' + "duration_fps_timestamps": L(duration_fps_text_timestamps.DurationFPSTextTimeStamps)( + input_keys=["ai_caption", "video", "conditioning_fps"], + output_keys=[ + "ai_caption", + "conditioning_duration", + "duration_fps_template", + ], # Add duration and template as output keys + args={ + "caption_key": "ai_caption", + "video_key": "video", + "fps_key": "conditioning_fps", + "enabled": append_duration_fps_timestamps, + }, + ), + # Resolution info augmentor - appends metadata like "This video is 480x854." + # Reads final_height/final_width from CropToMultiple (required). + "resolution_info": L(resolution_text_info.ResolutionTextInfo)( + input_keys=["ai_caption", "video", "image_size"], + output_keys=[ + "ai_caption", + ], + args={ + "caption_key": "ai_caption", + "video_key": "video", + "enabled": append_resolution_info, + }, + ), + "text_tokenization": L(text_tokenizer.TextTokenizerTransform)( + input_keys=["ai_caption"], + output_keys=["text_token_ids"], + args={ + "tokenizer_config": tokenizer_config, + "cfg_dropout_rate": cfg_dropout_rate, + "use_system_prompt": use_system_prompt, + }, + ), + } + ) + + if resize_on_read: + # When resize_on_read is True, we resize the video frames on read instead of during decoding. + # This is useful for reducing CPU memory usage by avoiding the need to load the entire video into memory. + augmentors["video_parsing"]["args"]["size"] = VIDEO_RES_SIZE_INFO[resolution] + del augmentors["resize_largest_side_aspect_ratio_preserving"] + return augmentors + + + +# Use video_basic_augmentor_v3_json_caption instead. +@augmentor_register("video_basic_augmentor_v3_with_audio") +def get_video_augmentor_v3_with_audio( + resolution: str, + caption_config: dict | None = None, + tokenizer_config: Optional[LazyDict] = None, + cfg_dropout_rate: float = 0.0, + append_duration_fps_timestamps: bool = False, + use_dynamic_fps: bool = False, + max_stride: int = 3, + resize_on_read: bool = False, + audio_sample_rate: int = 48000, + sound_generation_mode: str = "t2vs", + key_renames: dict[str, str] | None = None, + extract_audio: bool = True, + **kwargs, +): + """ + Same as video_basic_augmentor_v3 but with audio extraction enabled. + For use with V2A (tv2s) and T2VS datasets where both video and audio are loaded. + + Args: + sound_generation_mode: One of "t2vs", "tv2s", "ts2v", "ti2sv". + Controls how the SequencePlan is built for conditioning. + key_renames: Optional mapping of old_key -> new_key to rename data_dict keys + before the augmentor pipeline runs (e.g. {"metas_w_audio_caps": "metas"}). + extract_audio: When True (default), decodes audio from video bytes. + When False, emits placeholder sound=None and audio_sample_rate keys + without decoding audio. Useful for mixing video-only and audio + datasets in the same dataloader with consistent output keys. + """ + augmentors = get_video_augmentor_v3( + resolution=resolution, + caption_config=caption_config, + tokenizer_config=tokenizer_config, + cfg_dropout_rate=cfg_dropout_rate, + append_duration_fps_timestamps=append_duration_fps_timestamps, + use_dynamic_fps=use_dynamic_fps, + max_stride=max_stride, + resize_on_read=resize_on_read, + **kwargs, + ) + + # Insert key renamer at the very start if key_renames is provided + if key_renames: + renamed_augmentors = { + "key_rename": L(merge_datadict.KeyRenamer)( + input_keys=[], + args={"rename_map": key_renames}, + ), + } + renamed_augmentors.update(augmentors) + augmentors = renamed_augmentors + + # Configure audio extraction in the video parser + augmentors["video_parsing"]["args"]["extract_audio"] = extract_audio + augmentors["video_parsing"]["args"]["audio_sample_rate"] = audio_sample_rate + if not extract_audio: + augmentors["video_parsing"]["args"]["emit_placeholder_sound"] = True + + # Add sound and audio_sample_rate to merge keys so they propagate through the pipeline + augmentors["merge_datadict"]["output_keys"].extend(["sound", "audio_sample_rate"]) + + # Tell text_transform to keep metas — AudioCaptionAppender will clean it up + augmentors["text_transform"]["args"]["keep_metas"] = True + + # Insert audio caption appender BEFORE text_tokenization. + # We rebuild the ordered dict to ensure correct pipeline order. + text_tokenization = augmentors.pop("text_tokenization") + augmentors["audio_caption"] = L(audio_caption.AudioCaptionAppender)( + input_keys=["metas", "ai_caption"], + args={ + "audio_caption_key": "caption_audio", + "sound_key": "sound", + }, + ) + augmentors["text_tokenization"] = text_tokenization + + # Add sequence plan builder at the end of the pipeline (after all data is ready) + augmentors["sound_sequence_plan"] = L(sound_sequence_plan.SoundSequencePlanBuilder)( + input_keys=[], + args={ + "mode": sound_generation_mode, + "video_key": "video", + "sound_key": "sound", + }, + ) + return augmentors + + +@augmentor_register("video_basic_augmentor_v3_json_caption") +def get_video_augmentor_v3_json_caption( + resolution: str, + caption_config: dict | None = None, + tokenizer_config: Optional[LazyDict] = None, + cfg_dropout_rate: float = 0.0, + append_duration_fps_timestamps: bool = False, + append_resolution_info: bool = False, + use_dynamic_fps: bool = False, + max_stride: int = 3, + min_stride: int = 1, + use_system_prompt: bool = False, + resize_on_read: bool = False, + dataset_resolution_type: str = "all", + max_num_frames: int = 1000, + audio_sample_rate: int = 48000, + sound_generation_mode: str = "t2vs", + extract_audio: bool = True, + caption_key: str = "caption", + **kwargs, +): + """Build a video augmentation pipeline for JSON-captioned, chunked video datasets. + + The pipeline samples a single caption chunk per video, decodes only that chunk's + frames, optionally extracts audio, and injects ``metas["caption_audio"]`` into the + caption dict as an ``audio_description`` field when present. + + Args: + resolution: Target resolution key (e.g., "480p"). Looked up in VIDEO_RES_SIZE_INFO + to determine the resize target size. + caption_config: Caption sampling configuration mapping caption field names to + + { + "json_field_dropout_rate": 0.1, + } + caption_key: Metadata key containing the chunked JSON caption string. + tokenizer_config: Lazy config for the text tokenizer. Passed to TextTokenizerTransform + to convert captions into token IDs. + cfg_dropout_rate: Probability of dropping the caption (replacing with empty string) + for classifier-free guidance training. + append_duration_fps_timestamps: If True, injects duration/FPS fields into the + caption dict. + append_resolution_info: If True, injects resolution/aspect-ratio fields into the + caption dict. + use_dynamic_fps: If True, enables dynamic FPS sampling during video parsing, + allowing variable stride to simulate different frame rates. + max_stride: Maximum temporal stride for frame sampling during video parsing. + use_system_prompt: If True, prepends a system prompt to the tokenized caption + that instructs the model it is a video generation assistant. + resize_on_read: If True, resizes video frames during decoding rather than as a + separate augmentation step, reducing peak CPU memory usage. + audio_sample_rate: Sample rate for audio extraction. + sound_generation_mode: One of "t2vs", "tv2s", "ts2v", "ti2sv". + Controls how the SequencePlan is built for conditioning. + extract_audio: When True (default), decodes audio from video bytes. + When False, emits placeholder sound=None and audio_sample_rate keys + without decoding audio. Useful for mixing video-only and audio + datasets in the same dataloader with consistent output keys. + **kwargs: Additional keyword arguments forwarded via ``conditioning_config``, + ``uniform_conditioning``, ``temporal_compression_factor``. + + Returns: + dict: Ordered dictionary of augmentation stage name to LazyCall config. + """ + + conditioning_config = kwargs.get("conditioning_config", None) + uniform_conditioning = kwargs.get("uniform_conditioning", False) + temporal_compression_factor = kwargs.get("temporal_compression_factor", 4) + + print("Running video_augmentor_v3_json_caption...") + augmentors = { + # Caption parsing runs BEFORE video parsing so that VideoParsingChunkedFrames can + # decode only the frames belonging to a randomly sampled caption chunk. + # keep_metas=True so that VideoParsingChunkedFrames still has framerate/width/height/nb_frames. + "text_transform": L(text_transforms_for_video.TextTransformForVideoJsonCaption)( + input_keys=["metas", "video"], + args={ + "caption_config": caption_config, + "caption_key": caption_key, + "keep_metas": True, + }, + ), + "video_parsing": L(video_parsing.VideoParsingChunkedFrames)( + input_keys=["metas", "video"], + args={ + "video_decode_num_threads": 4, + "max_num_frames": max_num_frames, + "use_dynamic_fps": use_dynamic_fps, + "max_stride": max_stride, + "min_stride": min_stride, + "seek_mode": "exact", + "dataset_resolution_type": dataset_resolution_type, + "extract_audio": extract_audio, + "audio_sample_rate": audio_sample_rate, + "emit_placeholder_sound": not extract_audio, + }, + ), + "merge_datadict": L(merge_datadict.DataDictMerger)( + input_keys=["video"], + output_keys=[ + "video", + "fps", + "num_frames", + "frame_start", + "frame_end", + "n_orig_video_frames", + "conditioning_fps", + "sound", + "audio_sample_rate", + ], + ), + } + + if conditioning_config is not None or uniform_conditioning: + augmentors["sequence_plan"] = L(sequence_plan.SequencePlanAugmentor)( + input_keys=["video"], + args={ + "conditioning_config": conditioning_config, + "uniform_conditioning": uniform_conditioning, + "temporal_compression_factor": temporal_compression_factor, + }, + ) + augmentors.update( + { + "resize_largest_side_aspect_ratio_preserving": L(resize.ResizeLargestSideAspectPreserving)( + input_keys=["video"], + args={"size": VIDEO_RES_SIZE_INFO[resolution]}, + ), + "reflection_padding": L(padding.ReflectionPadding)( + input_keys=["video"], + args={"size": VIDEO_RES_SIZE_INFO[resolution]}, + ), + # Duration/FPS timestamp augmentor - appends metadata like "The video is 2.5 seconds long and is of 24 FPS." + # To customize the template or separator, add them to the args dict below: + # "template": "Custom format: {duration:.2f} seconds at {fps:.0f} FPS" + # Must include "{duration} seconds" and "{fps} FPS" in the template for the visualization callback + # "separator": " - " # Used when caption doesn't end with '.' + "duration_fps_timestamps": L(duration_fps_text_timestamps.DurationFPSTextTimeStamps)( + input_keys=["ai_caption", "video", "conditioning_fps"], + output_keys=[ + "ai_caption", + "conditioning_duration", + "duration_fps_template", + ], # Add duration and template as output keys + args={ + "caption_key": "ai_caption", + "video_key": "video", + "fps_key": "conditioning_fps", + "enabled": append_duration_fps_timestamps, + }, + ), + # Resolution info augmentor - appends metadata like "This video is 480x854." + # Reads final_height/final_width from CropToMultiple (required). + "resolution_info": L(resolution_text_info.ResolutionTextInfo)( + input_keys=["ai_caption", "video", "image_size"], + output_keys=[ + "ai_caption", + ], + args={ + "caption_key": "ai_caption", + "video_key": "video", + "enabled": append_resolution_info, + }, + ), + "text_tokenization": L(text_tokenizer.TextTokenizerTransform)( + input_keys=["ai_caption"], + output_keys=["text_token_ids"], + args={ + "tokenizer_config": tokenizer_config, + "cfg_dropout_rate": cfg_dropout_rate, + "use_system_prompt": use_system_prompt, + }, + ), + "sound_sequence_plan": L(sound_sequence_plan.SoundSequencePlanBuilder)( + input_keys=[], + args={ + "mode": sound_generation_mode, + "video_key": "video", + "sound_key": "sound", + }, + ), + } + ) + + if resize_on_read: + # When resize_on_read is True, we resize the video frames on read instead of during decoding. + # This is useful for reducing CPU memory usage by avoiding the need to load the entire video into memory. + augmentors["video_parsing"]["args"]["size"] = VIDEO_RES_SIZE_INFO[resolution] + del augmentors["resize_largest_side_aspect_ratio_preserving"] + return augmentors + + +@augmentor_register("video_basic_augmentor_v3_json_caption_crop_bottom") +def get_video_augmentor_v3_json_caption_crop_bottom( + crop_to_height: int, + **kwargs, +): + """Same as ``video_basic_augmentor_v3_json_caption``, but replaces the trailing + ``ReflectionPadding`` stage with a top-anchored ``BottomCrop(target_height=crop_to_height)``. + + The resize step still targets ``VIDEO_RES_SIZE_INFO[resolution]`` (either via the + standalone ``ResizeLargestSideAspectPreserving`` stage or, when ``resize_on_read=True``, + fused into ``video_parsing``), so the resize ratio is identical to the reflection-pad + variant. Only the trailing "fill or crop" step differs. + + For ``resolution="480"`` + a 1920x1080 16:9 source, the resize produces 832x468, and + ``crop_to_height=448`` trims to 832x448 (divisible by 32). Use only when + ``crop_to_height`` is <= the post-resize height (otherwise ``BottomCrop`` will fail + its source-height assertion). + """ + augmentors = get_video_augmentor_v3_json_caption(**kwargs) + assert "reflection_padding" in augmentors, ( + "expected reflection_padding stage to replace; did get_video_augmentor_v3_json_caption change its pipeline?" + ) + # Rebuild the ordered dict so bottom_crop sits exactly where reflection_padding was. + new_augmentors: dict = {} + for k, v in augmentors.items(): + if k == "reflection_padding": + new_augmentors["bottom_crop"] = L(cropping.BottomCrop)( + input_keys=["video"], + args={"target_height": crop_to_height}, + ) + else: + new_augmentors[k] = v + return new_augmentors + + +@augmentor_register("noframedrop_nocameramove_video_augmentor_v1") +def get_noframedrop_nocameramove_video_augmentor_v1( + resolution: str, + caption_type: str = "t2w_qwen2p5_7b", + embedding_type: Optional[str] = "t5_xxl", + min_fps: int = 10, + max_fps: int = 60, + long_caption_ratio: int = 7, + medium_caption_ratio: int = 2, + short_caption_ratio: int = 1, + user_caption_ratio: int = 90, + num_video_frames: int = -1, + use_native_fps: bool = True, + use_original_fps: bool = False, + use_random_consecutive_frames: bool = False, + dataset_resolution_type: str = "all", # Unused here; resolution check is only in VideoParsingWithFullFrames (v3) +): + """ + This augmentor is v2 + the following: + - no frame drop by ensure num_multipler is always 1 + - no camera move (indiciated by the camera related bad words in the caption) + """ + video_text_transform = get_video_text_transform( + caption_type=caption_type, + embedding_type=embedding_type, + long_caption_ratio=long_caption_ratio, + medium_caption_ratio=medium_caption_ratio, + short_caption_ratio=short_caption_ratio, + user_caption_ratio=user_caption_ratio, + ) + if caption_type == "t2w_qwen2p5_7b": + key_for_caption = "t2w_windows" + elif caption_type == "i2w_qwen2p5_7b_later_frames": + key_for_caption = "i2w_windows_later_frames" + else: + f"Unsupported caption type ({caption_type}) for video data" + if embedding_type is not None: + assert embedding_type in ( + "t5_xxl", + "umt5_xxl", + ), f"Unsupported embeddings type ({embedding_type}) for video data" + + contain_keyword = False # ensure no camera move + augmentations = { + "video_parsing": L(video_parsing.VideoParsing)( + input_keys=["metas", "video"], + args={ + "key_for_caption": key_for_caption, + "min_duration": 4.0, + "min_fps": min_fps, + "max_fps": max_fps, + "video_decode_num_threads": 4, + "num_video_frames": num_video_frames, + "use_native_fps": use_native_fps, + "use_original_fps": use_original_fps, + "use_random_consecutive_frames": use_random_consecutive_frames, + # Both use_original_fps=True and "allowed_num_multiplers": [1] prevent frame dropping. + # Key differences: + # - use_original_fps=True: Hard-codes num_multiplier=1 and ignores allowed_num_multiplers setting. + # Won't skip entire videos, but may discard head/tail frames, potentially causing + # video-caption misalignment. + # - "allowed_num_multiplers": [1]: Uses the multiplier system but restricts it to 1x only. May skip videos, causing slower dataloader + "allowed_num_multiplers": [1], + }, + ), + "merge_datadict": L(merge_datadict.DataDictMerger)( + input_keys=["video"], + output_keys=[ + "video", + "fps", + "num_frames", + "chunk_index", + "frame_start", + "frame_end", + "n_orig_video_frames", + ], + ), + "resize_largest_side_aspect_ratio_preserving": L(resize.ResizeLargestSideAspectPreserving)( + input_keys=["video"], + args={"size": VIDEO_RES_SIZE_INFO[resolution]}, + ), + "reflection_padding": L(padding.ReflectionPadding)( + input_keys=["video"], + args={"size": VIDEO_RES_SIZE_INFO[resolution]}, + ), + "text_transform": video_text_transform, + "caption_filter": L(caption_filter.CaptionFilter)( + input_keys=["ai_caption"], # Works with ai_caption from TextTransformForVideo + args={ + "keywords": CAMERA_MOVEMENT_PHRASES, + "contain_keyword": contain_keyword, + "log_filtered": False, # Enable logging to see what gets filtered + "filter_stats": True, + # For 4k and physics AI datasets, even if this has camera movement, it is still good + "dont_apply_on_webdataset_names": [ + "4k_", + "a2d2_", + "agibot_", + "alpamayo_", + "bridgev2p1_", + "droid_", + "gr00t_", + "nexar", + "onex", + "openx", + "physical-ai-special", + "physics-cosmos-db", + "wisa", + "robomind", + "smartspace_", + ], + }, + ), + } + mode_str = "contain" if contain_keyword else "exclude" + log.info( + f"[video] noframedrop_nocameramove_video_augmentor_v1: Added caption filter in '{mode_str}' mode " + f"with {len(CAMERA_MOVEMENT_PHRASES)} camera movement phrases" + ) + return augmentations + + +@augmentor_register("nocameramove_video_augmentor_v1") +def get_nocameramove_video_augmentor_v1( + resolution: str, + caption_type: str = "t2w_qwen2p5_7b", + embedding_type: Optional[str] = "t5_xxl", + min_fps: int = 10, + max_fps: int = 60, + long_caption_ratio: int = 7, + medium_caption_ratio: int = 2, + short_caption_ratio: int = 1, + user_caption_ratio: int = 90, + num_video_frames: int = -1, + use_native_fps: bool = True, + use_original_fps: bool = False, + use_random_consecutive_frames: bool = False, +): + """ + This augmentor is based on noframedrop_nocameramove_video_augmentor_v1 but: + - allows limited frame drop by setting allowed_num_multiplers to [1,2] + - no camera move (indicated by the camera related bad words in the caption) + """ + # Get the base augmentations from the no-frame-drop version + augmentations = get_noframedrop_nocameramove_video_augmentor_v1( + resolution=resolution, + caption_type=caption_type, + embedding_type=embedding_type, + min_fps=min_fps, + max_fps=max_fps, + long_caption_ratio=long_caption_ratio, + medium_caption_ratio=medium_caption_ratio, + short_caption_ratio=short_caption_ratio, + user_caption_ratio=user_caption_ratio, + num_video_frames=num_video_frames, + use_native_fps=use_native_fps, + use_original_fps=use_original_fps, + use_random_consecutive_frames=use_random_consecutive_frames, + ) + + # Modify only the allowed_num_multiplers parameter + augmentations["video_parsing"].args["allowed_num_multiplers"] = [1, 2] + + log.info( + "[video] nocameramove_video_augmentor_v1: Modified allowed_num_multiplers to [1, 2] " + "for limited frame dropping capability" + ) + return augmentations + + +@augmentor_register("image_basic_augmentor") +def get_image_augmentor( + resolution: str, + caption_type: str = "ai_v3p1", + embedding_type: str = "t5_xxl", + dataset_resolution_type: str = "all", +): + augmentation = { + "image_resolution_filter": L(image_resolution_filter.ImageResolutionFilter)( + input_keys=["images"], + args={"dataset_resolution_type": dataset_resolution_type, "image_key": "images"}, + ), + "resize_largest_side_aspect_ratio_preserving": L(resize.ResizeLargestSideAspectPreserving)( + input_keys=["images"], + args={"size": IMAGE_RES_SIZE_INFO[resolution]}, + ), + "reflection_padding": L(padding.ReflectionPadding)( + input_keys=["images"], + args={"size": IMAGE_RES_SIZE_INFO[resolution]}, + ), + "normalize": L(normalize.Normalize)( + input_keys=["images"], + args={"mean": 0.5, "std": 0.5}, + ), + "text_transform": L(text_transforms_for_image.TextTransformForImage)( + input_keys=[], + args={ + "caption_type": caption_type, + "embedding_type": embedding_type, + "weight_captions_gt": 0.05, + "caption_probs": {"ground_truth": 0.05, "vfc_fidelity": 0.95}, + "t5_tokens": {"num": 512, "dim": 1024}, + "is_mask_all_ones": True, + }, + ), + "append_fps_frames": L(append_fps_frames_for_image.AppendFPSFramesForImage)(), + } + + return augmentation + + +@augmentor_register("image_basic_augmentor_without_embeddings") +def get_image_augmentor_without_embeddings( + resolution: str, + caption_type: str = "ai_v3p1", + embedding_type: Optional[str] = None, + train_on_captions: list[str] = [], + dataset_resolution_type: str = "all", +): + augmentation = { + "image_resolution_filter": L(image_resolution_filter.ImageResolutionFilter)( + input_keys=["images"], + args={"dataset_resolution_type": dataset_resolution_type, "image_key": "images"}, + ), + "resize_largest_side_aspect_ratio_preserving": L(resize.ResizeLargestSideAspectPreserving)( + input_keys=["images"], + args={"size": IMAGE_RES_SIZE_INFO[resolution]}, + ), + "reflection_padding": L(padding.ReflectionPadding)( + input_keys=["images"], + args={"size": IMAGE_RES_SIZE_INFO[resolution]}, + ), + "normalize": L(normalize.Normalize)( + input_keys=["images"], + args={"mean": 0.5, "std": 0.5}, + ), + "text_transform": L(text_transforms_for_image.TextTransformForImageWithoutEmbeddings)( + input_keys=[], + args={ + "caption_type": caption_type, + "train_on_captions": train_on_captions, + }, + ), + "append_fps_frames": L(append_fps_frames_for_image.AppendFPSFramesForImage)(), + } + + return augmentation + + +@augmentor_register("image_basic_augmentor_with_tokenization") +def image_basic_augmentor_with_tokenization( + resolution: str, + caption_type: str = "ai_v3p1", + embedding_type: Optional[str] = None, + tokenizer_config: Optional[LazyDict] = None, + cfg_dropout_rate: float = 0.0, + append_resolution_info: bool = False, + use_system_prompt: bool = False, + train_on_captions: list[str] | None = None, + dataset_resolution_type: str = "all", + **kwargs, +): + """Build an image augmentation pipeline with resizing, captioning, and tokenization. + + Args: + resolution: Target resolution key (e.g., "256p", "512p"). Looked up in + IMAGE_RES_SIZE_INFO to determine the resize target size. + caption_type: Caption field name to extract from the data sample + (e.g., "ai_v3p1", "ai_caption"). + embedding_type: Unused. Kept for interface compatibility with other augmentors. + tokenizer_config: Lazy config for the text tokenizer. Passed to TextTokenizerTransform + to convert captions into token IDs. + cfg_dropout_rate: Probability of dropping the caption (replacing with empty string) + for classifier-free guidance training. + append_resolution_info: If True, appends a resolution metadata string + (e.g., "This image is 512x512.") to the caption. + use_system_prompt: If True, prepends a system prompt to the tokenized caption + that instructs the model it is an image generation assistant. + train_on_captions: If non-empty, only use these caption types (e.g. ["dense"], ["qwen3vl_30B_v1_dense"]). + If empty, caption type is inferred from the data. + + Returns: + dict: Ordered dictionary of augmentation stage name to LazyCall config. + """ + augmentation = { + "image_resolution_filter": L(image_resolution_filter.ImageResolutionFilter)( + input_keys=["images"], + args={"dataset_resolution_type": dataset_resolution_type, "image_key": "images"}, + ), + "resize_largest_side_aspect_ratio_preserving": L(resize.ResizeLargestSideAspectPreserving)( + input_keys=["images"], + args={"size": IMAGE_RES_SIZE_INFO[resolution]}, + ), + "reflection_padding": L(padding.ReflectionPadding)( + input_keys=["images"], + args={"size": IMAGE_RES_SIZE_INFO[resolution]}, + ), + "normalize": L(normalize.Normalize)( + input_keys=["images"], + args={"mean": 0.5, "std": 0.5}, + ), + "text_transform": L(text_transforms_for_image.TextTransformForImageWithoutEmbeddings)( + input_keys=[], + args={ + "caption_type": caption_type, + "train_on_captions": train_on_captions or [], + "caption_prefix": kwargs.get("caption_prefix", None), + }, + ), + # Resolution info augmentor - appends metadata like "This image is 512x512." + # Reads final_height/final_width from CropToMultiple (required). + "resolution_info": L(resolution_text_info.ResolutionTextInfo)( + input_keys=["ai_caption", "images", "image_size"], + output_keys=[ + "ai_caption", + ], + args={ + "caption_key": "ai_caption", + "image_key": "images", + "enabled": append_resolution_info, + }, + ), + "text_tokenization": L(text_tokenizer.TextTokenizerTransform)( + input_keys=["ai_caption"], + output_keys=["text_token_ids"], + args={ + "tokenizer_config": tokenizer_config, + "cfg_dropout_rate": cfg_dropout_rate, + "use_system_prompt": use_system_prompt, + }, + ), + "append_fps_frames": L(append_fps_frames_for_image.AppendFPSFramesForImage)(), + } + + return augmentation + + +@augmentor_register("image_basic_augmentor_json_caption") +def image_basic_augmentor_json_caption( + resolution: str, + caption_type: str = "ai_v3p1", + caption_config: dict | None = None, + embedding_type: Optional[str] = None, + tokenizer_config: Optional[LazyDict] = None, + cfg_dropout_rate: float = 0.0, + append_resolution_info: bool = False, + use_system_prompt: bool = False, + train_on_captions: list[str] | None = None, + dataset_resolution_type: str = "all", + **kwargs, +): + """Build an image augmentation pipeline with resizing, json captioning, and tokenization. + + Args: + resolution: Target resolution key (e.g., "256p", "512p"). Looked up in + IMAGE_RES_SIZE_INFO to determine the resize target size. + caption_type: Caption field name to extract from the data sample + (e.g., "ai_v3p1", "ai_caption"). + embedding_type: Unused. Kept for interface compatibility with other augmentors. + tokenizer_config: Lazy config for the text tokenizer. Passed to TextTokenizerTransform + to convert captions into token IDs. + cfg_dropout_rate: Probability of dropping the caption (replacing with empty string) + for classifier-free guidance training. + append_resolution_info: If True, appends a resolution metadata string + (e.g., "This image is 512x512.") to the caption. + use_system_prompt: If True, prepends a system prompt to the tokenized caption + that instructs the model it is an image generation assistant. + train_on_captions: If non-empty, only use these caption types (e.g. ["dense"], ["qwen3vl_30B_v1_dense"]). + If empty, caption type is inferred from the data. + + Returns: + dict: Ordered dictionary of augmentation stage name to LazyCall config. + """ + assert caption_config is not None and "json_field_dropout_rate" in caption_config, ( + "image_basic_augmentor_json_caption requires caption_config with " + "'json_field_dropout_rate'. Set it in your experiment config, e.g. " + "caption_config={'json_field_dropout_rate': 0.05}." + ) + augmentation = { + "image_resolution_filter": L(image_resolution_filter.ImageResolutionFilter)( + input_keys=["images"], + args={"dataset_resolution_type": dataset_resolution_type, "image_key": "images"}, + ), + "resize_largest_side_aspect_ratio_preserving": L(resize.ResizeLargestSideAspectPreserving)( + input_keys=["images"], + args={"size": IMAGE_RES_SIZE_INFO[resolution]}, + ), + "reflection_padding": L(padding.ReflectionPadding)( + input_keys=["images"], + args={"size": IMAGE_RES_SIZE_INFO[resolution]}, + ), + "normalize": L(normalize.Normalize)( + input_keys=["images"], + args={"mean": 0.5, "std": 0.5}, + ), + "text_transform": L(text_transforms_for_image.TextTransformForImageJsonCaption)( + input_keys=[], + args={ + "caption_type": caption_type, + "json_field_dropout_rate": caption_config["json_field_dropout_rate"], + }, + ), + # Resolution info augmentor - appends metadata like "This image is 512x512." + # Reads final_height/final_width from CropToMultiple (required). + "resolution_info": L(resolution_text_info.ResolutionTextInfo)( + input_keys=["ai_caption", "images", "image_size"], + output_keys=[ + "ai_caption", + ], + args={ + "caption_key": "ai_caption", + "image_key": "images", + "enabled": append_resolution_info, + }, + ), + "text_tokenization": L(text_tokenizer.TextTokenizerTransform)( + input_keys=["ai_caption"], + output_keys=["text_token_ids"], + args={ + "tokenizer_config": tokenizer_config, + "cfg_dropout_rate": cfg_dropout_rate, + "use_system_prompt": use_system_prompt, + }, + ), + "append_fps_frames": L(append_fps_frames_for_image.AppendFPSFramesForImage)(), + } + + return augmentation diff --git a/cosmos_framework/data/vfm/augmentors/__init__.py b/cosmos_framework/data/vfm/augmentors/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/data/vfm/augmentors/append_fps_frames_for_image.py b/cosmos_framework/data/vfm/augmentors/append_fps_frames_for_image.py new file mode 100644 index 0000000..2198d07 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/append_fps_frames_for_image.py @@ -0,0 +1,25 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Optional + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor + + +class AppendFPSFramesForImage(Augmentor): + def __init__( + self, input_keys: Optional[list] = None, output_keys: Optional[list] = None, args: Optional[dict] = None + ) -> None: + super().__init__(input_keys, output_keys, args) + + def __call__(self, data_dict: dict) -> dict: + r"""Remove the input keys from the data dict. + + Args: + data_dict (dict): Input data dict + Returns: + data_dict (dict): Output dict with keys removed. + """ + data_dict["fps"] = 30.0 # set image model fps = 30, which is the most common fps we used to train video. + data_dict["num_frames"] = 1 + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/audio_caption.py b/cosmos_framework/data/vfm/augmentors/audio_caption.py new file mode 100644 index 0000000..97c0cae --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/audio_caption.py @@ -0,0 +1,95 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Augmentor that appends audio captions to video captions. + +Reads an audio caption from the metadata JSON and appends it to the existing +video caption string before tokenization. This allows the model to condition +on both visual and audio descriptions. + +Placed AFTER text_transform (which sets ai_caption) and BEFORE text_tokenization. +""" + +import sys + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log + + +def _debug(msg: str) -> None: + """Write debug message to stderr (unbuffered, reliable in worker processes).""" + sys.stderr.write(f"[AudioCaptionAppender] {msg}\n") + sys.stderr.flush() + + +class AudioCaptionAppender(Augmentor): + """Appends audio caption text from metadata to the video caption. + + Args: + input_keys: Expected to be ["metas", "ai_caption"] but read from data_dict directly. + output_keys: Not used. + args: Dictionary with: + - audio_caption_key: Metadata key for audio caption (default: "audio_caption") + - separator: Text inserted between video and audio captions (default: " Audio description: ") + - sound_key: Key to check if sound data exists (default: "sound") + """ + + def __init__(self, input_keys: list, output_keys: list | None = None, args: dict | None = None) -> None: + super().__init__(input_keys, output_keys, args) + args = args or {} + self.audio_caption_key = args.get("audio_caption_key", "caption_audio") + self.separator = args.get("separator", " Audio description: ") + self.sound_key = args.get("sound_key", "sound") + self.caption_key = "ai_caption" + log.warning( + f"AudioCaptionAppender initialized: audio_caption_key='{self.audio_caption_key}', " + f"sound_key='{self.sound_key}', metas_key='{input_keys[0]}'", + rank0_only=True, + ) + + def _find_audio_caption(self, meta_dict: dict) -> str | None: + """Find audio caption in metas, supporting both flat and nested formats. + + Flat format (e.g., metas_w_audio_caps): + {"caption_audio": "...", ...} + + Nested format (e.g., midtrain dataset): + {"0_156": {"caption_sound": "..."}, ...} + The key is a frame range like "0_156" containing a dict with "caption_sound". + """ + # Try flat key first + value = meta_dict.get(self.audio_caption_key) + if isinstance(value, str) and len(value) > 0: + return value + + # Try nested: look for a dict value containing "caption_sound" + for key, val in meta_dict.items(): + if isinstance(val, dict) and "caption_sound" in val: + caption = val["caption_sound"] + if isinstance(caption, str) and len(caption) > 0: + return caption + + return None + + def __call__(self, data_dict: dict) -> dict | None: + """Append audio caption to the video caption if available. + + Only appends when sound data is present in the sample. If the metadata + does not contain the audio_caption_key, the video caption is left unchanged. + Always cleans up metas from data_dict since this is the last augmentor that reads it. + """ + metas_key = self.input_keys[0] + has_sound = self.sound_key in data_dict and data_dict.get(self.sound_key) is not None + meta_dict = data_dict.get(metas_key) + + if has_sound and meta_dict is not None: + audio_caption = self._find_audio_caption(meta_dict) + if isinstance(audio_caption, str) and len(audio_caption) > 0: + current_caption = data_dict.get(self.caption_key, "") + data_dict[self.caption_key] = current_caption + self.separator + audio_caption + + # Clean up metas from data_dict — this augmentor is the last consumer of metas + if metas_key in data_dict: + del data_dict[metas_key] + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/audio_parsing.py b/cosmos_framework/data/vfm/augmentors/audio_parsing.py new file mode 100644 index 0000000..779ed52 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/audio_parsing.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Audio parsing augmentor for T2A (Text-to-Audio) datasets. + +For audio-only datasets (AudioCaps, WavCaps, etc.) that have no video, +this augmentor: +1. Decodes audio from bytes +2. Creates a full-length dummy video (all zeros) matching the audio duration +3. Outputs data compatible with the v3 video training pipeline + +The dummy video ensures compatibility with the model architecture which +requires vision tokens in the sequence (sound_gen requires vision_gen). +The dummy video is fully conditioned (all frames clean), so it contributes +no loss — effectively making this a tv2s (text+video→sound) mode where +the video is a placeholder. +""" + +from typing import Optional + +import torch +from torchcodec.decoders import AudioDecoder + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log + + +class AudioParsingForFullClips(Augmentor): + """Audio parsing augmentor for audio-only datasets. + + Loads audio from bytes, creates a dummy video of matching duration, + and outputs data compatible with the VideoParsingWithFullFrames pipeline. + + Args: + input_keys: [meta_key, audio_key] — keys to fetch metadata and audio bytes + output_keys: Optional output keys + args: Dictionary with: + - target_sample_rate: Target audio sample rate (default: 48000) + - target_channels: Target audio channels (default: 2 for stereo) + - dummy_video_fps: FPS for dummy video (default: 24) + - dummy_video_size: (H, W) for dummy video (default: (256, 256)) + - max_audio_duration_sec: Max audio duration in seconds (default: 30.0) + - min_audio_duration_sec: Min audio duration in seconds (default: 1.0) + """ + + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + assert len(input_keys) == 2, "AudioParsingForFullClips requires two input keys: [meta_key, audio_key]" + self.meta_key = input_keys[0] + self.audio_key = input_keys[1] + + self.target_sample_rate = args.get("target_sample_rate", 48000) + self.target_channels = args.get("target_channels", 2) + self.dummy_video_fps = args.get("dummy_video_fps", 24.0) + self.dummy_video_size = args.get("dummy_video_size", (256, 256)) + self.max_audio_duration_sec = args.get("max_audio_duration_sec", 30.0) + self.min_audio_duration_sec = args.get("min_audio_duration_sec", 1.0) + + def __call__(self, data_dict: dict) -> dict | None: + try: + meta_dict = data_dict[self.meta_key] + audio_bytes = data_dict[self.audio_key] + except Exception: + log.warning( + f"Cannot find audio data. url: {data_dict.get('__url__', '?')}, key: {data_dict.get('__key__', '?')}", + rank0_only=False, + ) + return None + + if not isinstance(audio_bytes, bytes): + log.warning("Audio data is not bytes, skipping", rank0_only=False) + return None + + # Decode audio + try: + audio_decoder = AudioDecoder(audio_bytes) + audio_metadata = audio_decoder.metadata + orig_sample_rate = audio_metadata.sample_rate + + audio_samples = audio_decoder.get_samples_played_in_range() + audio_chunk = audio_samples.data # [C,N_orig] + del audio_decoder + except Exception as e: + log.warning(f"Failed to decode audio: {e}", rank0_only=False) + return None + + # Compute duration + audio_duration_sec = audio_chunk.shape[1] / orig_sample_rate + + # Filter by duration + if audio_duration_sec < self.min_audio_duration_sec: + log.debug(f"Audio too short: {audio_duration_sec:.2f}s < {self.min_audio_duration_sec}s", rank0_only=False) + return None + if audio_duration_sec > self.max_audio_duration_sec: + # Crop to max duration + max_samples = int(self.max_audio_duration_sec * orig_sample_rate) + audio_chunk = audio_chunk[:, :max_samples] + audio_duration_sec = self.max_audio_duration_sec + + # Resample if needed + if orig_sample_rate != self.target_sample_rate: + import torchaudio + + audio_chunk = torchaudio.functional.resample( + audio_chunk, orig_freq=orig_sample_rate, new_freq=self.target_sample_rate + ) # [C,N_resampled] + + # Handle channel count (mono → stereo or vice versa) + if audio_chunk.shape[0] == 1 and self.target_channels == 2: + audio_chunk = audio_chunk.repeat(2, 1) # [2,N_resampled] + elif audio_chunk.shape[0] > self.target_channels: + audio_chunk = audio_chunk[: self.target_channels] # [C_target,N_resampled] + + # Create dummy video matching audio duration + # VAE compress temporal by 4x, with 1 as condition → num_frames must be 1 + 4N + num_video_frames = int(audio_duration_sec * self.dummy_video_fps) + N = (num_video_frames - 1) // 4 + num_video_frames = max(1 + 4 * N, 1) + + h, w = self.dummy_video_size + dummy_video = torch.zeros(3, num_video_frames, h, w, dtype=torch.uint8) # [3,T,H,W] + + # Build output compatible with VideoParsingWithFullFrames + video_info = { + "frame_start": 0, + "frame_end": num_video_frames - 1, + "num_frames": num_video_frames, + "video": dummy_video, + "fps": self.dummy_video_fps, + "conditioning_fps": self.dummy_video_fps, + "n_orig_video_frames": num_video_frames, + "sound": audio_chunk, + "audio_sample_rate": self.target_sample_rate, + } + data_dict["video"] = video_info + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/caption_filter.py b/cosmos_framework/data/vfm/augmentors/caption_filter.py new file mode 100644 index 0000000..07a9f8c --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/caption_filter.py @@ -0,0 +1,161 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Optional + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log + + +class CaptionFilter(Augmentor): + """ + Caption filter augmentor for predict2 training. + + This augmentor filters video samples based on caption content with configurable behavior: + - contain_keyword=True: Only return videos that contain keywords in captions + - contain_keyword=False: Only return videos that do NOT contain keywords in captions + + When a sample doesn't match the filter criteria, it returns None, which causes + the webdataset pipeline to skip that sample and continue to the next one. + """ + + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + """ + Initialize the caption filter. + + Args: + input_keys: List containing the caption key (e.g., ["ai_caption"] or text embeddings key) + output_keys: Not used for filtering, can be None + args: Dictionary with filtering parameters: + - "keywords": List of keywords to filter by (e.g., ["camera pan"]) + - "contain_keyword": Boolean flag for filtering behavior: + * True: Only return videos that contain keywords + * False: Only return videos that do NOT contain keywords + - "log_filtered": Whether to log filtered samples (default: False) + - "filter_stats": Whether to track filtering statistics (default: True) + - "dont_apply_on_webdataset_names": List of webdataset names to not apply the filter on, it will just pass through without checking contain or not contain keywords + """ + super().__init__(input_keys, output_keys, args) + + # Parse arguments + if args is None: + args = {} + + self.keywords = args.get("keywords", []) + self.contain_keyword = args.get("contain_keyword", False) # Default to exclude mode + self.log_filtered = args.get("log_filtered", False) + self.filter_stats = args.get("filter_stats", True) + self.dont_apply_on_webdataset_names = args.get("dont_apply_on_webdataset_names", []) + + # Validate input_keys + if not input_keys or len(input_keys) == 0: + raise ValueError("CaptionFilter requires at least one input key for the caption field") + + self.caption_key = input_keys[0] # Use the first input key as the caption key + + # Statistics tracking + if self.filter_stats: + self.total_samples = 0 + self.filtered_samples = 0 + + # Validate configuration + if not self.keywords: + log.warning("CaptionFilter: No keywords provided, filter will not filter any samples") + + mode_str = "contain" if self.contain_keyword else "exclude" + log.info( + f"CaptionFilter initialized in '{mode_str}' mode with {len(self.keywords)} keywords using caption key '{self.caption_key}': {self.keywords}" + ) + + def __call__(self, data_dict: dict) -> Optional[dict]: + """ + Filter data based on caption content. + + This checks the caption field specified by the input_keys parameter. + Depending on contain_keyword flag: + - True: Returns data_dict only if caption contains any keyword, None otherwise + - False: Returns data_dict only if caption contains NO keywords, None otherwise + + Args: + data_dict: Input data dictionary containing the caption field specified in input_keys + + Returns: + data_dict: Original data dict if caption passes filter + None: If caption should be filtered out (causes sample to be skipped) + """ + data_dict_root = data_dict["__url__"].root + if any(n in data_dict_root for n in self.dont_apply_on_webdataset_names): + return data_dict + + if self.filter_stats: + self.total_samples += 1 + + # Check if caption key exists + if self.caption_key not in data_dict: + if self.log_filtered: + log.warning(f"CaptionFilter: No '{self.caption_key}' found in data_dict, passing through") + return data_dict + + caption = data_dict[self.caption_key] + if not isinstance(caption, str) or not caption.strip(): + if self.log_filtered: + log.warning(f"CaptionFilter: '{self.caption_key}' is empty or not a string, got {type(caption)}") + return data_dict + + # Check if any keywords are found in the caption + search_caption = caption.lower() + keyword_found = False + matched_keyword = None + + for keyword in self.keywords: + if keyword.lower() in search_caption: + keyword_found = True + matched_keyword = keyword + break + + # Apply filtering logic based on contain_keyword flag + should_filter = False + if self.contain_keyword: + # Include mode: filter out if NO keywords found + should_filter = not keyword_found + else: + # Exclude mode: filter out if ANY keyword found + should_filter = keyword_found + + if should_filter: + if self.log_filtered: + if self.contain_keyword: + log.info(f"CaptionFilter: excluded sample (no keywords found) - caption: '{caption[:100]}...'") + else: + log.info( + f"CaptionFilter: excluded sample due to keyword '{matched_keyword}' - caption: '{caption[:100]}...'" + ) + + if self.filter_stats: + self.filtered_samples += 1 + return None + + # Sample passes filter + return data_dict + + def get_filter_stats(self) -> dict: + """ + Get filtering statistics. + + Returns: + Dictionary with filtering statistics + """ + if not self.filter_stats: + return {"stats_disabled": True} + + filter_rate = (self.filtered_samples / self.total_samples * 100) if self.total_samples > 0 else 0 + mode_str = "contain" if self.contain_keyword else "exclude" + + return { + "total_samples": self.total_samples, + "filtered_samples": self.filtered_samples, + "passed_samples": self.total_samples - self.filtered_samples, + "filter_rate_percent": filter_rate, + "mode": mode_str, + "keywords": self.keywords, + } diff --git a/cosmos_framework/data/vfm/augmentors/cropping.py b/cosmos_framework/data/vfm/augmentors/cropping.py new file mode 100644 index 0000000..ce65744 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/cropping.py @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Optional + +import torch +import torchvision.transforms.functional as transforms_F +from PIL import Image + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor + + +class CropToMultiple(Augmentor): + """Crops images/videos to the nearest multiple of a specified value using center crop. + + This augmentor crops the height and width of images/videos to be divisible by + a given multiple (default 16). The crop is centered, removing equal amounts + from opposite edges. + + Supports: + - PIL Images (for image data) + - Torch tensors with shape (C, H, W) or (C, T, H, W) (for video data) + + Example: + Input: 209x187 with multiple=16 + Output: 208x176 (center cropped to nearest lower multiple of 16) + """ + + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + self.multiple = 16 + if self.args is not None and "multiple" in self.args: + self.multiple = self.args["multiple"] + + def __call__(self, data_dict: dict) -> dict: + """Center crops images/videos to the nearest multiple of the specified value. + + Args: + data_dict (dict): Input data dict containing images/videos to crop. + + Returns: + data_dict (dict): Output dict with center cropped images/videos. + """ + for key in self.input_keys: + if key not in data_dict: + continue + + data = data_dict[key] + + # Get dimensions based on data type + if isinstance(data, Image.Image): + # PIL Image: size returns (width, height) + w, h = data.size + elif isinstance(data, torch.Tensor): + # Torch tensor: (C, H, W) or (C, T, H, W) + if data.ndim == 3: + _, h, w = data.shape + elif data.ndim == 4: + _, _, h, w = data.shape + else: + raise ValueError(f"Unexpected tensor dimensions: {data.ndim}, expected 3 or 4") + else: + raise ValueError(f"Unexpected data type: {type(data)}, expected PIL Image or torch Tensor") + + # Calculate new dimensions (nearest lower multiple) + new_h = (h // self.multiple) * self.multiple + new_w = (w // self.multiple) * self.multiple + + # Center crop: calculate offsets to center the crop + if new_h != h or new_w != w: + top = (h - new_h) // 2 + left = (w - new_w) // 2 + # log.info(f"Data cropped from ({h}, {w}) to ({new_h}, {new_w})") + data_dict[key] = transforms_F.crop(data, top=top, left=left, height=new_h, width=new_w) + + # Store final dimensions for downstream use (e.g., resolution text info) + data_dict["final_height"] = new_h + data_dict["final_width"] = new_w + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/duration_fps_text_timestamps.py b/cosmos_framework/data/vfm/augmentors/duration_fps_text_timestamps.py new file mode 100644 index 0000000..590b5a1 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/duration_fps_text_timestamps.py @@ -0,0 +1,123 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Optional + +import torch + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log + +# Global template for duration and FPS text timestamps +DEFAULT_TEMPLATE = "The video is {duration:.1f} seconds long and is of {fps:.0f} FPS." + + +class DurationFPSTextTimeStamps(Augmentor): + """ + Augmentor that appends video duration and FPS as text timestamps to captions. + + This augmentor should run AFTER TextTransformForVideo to append metadata + to the already-selected caption in data_dict["ai_caption"]. + + IMPORTANT: Reads num_frames from the actual video tensor shape to get the + FINAL frame count after all video processing (subsampling, etc.) is complete. + + Example: + Original caption: "A cat playing with a ball" + Augmented caption: "A cat playing with a ball. The video is 1.4 seconds long and is of 24 FPS" + + Args: + input_keys (list): Input keys (not used, kept for API compatibility) + output_keys (list): Output keys (not used, kept for API compatibility) + args (dict): Configuration arguments: + - caption_key (str): Key for caption in data_dict. Default: "ai_caption" + - video_key (str): Key for video tensor in data_dict. Default: "video" + - fps_key (str): Key for FPS value in data_dict. Default: "conditioning_fps" + - template (str): Format string for metadata text. Default: DEFAULT_TEMPLATE constant + - separator (str): Separator between caption and metadata. Default: ". " + - enabled (bool): Whether augmentation is enabled. Default: True + - skip_on_error (bool): If True, skip on errors and return original data_dict. If False, return None. Default: True + - num_multiplier_key (str): Key for num_multiplier value in data_dict. Default: "num_multiplier" + """ + + def __init__( + self, input_keys: Optional[list] = None, output_keys: Optional[list] = None, args: Optional[dict] = None + ) -> None: + super().__init__(input_keys, output_keys, args) + + # Configuration with sensible defaults + self.caption_key = args.get("caption_key", "ai_caption") if args else "ai_caption" + self.video_key = args.get("video_key", "video") if args else "video" + self.fps_key = args.get("fps_key", "conditioning_fps") if args else "conditioning_fps" + self.template = args.get("template", DEFAULT_TEMPLATE) if args else DEFAULT_TEMPLATE + self.default_separator = args.get("separator", ". ") if args else ". " + self.enabled = args.get("enabled", True) if args else True + self.skip_on_error = args.get("skip_on_error", True) if args else True + self.num_multiplier_key = args.get("num_multiplier_key", "num_multiplier") if args else "num_multiplier" + + def __call__(self, data_dict: dict) -> dict | None: + """ + Append video duration and FPS as text timestamps to the caption. + + Args: + data_dict (dict): Input data dict containing caption, fps, and video tensor + + Returns: + data_dict (dict): Output dict with augmented caption, or None if error and skip_on_error=False + """ + if not self.enabled: + return data_dict + # Get caption - must exist at this point (set by TextTransformForVideo) + if self.caption_key not in data_dict: + if self.skip_on_error: + log.warning( + f"DurationFPSTextTimeStamps: '{self.caption_key}' not found in data_dict. Skipping.", + rank0_only=False, + ) + return data_dict + else: + return None + caption = data_dict[self.caption_key] + if (not isinstance(caption, str) and not isinstance(caption, dict)) or caption == "": + if self.skip_on_error: + return data_dict + else: + return None + + # Use pre-calculated conditioning_fps from VideoParsing augmentor + # This already accounts for frame skipping (fps / num_multiplier) + fps_value = data_dict[self.fps_key] + if isinstance(fps_value, torch.Tensor): + fps = fps_value.item() if fps_value.numel() == 1 else fps_value[0].item() + else: + fps = float(fps_value) + + # Extract ACTUAL number of frames from the video tensor shape + # This is critical - we need the final frame count after all processing + video = data_dict[self.video_key] + + # Video shape is (C, T, H, W) + num_frames = video.shape[1] + + # Compute duration and append to caption + if fps > 0: + duration = int(num_frames / fps) + if isinstance(caption, str): + # Case 1: Caption is a string (existing behavior). + metadata_text = self.template.format(duration=duration, fps=fps) + + # Choose separator based on whether caption ends with a period + separator = " " if caption.rstrip().endswith(".") else self.default_separator + + # Update caption text + data_dict[self.caption_key] = caption + separator + metadata_text + elif isinstance(caption, dict): + # Case 2: Caption is JSON. Add structured duration/FPS fields. + data_dict[self.caption_key].update( + { + "duration": str(duration) + "s", + "fps": fps, + } + ) + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/idle_frames_text_info.py b/cosmos_framework/data/vfm/augmentors/idle_frames_text_info.py new file mode 100644 index 0000000..302715c --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/idle_frames_text_info.py @@ -0,0 +1,180 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Augmentor that appends idle-frame count metadata to the caption. + +The label is a Pi0.7-style episode-metadata field encoded as plain text. It +records how many frames of the action chunk were "idle" out of the total action +frames (i.e. the relative-pose delta is close to identity and the gripper +command does not change). The upstream dataset is responsible for populating +``data_dict[idle_frames_key]`` via +:func:`projects.cosmos3.vfm.datasets.action.pose_utils.compute_idle_frames`. + +Per-field dropout (default 5%) is applied here, matching Pi0.7's approach of +independently dropping each metadata component. This is complementary to the +global ``cfg_dropout_rate`` in :class:`TextTokenizerTransform`, which still +empties the whole caption. +""" + +from __future__ import annotations + +import random + +import torch + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log + +DEFAULT_TEMPLATE = "IdleFrames: {n} out of {m}." +FALLBACK_TEMPLATE = "IdleFrames: {n}." + + +class IdleFramesTextInfo(Augmentor): + """Augmentor that appends ``IdleFrames: N out of M.`` to the caption. + + Reads ``data_dict[idle_frames_key]`` (set by the dataset layer) and appends + a textual marker to the caption, modeled after + :class:`ResolutionTextInfo` and :class:`DurationFPSTextTimeStamps`. + + Per-field dropout is supported: with probability ``dropout_rate`` the + segment is omitted entirely (the caption is left unchanged). This is + independent from the global classifier-free-guidance dropout in the + tokenizer. + + Example: + Original caption: "pick up the cup" + Augmented: "pick up the cup. IdleFrames: 0 out of 16." + + Args: + input_keys (list): Input keys (not used, kept for API compatibility). + output_keys (list): Output keys (not used, kept for API compatibility). + args (dict): Configuration arguments: + - caption_key (str): Key for caption in data_dict. Default ``"ai_caption"``. + - idle_frames_key (str): Key for the idle-frame integer in data_dict. + Default ``"idle_frames"``. + - total_frames_key (str): Optional key for the total frame integer + in data_dict. Default ``"idle_frames_total"``. + - action_key (str): Key for the action tensor used to infer total + frames when ``total_frames_key`` is missing. Default ``"action"``. + - template (str): Format string for the appended segment. + Default ``"IdleFrames: {n} out of {m}."``. + - separator (str): Separator inserted between the original caption + and the new segment. Default ``". "``. + - dropout_rate (float): Probability of skipping the append step + (per-field dropout). Default 0.05. + - enabled (bool): Whether the augmentor is active. Default True. + """ + + def __init__( + self, + input_keys: list | None = None, + output_keys: list | None = None, + args: dict | None = None, + ) -> None: + super().__init__(input_keys, output_keys, args) + + args = args or {} + self.caption_key: str = args.get("caption_key", "ai_caption") + self.idle_frames_key: str = args.get("idle_frames_key", "idle_frames") + self.total_frames_key: str = args.get("total_frames_key", "idle_frames_total") + self.action_key: str = args.get("action_key", "action") + self.template: str = args.get("template", DEFAULT_TEMPLATE) + self.default_separator: str = args.get("separator", ". ") + self.dropout_rate: float = float(args.get("dropout_rate", 0.05)) + self.enabled: bool = bool(args.get("enabled", True)) + + if not 0.0 <= self.dropout_rate <= 1.0: + raise ValueError(f"dropout_rate must be in [0, 1]; got {self.dropout_rate}") + + def _get_scalar_int(self, value: object, key: str) -> int | None: + """Parse an optional scalar integer metadata value.""" + + if value is None: + return None + + if isinstance(value, torch.Tensor): + if value.numel() != 1: + log.warning( + f"IdleFramesTextInfo: expected scalar tensor at '{key}', got shape {tuple(value.shape)}. Skipping.", + rank0_only=False, + ) + return None + return int(value.item()) + + try: + return int(value) + except (TypeError, ValueError): + log.warning( + f"IdleFramesTextInfo: expected integer-compatible value at " + f"'{key}', got {type(value).__name__}. Skipping.", + rank0_only=False, + ) + return None + + def _get_total_frames(self, data_dict: dict) -> int | None: + """Resolve the total action-frame count for the idle-frame text.""" + + total_frames = self._get_scalar_int(data_dict.get(self.total_frames_key), self.total_frames_key) + if total_frames is not None: + return total_frames + + action = data_dict.get(self.action_key) + if isinstance(action, torch.Tensor): + if action.ndim == 0: + log.warning( + f"IdleFramesTextInfo: expected action tensor at " + f"'{self.action_key}' to have a frame dimension. Skipping total frames.", + rank0_only=False, + ) + return None + return int(action.shape[0]) + + try: + return len(action) if action is not None else None + except TypeError: + return None + + def __call__(self, data_dict: dict) -> dict | None: + """Append ``IdleFrames: N out of M.`` to ``data_dict[caption_key]`` in place. + + Returns the input dict unchanged when: + + - the augmentor is disabled, + - the per-field dropout fires, + - ``idle_frames_key`` is missing or ``None`` (e.g. non-action sample), + - the caption is missing, empty, or not a string/dict (unconditional case). + + For dict-typed captions (the JSON-caption code path), the idle-frame + integer is added under ``"idle_frames"`` and the total count, when + available, is added under ``"idle_frames_total"``. + """ + if not self.enabled: + return data_dict + + if random.random() < self.dropout_rate: + return data_dict + + n = self._get_scalar_int(data_dict.get(self.idle_frames_key), self.idle_frames_key) + if n is None: + return data_dict + + m = self._get_total_frames(data_dict) + + if self.caption_key not in data_dict: + return data_dict + caption = data_dict[self.caption_key] + + if isinstance(caption, str): + if caption == "": + return data_dict + metadata_text = self.template.format(n=n, m=m) if m is not None else FALLBACK_TEMPLATE.format(n=n) + separator = " " if caption.rstrip().endswith(".") else self.default_separator + data_dict[self.caption_key] = caption + separator + metadata_text + elif isinstance(caption, dict): + data_dict[self.caption_key]["idle_frames"] = n + if m is not None: + data_dict[self.caption_key]["idle_frames_total"] = m + else: + return data_dict + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/image_editing_transform.py b/cosmos_framework/data/vfm/augmentors/image_editing_transform.py new file mode 100644 index 0000000..fdaaa40 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/image_editing_transform.py @@ -0,0 +1,370 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +""" +Augmentors for image editing tasks in the cosmos3 VFM pipeline. + +These augmentors process conversation-format image editing data and produce +the output format expected by the main training pipeline: + - images: List[torch.Tensor] (source + target images as a two-frame "video") + - image_size: List[torch.Tensor] + - ai_caption: List[str] + - selected_caption_type: List[str] + - fps: List[float] + - num_frames: List[int] + - dataset_name: str + - sequence_plan: SequencePlan +""" + +from __future__ import annotations + +import random + +import torch +import torchvision.transforms.functional as transforms_F +from PIL import Image + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.sequence_packing import SequencePlan + + +class ExtractImageEditingConversation(Augmentor): + """Extract and validate image editing conversation from standard annotation format. + + This augmentor processes the cosmos-interleaved conversation format for image editing: + - Validates that the conversation has exactly one round (user + assistant) + - User message must contain at least one image and text instruction + - Assistant message must contain exactly one image (the edited result) + - If multi-round conversation is found, only the first round is kept + + Input Format (from data_dict): + - texts: Dict containing "content" with conversation data + - mllm_media_list: Dict mapping image keys to PIL images (for understanding) + - diffusion_media_list: Dict mapping image keys to PIL images (for diffusion/VAE) + + Output Format (added to data_dict): + - source_image: PIL.Image (the input image for editing) + - target_image: PIL.Image (the edited output image) + - editing_instruction: str (the user's editing instruction) + """ + + def __init__( + self, + input_keys: list | None = None, + max_round: int = 1, + args: dict | None = None, + ) -> None: + super().__init__(input_keys or [], None, args) + self.max_round = max_round + + def __call__(self, data_dict: dict) -> dict | None: + """Extract image editing conversation. + + Args: + data_dict: Input data dictionary. + + Returns: + Updated data_dict with source_image, target_image, editing_instruction, + or None if the data is invalid. + """ + # Validate required keys + for required_key in ["mllm_media_list", "diffusion_media_list", "texts"]: + if required_key not in data_dict: + log.warning( + f"{required_key} not found in data_dict: {data_dict.get('__key__', 'unknown')}", + rank0_only=False, + ) + return None + + mllm_media_list = data_dict["mllm_media_list"] + diffusion_media_list = data_dict["diffusion_media_list"] + + # Get conversation content + try: + texts_content = data_dict["texts"].get("content") + if texts_content is None: + log.warning( + f"texts.content is None: {data_dict.get('__key__', 'unknown')}", + rank0_only=False, + ) + return None + + # Handle case where content is a list of conversation options + if isinstance(texts_content, list) and len(texts_content) > 0: + if isinstance(texts_content[0], list): + # Multiple conversation options, randomly select one + selected_conversations = random.choice(texts_content) + else: + selected_conversations = texts_content + else: + log.warning( + f"Unexpected texts.content format: {data_dict.get('__key__', 'unknown')}", + rank0_only=False, + ) + return None + except Exception as e: + log.warning( + f"Error accessing texts.content: {data_dict.get('__key__', 'unknown')}, {str(e)}", + rank0_only=False, + ) + return None + + # For image editing, we only keep the first round (user + assistant) + # Trim to first round if multiple rounds exist + if len(selected_conversations) > 2: + log.warning( + f"Multi-round conversation found ({len(selected_conversations)} messages), " + f"keeping only first round: {data_dict.get('__key__', 'unknown')}", + rank0_only=False, + ) + selected_conversations = selected_conversations[:2] + + if len(selected_conversations) < 2: + log.warning( + f"Expected at least 2 messages (user + assistant), got {len(selected_conversations)}: " + f"{data_dict.get('__key__', 'unknown')}", + rank0_only=False, + ) + return None + + # Validate roles: first must be user, second must be assistant + user_msg = selected_conversations[0] + assistant_msg = selected_conversations[1] + + if user_msg.get("role") != "user": + log.warning( + f"First message role is not 'user': {data_dict.get('__key__', 'unknown')}", + rank0_only=False, + ) + return None + + if assistant_msg.get("role") != "assistant": + log.warning( + f"Second message role is not 'assistant': {data_dict.get('__key__', 'unknown')}", + rank0_only=False, + ) + return None + + # Extract user content: must have at least one image and one text + user_content = user_msg.get("content", []) + if isinstance(user_content, str): + user_content = [{"type": "text", "text": user_content}] + + user_text_parts: list[str] = [] + user_image_key: str | None = None + + for item in user_content: + if not isinstance(item, dict): + continue + content_type = item.get("type") + if content_type == "text": + user_text_parts.append(item.get("text", "")) + elif content_type == "image": + if user_image_key is None: + user_image_key = item.get("image") + # If multiple user images, we only take the first one + + if user_image_key is None: + log.warning( + f"No image found in user message: {data_dict.get('__key__', 'unknown')}", + rank0_only=False, + ) + return None + + editing_instruction = " ".join(user_text_parts).strip() + if not editing_instruction: + log.warning( + f"No text instruction found in user message: {data_dict.get('__key__', 'unknown')}", + rank0_only=False, + ) + return None + + # Extract assistant content: must have exactly one image + assistant_content = assistant_msg.get("content", []) + if isinstance(assistant_content, str): + log.warning( + f"Assistant content is text-only (no image): {data_dict.get('__key__', 'unknown')}", + rank0_only=False, + ) + return None + + assistant_image_key: str | None = None + for item in assistant_content: + if not isinstance(item, dict): + continue + if item.get("type") == "image": + assistant_image_key = item.get("image") + break + + if assistant_image_key is None: + log.warning( + f"No image found in assistant message: {data_dict.get('__key__', 'unknown')}", + rank0_only=False, + ) + return None + + # Validate images exist in media lists + for media_key in [user_image_key, assistant_image_key]: + if media_key not in diffusion_media_list: + log.warning( + f"Image {media_key} not found in diffusion_media_list: {data_dict.get('__key__', 'unknown')}", + rank0_only=False, + ) + return None + + # Get PIL images + source_image = diffusion_media_list[user_image_key] + target_image = diffusion_media_list[assistant_image_key] + + # Handle video (list of frames) - use first frame + if isinstance(source_image, list): + source_image = source_image[0] if source_image else None + if isinstance(target_image, list): + target_image = target_image[0] if target_image else None + + if source_image is None or target_image is None: + log.warning( + f"Source or target image is None: {data_dict.get('__key__', 'unknown')}", + rank0_only=False, + ) + return None + + data_dict["source_image"] = source_image + data_dict["target_image"] = target_image + data_dict["editing_instruction"] = editing_instruction + + return data_dict + + +class ImageEditingToTrainingFormat(Augmentor): + """Convert extracted image editing data to the training-compatible format. + + This augmentor takes the source image, target image, and editing instruction + and produces the output format expected by the main training pipeline. + + Images are assumed to have been already resized by an upstream augmentor + (e.g. ``OmniInterleavedMediaResize``). This augmentor only normalises the + PIL images to tensors and assembles the remaining metadata fields. + + Input (from data_dict): + - source_image: PIL.Image (already resized by upstream augmentor) + - target_image: PIL.Image (already resized by upstream augmentor) + - editing_instruction: str + + Output (added to data_dict): + - images: list[torch.Tensor] — ``[source (C,H_s,W_s), target (C,H_t,W_t)]`` + - ai_caption: str + - selected_caption_type: str + - fps: float + - num_frames: int + - sequence_plan: SequencePlan + """ + + def __init__( + self, + input_keys: list | None = None, + mean: float = 0.5, + std: float = 0.5, + args: dict | None = None, + ) -> None: + super().__init__(input_keys or [], None, args) + self.mean = mean + self.std = std + + def _normalize_image(self, image: Image.Image) -> torch.Tensor: + """Convert PIL image to normalized tensor (C, H, W).""" + tensor = transforms_F.to_tensor(image) + tensor = transforms_F.normalize(tensor, mean=[self.mean] * 3, std=[self.std] * 3) + return tensor + + def __call__(self, data_dict: dict) -> dict | None: + """Convert image editing data to training format. + + Args: + data_dict: Input data dictionary with source_image, target_image, editing_instruction. + + Returns: + Updated data_dict with training-compatible fields, or None on error. + """ + source_image: Image.Image = data_dict.get("source_image") + target_image: Image.Image = data_dict.get("target_image") + editing_instruction: str = data_dict.get("editing_instruction", "") + + if source_image is None or target_image is None: + return None + + try: + # Normalize PIL images to tensors (upstream augmentor already handled resizing) + source_tensor = self._normalize_image(source_image) # [C,H_s,W_s] + target_tensor = self._normalize_image(target_image) # [C,H_t,W_t] + + # Store as list of tensors for the batch collation. + # Each image keeps its own spatial size; the model encodes them separately. + data_dict["images"] = [source_tensor, target_tensor] + + # Set text fields + data_dict["ai_caption"] = editing_instruction + data_dict["selected_caption_type"] = "editing_instruction" + + # Set metadata + data_dict["fps"] = 30.0 # Same as standard image training + data_dict["num_frames"] = 2 # Source + target = 2 frames + data_dict["image_size"] = [ + torch.tensor( + [source_image.height, source_image.width, source_image.height, source_image.width], + dtype=torch.float, + ), # [4] + torch.tensor( + [target_image.height, target_image.width, target_image.height, target_image.width], + dtype=torch.float, + ), # [4] + ] + # Set the dataset name if not already present + if "dataset_name" not in data_dict: + data_dict["dataset_name"] = "image_editing" + + # Build sequence plan for image editing. + # The number of vision items per sample (e.g. 2 for source + target) is tracked + # by GenerationDataClean.num_vision_items_per_sample (set in get_data_and_condition). + # In pack_input_sequence, all items except the last are fully conditioned; + # the last item uses condition_frame_indexes_vision ([] = fully generated). + data_dict["sequence_plan"] = SequencePlan( + has_text=True, + has_vision=True, + condition_frame_indexes_vision=[], # Target (last item) is fully generated + ) + + except Exception as e: + log.warning( + f"Error processing image editing data: {data_dict.get('__key__', 'unknown')}, {str(e)}", + rank0_only=False, + ) + return None + + return data_dict + + +class RemoveKeys(Augmentor): + """Remove specified keys from the data dictionary. + + This is useful for cleaning up intermediate keys that are not needed + downstream (e.g. raw PIL images, media lists) so that every remaining + value is a tensor, number, dict, or list — as required by the dataloader + collation. + + Args: + input_keys: Keys to remove from ``data_dict``. + """ + + def __init__( + self, + input_keys: list | None = None, + args: dict | None = None, + ) -> None: + super().__init__(input_keys or [], None, args) + + def __call__(self, data_dict: dict) -> dict: + for key in self.input_keys: + data_dict.pop(key, None) + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/image_resolution_filter.py b/cosmos_framework/data/vfm/augmentors/image_resolution_filter.py new file mode 100644 index 0000000..50fb7fe --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/image_resolution_filter.py @@ -0,0 +1,44 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Optional + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.data.vfm.utils import IMAGE_RES_SIZE_INFO + +# Map dataset_resolution_type to resolution tier key in IMAGE_RES_SIZE_INFO +_DATASET_RESOLUTION_TIER: dict[str, str] = {"gt480p": "480", "gt720p": "720", "gt1080p": "1080"} + + +class ImageResolutionFilter(Augmentor): + """ + Filters out image samples whose (width, height) are below the minimum for + the sample's aspect ratio when dataset_resolution_type is not "all". + Mirrors the resolution check used in video_parsing. + """ + + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + self.image_key = args.get("image_key", "images") if args else "images" + self.dataset_resolution_type = args.get("dataset_resolution_type", "all") if args else "all" + self.resolution_tier = _DATASET_RESOLUTION_TIER.get(self.dataset_resolution_type) + + def __call__(self, data_dict: dict) -> dict | None: + image = data_dict.get(self.image_key) + if image is None: + return data_dict + + # PIL Image has .size as (width, height) + width, height = image.size + + aspect_ratio: str | None = None + if "__url__" in data_dict: + aspect_ratio = data_dict["__url__"].meta.opts["aspect_ratio"] + + # If the resolution of the image is smaller than the minimum resolution for the aspect ratio, skip the sample. This will ensure that we do not upsample any image. + if self.resolution_tier is not None: + min_w, min_h = IMAGE_RES_SIZE_INFO[self.resolution_tier][aspect_ratio] + if width < min_w and height < min_h: + return None + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/interleaved_image_transform.py b/cosmos_framework/data/vfm/augmentors/interleaved_image_transform.py new file mode 100644 index 0000000..8faa972 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/interleaved_image_transform.py @@ -0,0 +1,266 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +""" +Visual transformation augmentors for Omni models. +""" + +import math +from typing import Dict, List, Optional + +import torch +import torchvision.transforms.functional as transforms_F +from PIL import Image + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.data.imaginaire.webdataset.augmentors.image.misc import obtain_image_size + + +class ResizeToPaddingDivisor(Augmentor): + """Resize images so that both width and height are multiples of padding_divisor.""" + + def __init__(self, input_keys: list, padding_divisor: int = 16) -> None: + super().__init__(input_keys) + self.padding_divisor = padding_divisor + + def __call__(self, data_dict: dict) -> dict: + """Resize images to the nearest multiple of padding_divisor. + + Args: + data_dict (dict): Input data dict + Returns: + data_dict (dict): Output dict with resized images and metadata + """ + + if self.output_keys is None: + self.output_keys = self.input_keys + + # Get original image size + orig_w, orig_h = obtain_image_size(data_dict, self.input_keys) + + # Calculate new dimensions as multiples of padding_divisor + new_w = math.ceil(orig_w / self.padding_divisor) * self.padding_divisor + new_h = math.ceil(orig_h / self.padding_divisor) * self.padding_divisor + + # Resize images + for inp_key, out_key in zip(self.input_keys, self.output_keys): + data_dict[out_key] = transforms_F.resize( + data_dict[inp_key], + size=(new_h, new_w), + interpolation=transforms_F.InterpolationMode.BICUBIC, + antialias=True, + ) + if out_key != inp_key: + del data_dict[inp_key] + + # Store image size information (new_h, new_w, orig_h, orig_w) + data_dict["image_size"] = torch.tensor([new_h, new_w, orig_h, orig_w], dtype=torch.float) # [4] + + return data_dict + + +class InterleavedMediaResize(Augmentor): + """Resizes interleaved media content (images and videos) for both diffusion and MLLM models. + + This augmentor processes mixed media content containing both images and videos, creating two + versions of each media item: one optimized for diffusion models and another for Multimodal + Large Language Models (MLLMs). It preserves aspect ratios while ensuring dimensions meet + specific constraints for each model type. + + The resizing process follows these steps: + 1. Maintains aspect ratio while ensuring no side exceeds the maximum allowed length + 2. Adjusts dimensions to be divisible by model-specific padding constants + 3. Uses high-quality LANCZOS resampling for optimal visual quality + + Args: + input_keys (List, optional): List containing the key to access media content in data_dict. + Must contain exactly one key. Defaults to ['media_list']. + max_diffusion_image_side_length (int, optional): Maximum side length for diffusion model + images. Defaults to 1024. + max_mllm_image_side_length (int, optional): Maximum side length for MLLM images. + Defaults to 768. + diffusion_image_padding_constant (int, optional): Divisor for diffusion model image + dimensions. Both width and height must be divisible by this value. Defaults to 16. + mllm_image_padding_constant (int, optional): Divisor for MLLM image dimensions. + Both width and height must be divisible by this value. Defaults to 28. + use_center_crop (bool, optional): If True, uses center cropping to ensure dimensions + are divisible by padding constants, avoiding distortion. If False, uses resizing + which may cause slight distortion. Defaults to False. + args (Optional[dict], optional): Additional arguments passed to parent class. + Defaults to None. + + Input Format: + The data_dict should contain a key (specified in input_keys) with value structured as: + { + "image_0": PIL.Image, # Single image + "image_1": PIL.Image, # Another single image + "video_0": List[PIL.Image], # Video as list of frames + "video_1": List[PIL.Image], # Another video + ... + } + + Output Format: + The method adds two new keys to data_dict: + - 'diffusion_media_content': Resized media for diffusion models + - 'mllm_media_content': Resized media for MLLMs + + Both follow the same structure as the input, with resized versions of each media item. + + Example: + >>> # Using resize (default, may cause slight distortion) + >>> augmentor = OmniInterleavedMediaResize( + ... input_keys=['media_list'], + ... max_diffusion_image_side_length=1024, + ... max_mllm_image_side_length=768 + ... ) + >>> + >>> # Using center crop (no distortion) + >>> augmentor_crop = OmniInterleavedMediaResize( + ... input_keys=['media_list'], + ... max_diffusion_image_side_length=1024, + ... max_mllm_image_side_length=768, + ... use_center_crop=True + ... ) + >>> + >>> data_dict = { + ... 'media_list': { + ... 'image_0': pil_image, + ... 'video_0': [frame1, frame2, frame3] + ... } + ... } + >>> result = augmentor(data_dict) + >>> # result now contains 'diffusion_media_content' and 'mllm_media_content' + + Note: + - Images are only scaled down, never up, to preserve quality + - Videos are processed frame by frame, maintaining temporal consistency + - Unsupported media types will raise a ValueError + - When use_center_crop=True, images are center-cropped to achieve padding divisibility + without distortion. When False, images are resized which may cause slight distortion. + """ + + def __init__( + self, + input_keys: List = ["media_list"], + max_diffusion_image_side_length: int = 1024, + max_mllm_image_side_length: int = 768, + diffusion_image_padding_constant: int = 16, + use_center_crop: bool = False, + args: Optional[dict] = None, + ) -> None: + super().__init__(input_keys, None, args) + self.max_diffusion_image_side_length = max_diffusion_image_side_length + self.max_mllm_image_side_length = max_mllm_image_side_length + self.diffusion_image_padding_constant = diffusion_image_padding_constant + self.use_center_crop = use_center_crop + + def __call__(self, data_dict: Dict) -> Dict: + assert len(self.input_keys) == 1, ( + "This transform only supports one input key. Try to organize all the media contents under one key." + ) + if self.input_keys[0] not in data_dict: + print(f"Input key {self.input_keys[0]} not found in data_dict: {data_dict['__key__']}") + return None + original_media_content = data_dict[self.input_keys[0]] + + diffusion_media_content = {} + mllm_media_content = {} + + for key, media in original_media_content.items(): + # Check if it's an image or video + if isinstance(media, Image.Image): + # Process single image + diffusion_media_content[key] = self._resize_image( + media, + self.max_diffusion_image_side_length, + self.diffusion_image_padding_constant, + self.use_center_crop, + ) + mllm_media_content[key] = self._resize_image( + media, + self.max_mllm_image_side_length, + None, # we don't need to resize the mllm media content to a specific padding constant since it will be handled by the processor + self.use_center_crop, + ) + elif isinstance(media, list) and all(isinstance(frame, Image.Image) for frame in media): + # Process video (list of images) + diffusion_media_content[key] = [ + self._resize_image( + frame, + self.max_diffusion_image_side_length, + self.diffusion_image_padding_constant, + self.use_center_crop, + ) + for frame in media + ] + mllm_media_content[key] = [ + self._resize_image( + frame, + self.max_mllm_image_side_length, + None, # we don't need to resize the mllm media content to a specific padding constant since it will be handled by the processor + self.use_center_crop, + ) + for frame in media + ] + else: + raise ValueError(f"Unsupported media type for key {key}: {type(media)}") + + # Add the resized media content to data_dict + data_dict["diffusion_media_list"] = diffusion_media_content + data_dict["mllm_media_list"] = mllm_media_content + + return data_dict + + def _resize_image( + self, image: Image.Image, max_side_length: int, padding_divisor=None, use_center_crop: bool = False + ) -> Image.Image: + """Resize image while preserving aspect ratio and ensuring dimensions are divisible by padding_divisor. + + Args: + image: Input PIL Image + max_side_length: Maximum allowed side length + padding_divisor: Both dimensions must be divisible by this value + use_center_crop: If True, use center crop to achieve divisibility; if False, use resize + + Returns: + Resized PIL Image + """ + # Get original dimensions + width, height = image.size + + # Calculate scale factor to ensure max side length constraint + scale_factor = min(max_side_length / width, max_side_length / height) + + # Only scale down, not up + if scale_factor < 1.0: + new_width = max(1, int(width * scale_factor)) + new_height = max(1, int(height * scale_factor)) + else: + new_width = width + new_height = height + + # Resize image to maintain aspect ratio + resized_image = image.resize((new_width, new_height), Image.Resampling.LANCZOS) + + # Calculate target dimensions that are divisible by padding_divisor + if padding_divisor is not None: + final_width = max(1, (new_width // padding_divisor)) * padding_divisor + final_height = max(1, (new_height // padding_divisor)) * padding_divisor + else: + final_width = new_width + final_height = new_height + + # If dimensions need adjustment + if final_width != new_width or final_height != new_height: + if use_center_crop: + # Use center crop to achieve target dimensions + left = (new_width - final_width) // 2 + top = (new_height - final_height) // 2 + right = left + final_width + bottom = top + final_height + resized_image = resized_image.crop((left, top, right, bottom)) + else: + # Use resize (may cause distortion) + resized_image = resized_image.resize((final_width, final_height), Image.Resampling.LANCZOS) + + return resized_image diff --git a/cosmos_framework/data/vfm/augmentors/interleaved_video_parsing.py b/cosmos_framework/data/vfm/augmentors/interleaved_video_parsing.py new file mode 100644 index 0000000..aea759e --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/interleaved_video_parsing.py @@ -0,0 +1,571 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import random +from collections.abc import Callable +from typing import Optional + +import numpy as np +import omegaconf +import torch +from einops import rearrange +from torchcodec.decoders import VideoDecoder +from torchvision.transforms.v2 import Resize, UniformTemporalSubsample + +from cosmos_framework.data.imaginaire.webdataset.augmentors.image.misc import obtain_augmentation_size +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.augmentors.video_parsing import VideoParsingWithFullFrames + +# Local copies of the torchcodec decoder helpers so this module does not depend on +# private symbols of ``video_parsing.py``. Behavior matches the originals. +_PostDecodeTransforms = list[Callable[[torch.Tensor], torch.Tensor]] | None +_SUPPORTS_VIDEO_DECODER_TRANSFORMS: bool | None = None +_WARNED_POST_DECODE_TRANSFORMS = False + + +def _create_video_decoder( + video: bytes, + seek_mode: str, + num_ffmpeg_threads: int, + transforms: _PostDecodeTransforms = None, +) -> tuple[VideoDecoder, _PostDecodeTransforms]: + global _SUPPORTS_VIDEO_DECODER_TRANSFORMS, _WARNED_POST_DECODE_TRANSFORMS + + kwargs = {"seek_mode": seek_mode, "num_ffmpeg_threads": num_ffmpeg_threads} + if transforms is None: + return VideoDecoder(video, **kwargs), None + + if _SUPPORTS_VIDEO_DECODER_TRANSFORMS is not False: + try: + decoder = VideoDecoder(video, transforms=transforms, **kwargs) + _SUPPORTS_VIDEO_DECODER_TRANSFORMS = True + return decoder, None + except TypeError as e: + if "transforms" not in str(e): + raise + _SUPPORTS_VIDEO_DECODER_TRANSFORMS = False + + if not _WARNED_POST_DECODE_TRANSFORMS: + log.warning( + "Installed torchcodec does not support VideoDecoder(transforms=...); " + "applying video transforms after frame decode.", + rank0_only=False, + ) + _WARNED_POST_DECODE_TRANSFORMS = True + return VideoDecoder(video, **kwargs), transforms + + +def _apply_post_decode_transforms( + frames: torch.Tensor, transforms: _PostDecodeTransforms +) -> torch.Tensor: # frames: [T,C,H,W], returns: [T,C,H,W] + if transforms is None: + return frames + + for transform in transforms: + frames = transform(frames) # [T,C,H,W] + return frames + + +class VideoTransferAlignedFullFramesParsing(VideoParsingWithFullFrames): + """Decode RGB and precomputed control videos with one shared v3 frame plan. + + This is the variable-length counterpart of the fixed-window transfer parser. + The RGB stream determines the sampled stride and frame indices. Any extra + input video streams, such as depth or segmentation, are decoded with the same + frame indices so the control video stays temporally aligned with the target. + """ + + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + assert len(input_keys) >= 2, "VideoTransferAlignedFullFramesParsing requires [metas, video, ...]." + super().__init__(input_keys=input_keys[:2], output_keys=output_keys, args=args) + self.input_keys = input_keys + self.control_video_keys = input_keys[2:] + self.min_stride_key = self.args.get("min_stride_key", "_full_frames_min_stride") + + def _build_rgb_decode_transform(self, data_dict: dict, meta_dict: dict) -> list[Resize] | None: + if not self.perform_resize: + return None + + img_size = obtain_augmentation_size(data_dict, {"size": self.size}) + assert isinstance(img_size, (tuple, omegaconf.listconfig.ListConfig)), ( + f"Arg size in resize should be a tuple, get {type(img_size)}, {img_size}" + ) + img_w, img_h = img_size + orig_w, orig_h = meta_dict["width"], meta_dict["height"] + + scaling_ratio = min((img_w / orig_w), (img_h / orig_h)) + target_size = (int(scaling_ratio * orig_h + 0.5), int(scaling_ratio * orig_w + 0.5)) + assert target_size[0] <= img_h and target_size[1] <= img_w, ( + f"Resize error. orig {(orig_w, orig_h)} desire {img_size} compute {target_size}" + ) + return [Resize(target_size)] + + def _sample_frame_indices(self, decoder_len: int, min_stride_override: int | None = None) -> tuple[list[int], int]: + min_stride = int(min_stride_override) if min_stride_override is not None else self.min_stride + max_stride = max(self.max_stride, min_stride) + stride = self._sample_stride_with_bias(max_stride, min_stride) + frame_indices = np.arange(0, decoder_len, stride).tolist() + max_num_frames = min(len(frame_indices), self.args.get("max_num_frames", 1000)) + if max_num_frames < 1: + return [], stride + + # Wan VAE temporal compression expects 1 + 4N video frames. + num_video_frames = 1 + 4 * ((max_num_frames - 1) // 4) + return frame_indices[:num_video_frames], stride + + def _probe_video_len(self, video: bytes) -> int: + video_decoder = VideoDecoder( + video, + seek_mode=self.seek_mode, + num_ffmpeg_threads=self.video_decode_num_threads, + ) + try: + return len(video_decoder) + finally: + del video_decoder + + def _decode_frames_at( + self, + video: bytes, + frame_indices: list[int], + transforms: list[Resize] | None = None, + ) -> torch.Tensor: # returns [C,T,H,W] + video_decoder, post_decode_transforms = _create_video_decoder( + video, + self.seek_mode, + self.video_decode_num_threads, + transforms, + ) + try: + frame_batch = video_decoder.get_frames_at(frame_indices) + frames = frame_batch.data # [T,C,H,W] + frames = _apply_post_decode_transforms(frames, post_decode_transforms) # [T,C,H,W] + frames = frames.permute(1, 0, 2, 3) # [C,T,H,W] + finally: + del video_decoder + return frames # [C,T,H,W] + + def __call__(self, data_dict: dict) -> dict | None: + try: + meta_dict = data_dict[self.meta_key] + video = data_dict[self.video_key] + except Exception: + log.warning( + f"Cannot find video. url: {data_dict['__url__']}, key: {data_dict['__key__']}", + rank0_only=False, + ) + return None + + if not self._validate_and_probe(video, meta_dict, data_dict): + return None + + rgb_transform = self._build_rgb_decode_transform(data_dict, meta_dict) + control_videos: dict[str, bytes] = {} + try: + decoder_len = self._probe_video_len(video) + + control_decoder_lens = [] + for control_video_key in self.control_video_keys: + control_video = data_dict.get(control_video_key) + if not isinstance(control_video, bytes): + log.warning( + f"VideoTransferAlignedFullFramesParsing: missing bytes for {control_video_key}. " + f"url: {data_dict['__url__']}, key: {data_dict['__key__']}", + rank0_only=False, + ) + return None + control_videos[control_video_key] = control_video + control_decoder_lens.append(self._probe_video_len(control_video)) + + # Precomputed control streams can be one frame shorter than RGB; sample + # only frames present in every stream to keep all modalities aligned. + aligned_decoder_len = min([decoder_len, *control_decoder_lens]) if control_decoder_lens else decoder_len + + min_stride_override = data_dict.pop(self.min_stride_key, None) + frame_indices, stride = self._sample_frame_indices( + aligned_decoder_len, min_stride_override=min_stride_override + ) + if len(frame_indices) == 0: + log.warning( + f"VideoTransferAlignedFullFramesParsing: no valid frame indices. " + f"url: {data_dict['__url__']}, key: {data_dict['__key__']}", + rank0_only=False, + ) + return None + + video_frames = self._decode_frames_at(video, frame_indices, rgb_transform) # [C,T,H,W] + except Exception as e: + log.warning( + f"Failed to decode RGB video. url: {data_dict['__url__']}, key: {data_dict['__key__']}, error: {e}", + rank0_only=False, + ) + return None + + base_video_info = { + "frame_start": frame_indices[0], + "frame_end": frame_indices[-1], + "frame_indices": frame_indices, + "num_frames": len(frame_indices), + "fps": meta_dict["framerate"], + "conditioning_fps": meta_dict["framerate"] / stride, + "num_multiplier": stride, + "n_orig_video_frames": decoder_len, + } + data_dict[self.video_key] = { + **base_video_info, + "video": video_frames, # [C,T,H,W] + } + + for control_video_key, control_video in control_videos.items(): + try: + control_frames = self._decode_frames_at(control_video, frame_indices) # [C,T,H,W] + except Exception as e: + log.warning( + f"Failed to decode {control_video_key}. " + f"url: {data_dict['__url__']}, key: {data_dict['__key__']}, error: {e}", + rank0_only=False, + ) + return None + data_dict[control_video_key] = { + **base_video_info, + "video": control_frames, # [C,T,H,W] + } + + return data_dict + + +class VideoTransferAlignedLegacyChunkParsing(VideoTransferAlignedFullFramesParsing): + """Decode legacy caption-window transfer streams with shared RGB/control frame indices.""" + + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys=input_keys, output_keys=output_keys, args=args) + self.key_for_caption = self.args["key_for_caption"] + assert self.key_for_caption in [ + "t2w_windows", + "i2w_windows_later_frames", + ], "key_for_caption must be either t2w_windows or i2w_windows_later_frames" + self.min_duration = self.args["min_duration"] + self.num_frames = self.args["num_video_frames"] + self.use_native_fps = self.args["use_native_fps"] + self.use_original_fps = self.args["use_original_fps"] + self.use_dynamic_fps = self.args.get("use_dynamic_fps", False) + self.low_fps_bias = self.args.get("low_fps_bias", 0.5) + assert 0.0 <= self.low_fps_bias <= 1.0, f"low_fps_bias must be in [0, 1], got {self.low_fps_bias}" + mode_count = sum([self.use_dynamic_fps, self.use_native_fps, self.use_original_fps]) + assert mode_count <= 1, ( + f"Only one FPS mode can be enabled at a time. Got: " + f"use_dynamic_fps={self.use_dynamic_fps}, " + f"use_native_fps={self.use_native_fps}, " + f"use_original_fps={self.use_original_fps}" + ) + self.allowed_num_multiplers = self.args.get("allowed_num_multiplers", list(range(1, 100))) + if self.num_frames > 0: + self.sampler = UniformTemporalSubsample(self.num_frames) + + def _sample_legacy_stride_with_bias(self, max_stride: int) -> int: + if max_stride == 1: + return 1 + + strides = np.arange(1, max_stride + 1) + weights = np.linspace(1 - self.low_fps_bias, self.low_fps_bias, max_stride) + weights = np.maximum(weights, 0.01) + probs = weights / weights.sum() + return int(np.random.choice(strides, p=probs)) + + def _decode_all_streams_at( + self, + video: bytes, + control_videos: dict[str, bytes], + frame_indices: list[int], + rgb_transform: list[Resize] | None, + ) -> tuple[torch.Tensor, dict[str, torch.Tensor]]: + video_frames = self._decode_frames_at(video, frame_indices, rgb_transform) # [C,T,H,W] + control_frames_by_key = { + control_video_key: self._decode_frames_at(control_video, frame_indices) # [C,T,H,W] + for control_video_key, control_video in control_videos.items() + } + return video_frames, control_frames_by_key + + def _subsample_all_streams( + self, video_frames: torch.Tensor, control_frames_by_key: dict[str, torch.Tensor] + ) -> tuple[torch.Tensor, dict[str, torch.Tensor]]: + video_frames = rearrange( + self.sampler(rearrange(video_frames, "c t h w -> t c h w")), "t c h w -> c t h w" + ) # [C,T,H,W] + control_frames_by_key = { + key: rearrange(self.sampler(rearrange(frames, "c t h w -> t c h w")), "t c h w -> c t h w") + for key, frames in control_frames_by_key.items() + } # [C,T,H,W] + return video_frames, control_frames_by_key + + def __call__(self, data_dict: dict) -> dict | None: + try: + meta_dict = data_dict[self.meta_key] + video = data_dict[self.video_key] + except Exception: + log.warning( + f"Cannot find video. url: {data_dict['__url__']}, key: {data_dict['__key__']}", + rank0_only=False, + ) + return None + + if not self._validate_and_probe(video, meta_dict, data_dict): + return None + + control_videos: dict[str, bytes] = {} + control_decoder_lens: list[int] = [] + try: + decoder_len = self._probe_video_len(video) + for control_video_key in self.control_video_keys: + control_video = data_dict.get(control_video_key) + if not isinstance(control_video, bytes): + log.warning( + f"VideoTransferAlignedLegacyChunkParsing: missing bytes for {control_video_key}. " + f"url: {data_dict['__url__']}, key: {data_dict['__key__']}", + rank0_only=False, + ) + return None + control_videos[control_video_key] = control_video + control_decoder_lens.append(self._probe_video_len(control_video)) + except Exception as e: + log.warning( + f"Failed to probe video streams. url: {data_dict['__url__']}, key: {data_dict['__key__']}, error: {e}", + rank0_only=False, + ) + return None + + aligned_decoder_len = min([decoder_len, *control_decoder_lens]) if control_decoder_lens else decoder_len + options: list = list((i, item) for i, item in enumerate(meta_dict[self.key_for_caption])) + if len(options) > 1: + options = options[:-1] + random.shuffle(options) + + rgb_transform = self._build_rgb_decode_transform(data_dict, meta_dict) + video_frames = None + control_frames_by_key: dict[str, torch.Tensor] = {} + dynamic_conditioning_fps = None + num_multiplier: float | int = 1 + frame_indices: list[int] = [] + chunk_index = 0 + start_frame = 0 + end_frame = 0 + + for chunk_index, option in options: + start_frame = int(option["start_frame"]) + end_frame = min(int(option["end_frame"]), aligned_decoder_len) + if (end_frame - start_frame) < self.min_duration * meta_dict["framerate"]: + continue + if self.use_native_fps or self.use_original_fps or self.use_dynamic_fps: + if "alpamayo" in data_dict["__url__"].root: + start_frame += 5 + if (end_frame - start_frame) < self.num_frames: + continue + total_frames = end_frame - start_frame + if self.use_dynamic_fps: + max_stride = total_frames // self.num_frames + if max_stride < 1: + continue + num_multiplier = self._sample_legacy_stride_with_bias(max_stride) + dynamic_conditioning_fps = meta_dict["framerate"] / num_multiplier + elif self.use_native_fps: + num_multiplier = total_frames // self.num_frames + if num_multiplier not in self.allowed_num_multiplers: + continue + else: + num_multiplier = 1 + + expected_length = self.num_frames * int(num_multiplier) + if total_frames < expected_length: + continue + frame_start = start_frame + (total_frames - expected_length) // 2 + frame_end = frame_start + expected_length + frame_indices = list(range(frame_start, frame_end, int(num_multiplier))) + else: + frame_indices = list(range(start_frame, end_frame)) + if "alpamayo" in data_dict["__url__"].root: + if len(frame_indices) < 5: + continue + frame_indices = frame_indices[5:] + start_frame += 5 + + try: + video_frames, control_frames_by_key = self._decode_all_streams_at( + video, control_videos, frame_indices, rgb_transform + ) # [C,T,H,W] + except Exception as e: + log.warning( + f"Failed to decode aligned video streams. " + f"url: {data_dict['__url__']}, key: {data_dict['__key__']}, error: {e}", + rank0_only=False, + ) + return None + break + + if video_frames is None: + log.warning( + f"No valid video frames found, return None. url: {data_dict['__url__']}, key: {data_dict['__key__']}", + rank0_only=False, + ) + return None + + if self.num_frames > 0 and not (self.use_dynamic_fps or self.use_native_fps or self.use_original_fps): + video_frames, control_frames_by_key = self._subsample_all_streams( + video_frames, control_frames_by_key + ) # [C,T,H,W] + num_multiplier = (end_frame - start_frame) / self.num_frames + + + # variable-length fields like ``frame_indices`` here -- ``video_flatten_keys`` in + # ``get_video_transfer_augmentor`` lists ``frame_indices``, and surfacing a + # per-sample list there would crash ``custom_collate_fn`` (default_collate requires + # equal-size elements across the batch). + base_video_info = { + "fps": meta_dict["framerate"], + "n_orig_video_frames": meta_dict["nb_frames"], + "chunk_index": chunk_index, + "frame_start": start_frame, + "frame_end": end_frame, + "num_frames": end_frame - start_frame, + "num_multiplier": num_multiplier, + "conditioning_fps": dynamic_conditioning_fps or meta_dict["framerate"] / num_multiplier, + } + data_dict[self.video_key] = { + **base_video_info, + "video": video_frames, # [C,T,H,W] + } + for control_video_key, control_frames in control_frames_by_key.items(): + data_dict[control_video_key] = { + **base_video_info, + "video": control_frames, # [C,T,H,W] + } + return data_dict + + +class VideoTransferAlignedChunkedFramesParsing(VideoTransferAlignedFullFramesParsing): + """Decode RGB and aligned control videos for a selected caption chunk.""" + + def _sample_frame_indices_for_chunk( + self, + decoder_len: int, + chunk_start: int, + chunk_end: int, + min_stride_override: int | None = None, + ) -> tuple[list[int], int]: + chunk_start = max(0, min(chunk_start, decoder_len)) + chunk_end = max(chunk_start, min(chunk_end, decoder_len)) + if chunk_end <= chunk_start: + return [], 0 + + min_stride = int(min_stride_override) if min_stride_override is not None else self.min_stride + max_stride = max(self.max_stride, min_stride) + stride = self._sample_stride_with_bias(max_stride, min_stride) + frame_indices = np.arange(chunk_start, chunk_end, stride).tolist() + max_num_frames = min(len(frame_indices), self.args.get("max_num_frames", 1000)) + if max_num_frames < 1: + return [], stride + + # Wan VAE temporal compression expects 1 + 4N video frames. + num_video_frames = 1 + 4 * ((max_num_frames - 1) // 4) + return frame_indices[:num_video_frames], stride + + def __call__(self, data_dict: dict) -> dict | None: + try: + meta_dict = data_dict[self.meta_key] + video = data_dict[self.video_key] + except Exception: + log.warning( + f"Cannot find video. url: {data_dict['__url__']}, key: {data_dict['__key__']}", + rank0_only=False, + ) + return None + + if not self._validate_and_probe(video, meta_dict, data_dict): + return None + + if "chunk_start_frame" not in data_dict or "chunk_end_frame" not in data_dict: + log.warning( + f"VideoTransferAlignedChunkedFramesParsing: missing chunk_start_frame/chunk_end_frame. " + f"url: {data_dict['__url__']}, key: {data_dict['__key__']}", + rank0_only=False, + ) + return None + chunk_start = int(data_dict["chunk_start_frame"]) + chunk_end = int(data_dict["chunk_end_frame"]) + + rgb_transform = self._build_rgb_decode_transform(data_dict, meta_dict) + control_videos: dict[str, bytes] = {} + try: + decoder_len = self._probe_video_len(video) + + control_decoder_lens = [] + for control_video_key in self.control_video_keys: + control_video = data_dict.get(control_video_key) + if not isinstance(control_video, bytes): + log.warning( + f"VideoTransferAlignedChunkedFramesParsing: missing bytes for {control_video_key}. " + f"url: {data_dict['__url__']}, key: {data_dict['__key__']}", + rank0_only=False, + ) + return None + control_videos[control_video_key] = control_video + control_decoder_lens.append(self._probe_video_len(control_video)) + + # Clamp the caption chunk to frames available in every loaded stream. + aligned_decoder_len = min([decoder_len, *control_decoder_lens]) if control_decoder_lens else decoder_len + min_stride_override = data_dict.pop(self.min_stride_key, None) + frame_indices, stride = self._sample_frame_indices_for_chunk( + aligned_decoder_len, + chunk_start, + chunk_end, + min_stride_override=min_stride_override, + ) + if len(frame_indices) == 0: + log.warning( + f"VideoTransferAlignedChunkedFramesParsing: empty chunk after clamping/stride. " + f"chunk=[{chunk_start},{chunk_end}), aligned_decoder_len={aligned_decoder_len}, " + f"url: {data_dict['__url__']}, key: {data_dict['__key__']}", + rank0_only=False, + ) + return None + + video_frames = self._decode_frames_at(video, frame_indices, rgb_transform) # [C,T,H,W] + except Exception as e: + log.warning( + f"Failed to decode RGB video. url: {data_dict['__url__']}, key: {data_dict['__key__']}, error: {e}", + rank0_only=False, + ) + return None + + base_video_info = { + "frame_start": frame_indices[0], + "frame_end": frame_indices[-1], + "frame_indices": frame_indices, + "num_frames": len(frame_indices), + "fps": meta_dict["framerate"], + "conditioning_fps": meta_dict["framerate"] / stride if stride > 0 else meta_dict["framerate"], + "num_multiplier": stride, + "n_orig_video_frames": decoder_len, + } + data_dict[self.video_key] = { + **base_video_info, + "video": video_frames, # [C,T,H,W] + } + + for control_video_key, control_video in control_videos.items(): + try: + control_frames = self._decode_frames_at(control_video, frame_indices) # [C,T,H,W] + except Exception as e: + log.warning( + f"Failed to decode {control_video_key}. " + f"url: {data_dict['__url__']}, key: {data_dict['__key__']}, error: {e}", + rank0_only=False, + ) + return None + data_dict[control_video_key] = { + **base_video_info, + "video": control_frames, # [C,T,H,W] + } + + data_dict.pop("chunk_start_frame", None) + data_dict.pop("chunk_end_frame", None) + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/merge_datadict.py b/cosmos_framework/data/vfm/augmentors/merge_datadict.py new file mode 100644 index 0000000..813d6d2 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/merge_datadict.py @@ -0,0 +1,67 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Optional + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log + + +class KeyRenamer(Augmentor): + """Renames keys in data_dict. Runs as the first augmentor to normalize key names. + + Args: + input_keys: Not used (required by Augmentor interface). + output_keys: Not used. + args: Dictionary with: + - rename_map: dict[str, str] mapping old_key -> new_key. + """ + + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + self.rename_map: dict[str, str] = args.get("rename_map", {}) if args else {} + + def __call__(self, data_dict: dict) -> dict: + if not self.rename_map: + return data_dict + + for old_key, new_key in self.rename_map.items(): + if old_key in data_dict: + data_dict[new_key] = data_dict.pop(old_key) + return data_dict + + +class DataDictMerger(Augmentor): + def __init__(self, input_keys: list, output_keys: list, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + + def __call__(self, data_dict: dict) -> dict | None: + r"""Merge the dictionary associated with the input keys into data_dict. Only keys in output_keys are merged. + + Supports transfer-style keys (e.g. depth_pervideo_video_depth_anything): when "depth" in key + assigns key_dict["video"] to data_dict["depth"]; when "segmentation" in key assigns + key_dict["video"] or key_dict to data_dict["segmentation"]. + + Args: + data_dict (dict): Input data dict + Returns: + data_dict (dict): Output dict with dictionary associated with the input keys merged. + """ + for key in self.input_keys: + if key not in data_dict: + log.warning( + f"DataDictMerger dataloader error: missing {key}, {data_dict['__url__']}, {data_dict['__key__']}", + rank0_only=False, + ) + return None + key_dict = data_dict.pop(key) + if "depth" in key and "depth" in self.output_keys: + data_dict["depth"] = key_dict["video"] + elif "segmentation" in key and "segmentation" in self.output_keys: + data_dict["segmentation"] = key_dict["video"] if "video" in key_dict else key_dict + if isinstance(key_dict, dict): + for sub_key in key_dict: + if sub_key in self.output_keys and sub_key not in data_dict: + data_dict[sub_key] = key_dict[sub_key] + del key_dict + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/pkl_to_media.py b/cosmos_framework/data/vfm/augmentors/pkl_to_media.py new file mode 100644 index 0000000..54d47b8 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/pkl_to_media.py @@ -0,0 +1,317 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Augmentors for handling video loading from pickled bytes.""" + +import io +import pickle as pkl +import random +import re +from typing import Dict, Optional + +import torch +from PIL import Image, UnidentifiedImageError +from qwen_vl_utils.vision_process import smart_nframes, smart_resize +from torchvision import transforms +from torchvision.transforms import InterpolationMode + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log +from cosmos_framework.utils.vfm.video_preprocess import tensor_to_pil_images + +Image.MAX_IMAGE_PIXELS = 933120000 +_VIDEO_EXTENSIONS = "mp4 avi webm mov".split() + +VIDEO_DECODER_OPTIONS = {} + + +def token_to_pixels(token_length: int, patch_size: int = 14, temporal_patch_size: int = 2) -> int: + """Convert token length to pixels based on patch size and temporal patch size.""" + + merged_patch_size = patch_size * 2 + return token_length * merged_patch_size**2 * temporal_patch_size + + +def pixels_to_token(pixels: int, patch_size: int = 14, temporal_patch_size: int = 2) -> int: + """Convert pixels to token length based on patch size and temporal patch size.""" + + merged_patch_size = patch_size * 2 + return pixels // merged_patch_size**2 // temporal_patch_size + + +def _video_decoder_qwen_func( + key: str, + data: bytes, + min_fps_thres: int = 4, + max_fps_thres: int = 60, + target_fps: float = 2.0, + min_video_token_length: int = 16, + max_video_token_length: int = 8192, + num_threads: int = 0, + random_augmentation: bool = False, + fps_random_range: list[float] = [0.5, 1.5], + max_video_token_length_random_range: list[float] = [0.75, 1.25], + frame_count_random_range: Optional[list[int]] = None, + start_frame: Optional[int] = None, + end_frame: Optional[int] = None, + **kwargs, +) -> dict | None: + """Actual video decoder function. + + Args: + key (str): Video file name/key + data (bytes): Video binary data + min_fps_thres (int, optional): Minimum FPS threshold. Defaults to 4. + max_fps_thres (int, optional): Maximum FPS threshold. Defaults to 60. + target_fps (float, optional): Target FPS. Defaults to 2.0. + min_video_token_length (int, optional): Minimum token length. Defaults to 16. + max_video_token_length (int, optional): Maximum token length. Defaults to 8192. + num_threads (int, optional): Number of threads for decord. Defaults to 0. + random_augmentation (bool, optional): Whether to randomize the FPS and max_video_token_length. Defaults to False. + fps_random_range (list[float], optional): Random FPS range. Defaults to [10.0, 24.0]. + max_video_token_length_random_range (list[float], optional): Random max_video_token_length range. Defaults to [0.75, 1.25]. + frame_count_random_range (list[int], optional): Random frame count range. If provided, take priority over fps_random_range. + start_frame (Optional[int], optional): Start frame. Defaults to None. If both start_frame and end_frame are provided, the video will be decoded from start_frame to end_frame. + end_frame (Optional[int], optional): End frame. Defaults to None. If both start_frame and end_frame are provided, the video will be decoded from start_frame to end_frame. + + Raises: + ValueError: Video fps lower than 1, skipping + ValueError: Video fps lower than min_fps_thres, skipping + ValueError: Video fps higher than max_fps_thres, skipping + + Returns: + dict | None: Dictionary with video frames tensor and target FPS + """ + import decord + + # Check video extension + extension = re.sub(r".*[.]", "", key) + if extension.lower() not in _VIDEO_EXTENSIONS: + return None + + # Read video + video_buffer = io.BytesIO(data) + video_reader = decord.VideoReader(video_buffer, num_threads=num_threads) + total_frames, video_fps = len(video_reader), video_reader.get_avg_fps() + + if start_frame is not None and end_frame is not None: + total_frames = end_frame - start_frame + + if video_fps < 1: + raise ValueError("Video fps lower than 1, skipping") + if video_fps < min_fps_thres: + raise ValueError(f"Video fps {video_fps} lower than {min_fps_thres}, skipping") + if video_fps > max_fps_thres: + raise ValueError(f"Video fps {video_fps} higher than {max_fps_thres}, skipping") + + if random_augmentation: + if frame_count_random_range is not None: + # Random number of frames + min_frames_range, max_frames_range = frame_count_random_range + min_frames_range = min(min_frames_range, total_frames) + max_frames_range = min(max_frames_range, total_frames) + target_frames = random.uniform(min_frames_range, max_frames_range) + target_fps = target_frames / total_frames * video_fps + else: + # randomize fps + target_fps = ( + random.uniform(fps_random_range[0], fps_random_range[1]) * target_fps + if random.random() < 0.5 + else target_fps + ) + # randomize max_video_token_length + max_video_token_length = int( + random.uniform(max_video_token_length_random_range[0], max_video_token_length_random_range[1]) + * max_video_token_length + ) + log.debug(f"random_augmentation: max_video_token_length: {max_video_token_length}, target_fps: {target_fps}") + + patch_size = 14 + min_height_width = 56 # https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen2_vl/image_processing_qwen2_vl.py#L57 + temporal_patch_size = 2 + min_pixels: int = token_to_pixels(min_video_token_length, patch_size, temporal_patch_size) + max_pixels: int = token_to_pixels(max_video_token_length, patch_size, temporal_patch_size) + max_frames: int = max_pixels // (min_height_width) ** 2 // temporal_patch_size + + # sample based on target fps + nframes = smart_nframes(dict(fps=target_fps), total_frames=total_frames, video_fps=video_fps) + nframes = min(nframes, max_frames) + if start_frame is not None and end_frame is not None: + idx = torch.linspace(start_frame, end_frame - 1, nframes).round().long().tolist() # [nframes] + else: + idx = torch.linspace(0, total_frames - 1, nframes).round().long().tolist() # [nframes] + video_frames = video_reader.get_batch(idx).asnumpy() + video_frames = torch.tensor(video_frames).permute(0, 3, 1, 2) # [T,C,H,W] + sample_fps = nframes / max(total_frames, 1e-6) * video_fps + + # recompute max_pixels based on number of sampled frames + nframes, _, height, width = video_frames.shape + max_pixels = max_pixels // nframes + resized_height, resized_width = smart_resize( + height, + width, + min_pixels=min_pixels, + max_pixels=max_pixels, + ) + video_frames = transforms.functional.resize( + video_frames, + [resized_height, resized_width], + interpolation=InterpolationMode.BICUBIC, + antialias=True, + ).float() + video_frames = video_frames.permute(1, 0, 2, 3) # [C,T,H,W] + + # Clean up + video_reader.seek(0) # set video reader point back to 0 to clean up cache + del video_reader # delete the reader to avoid memory leak + + return dict(videos=video_frames, fps=sample_fps) + + +class PKLToMedia(Augmentor): + """ + Converts PKL bytes stored in a data dictionary into media. + + Handles input formats for the specified input key: + A dictionary mapping media names (str) to bytes objects. + + The output format is a dictionary mapping names to their respective decoded objects: + Input dict[str, bytes] -> Output dict[str, torch.Tensor | PIL.Image] + + Corrupted or non-decodable bytes are skipped with a warning. + """ + + def __init__( + self, + input_key: str = "media", + output_key: str = "media", + min_fps_thres: int = 4, + max_fps_thres: int = 60, + target_fps: float = 4.0, + min_video_token_length: int = 16, + max_video_token_length: int = 8192, + num_threads: int = 0, + random_augmentation: bool = False, + is_input_in_dict: bool = False, + use_start_frame_end_frame: bool = False, + frame_count_random_range: Optional[list[int]] = None, + ) -> None: + """ + Args: + input_key (str): Key in the data_dict containing video/image data. + output_key (str): Key to store the resulting video frame tensors or PIL images. + min_fps_thres (int): Minimum FPS threshold for video decoding. + max_fps_thres (int): Maximum FPS threshold for video decoding. + target_fps (float): Target FPS for video decoding. + min_video_token_length (int): Minimum token length for video decoding. + max_video_token_length (int): Maximum token length for video decoding. + num_threads (int): Number of threads for video decoding. + random_augmentation (bool): Whether to apply random augmentation during decoding. + is_input_in_dict (bool): Whether the input key is in the data_dict instead of pkl files. (For cosmos predict2 videos) + use_start_frame_end_frame (bool): Whether to use start_frame and end_frame to decode the video. (For cosmos predict2 videos) + frame_count_random_range (list[int], optional): Random frame count range. Defaults to None. + """ + self.input_key = input_key + self.output_key = output_key + self.video_decoder_params = { + "min_fps_thres": min_fps_thres, + "max_fps_thres": max_fps_thres, + "target_fps": target_fps, + "min_video_token_length": min_video_token_length, + "max_video_token_length": max_video_token_length, + "num_threads": num_threads, + "random_augmentation": random_augmentation, + "frame_count_random_range": frame_count_random_range, + } + self.is_input_in_dict = is_input_in_dict + self.use_start_frame_end_frame = use_start_frame_end_frame + + def _bytes_to_video_frames(self, video_bytes: bytes, identifier: str = "video") -> Optional[Dict]: + """Converts video bytes to video frame tensors using the video decoder.""" + try: + result = _video_decoder_qwen_func( + key=f"{identifier}.mp4", # Add .mp4 extension for the decoder + data=video_bytes, + **self.video_decoder_params, + ) + result["videos"] = tensor_to_pil_images(result["videos"]) # 3,T,H,W -> list of PIL images + if result is not None: + return result + else: + log.warning(f"Skipping item '{identifier}': Video decoder returned None.") + return None + except Exception as e: + log.warning(f"Skipping item '{identifier}': Error decoding video bytes: {e}") + return None + + def _bytes_to_pil(self, image_bytes: bytes, identifier: str = "image") -> Optional[Image.Image]: + """Converts a single bytes object to a PIL Image.""" + try: + with io.BytesIO(image_bytes) as stream: + img = Image.open(stream) + img.load() # Verify the image data + return img.convert("RGB") # Convert to standard RGB format + except UnidentifiedImageError: + log.warning(f"Skipping item '{identifier}': Cannot identify image file from bytes.") + except Exception as e: + log.warning(f"Skipping item '{identifier}': Error decoding image bytes: {e}") + return None + + def __call__(self, data_dict: Dict) -> Dict: + """ + Processes the data_dict to convert video/image bytes to their respective formats. + + Args: + data_dict (Dict): The input data dictionary. + + Returns: + Dict: The modified data dictionary with video frame tensors and/or PIL images. + """ + input_key = self.input_key + output_key = self.output_key + + if input_key not in data_dict: + log.debug( + f"Input key '{input_key}' not found in data_dict. Skipping PKLToMedia. Available keys: {data_dict.keys()}" + ) + return data_dict + + if not self.is_input_in_dict: + data = pkl.loads(data_dict[input_key]) + else: + data = data_dict[input_key] + + output_data = {} + + if isinstance(data, dict): + for name, item in data.items(): + if isinstance(item, bytes): + # Determine if this is video or image based on the key name + if "video" in name.lower(): + # Decode as video + result = self._bytes_to_video_frames(item, identifier=f"{input_key}['{name}']") + if result: + output_data[name] = result + elif "image" in name.lower(): + # Decode as image + result = self._bytes_to_pil(item, identifier=f"{input_key}['{name}']") + if result: + output_data[name] = result + else: + log.warning( + f"Skipping item with key '{name}' in '{input_key}': Key does not contain 'video' or 'image'." + ) + else: + log.warning(f"Skipping item with key '{name}' in '{input_key}': Expected bytes, got {type(item)}.") + else: + raise ValueError( + f"Input key '{input_key}' has unsupported type {type(data)}. " + f"Expected dict[str, bytes] for video/image data." + ) + + # Add the processed data and optionally remove the input key + data_dict[output_key] = output_data + if input_key != output_key: + del data_dict[input_key] + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/resolution_text_info.py b/cosmos_framework/data/vfm/augmentors/resolution_text_info.py new file mode 100644 index 0000000..8e6f3d4 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/resolution_text_info.py @@ -0,0 +1,122 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Optional + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor + +# Default templates for resolution info +DEFAULT_IMAGE_TEMPLATE = "This image is of {height}x{width} resolution." +DEFAULT_VIDEO_TEMPLATE = "This video is of {height}x{width} resolution." + + +class ResolutionTextInfo(Augmentor): + """ + Augmentor that appends resolution (height x width) info to captions. + + This augmentor should run AFTER CropToMultiple (which sets final_height/final_width) + and AFTER text transforms (so ai_caption exists), but BEFORE tokenization. + + Reads resolution from metadata keys (final_height, final_width) set by CropToMultiple. + Does NOT fall back to tensor shape to avoid incorrect latent dimensions. + + Automatically detects whether the input is an image or video based on which + key is present in the data_dict, and uses the appropriate template. + + Example: + Original caption: "A cat playing with a ball" + Augmented (image): "A cat playing with a ball. This image is 512x512." + Augmented (video): "A cat playing with a ball. This video is 480x854." + + Args: + input_keys (list): Input keys (not used, kept for API compatibility) + output_keys (list): Output keys (not used, kept for API compatibility) + args (dict): Configuration arguments: + - caption_key (str): Key for caption in data_dict. Default: "ai_caption" + - video_key (str): Key for video tensor in data_dict. Default: "video" + - image_size_key (str): Key for image size tensor in data_dict. Default: "image_size" + - image_template (str): Format string for image metadata. Default: DEFAULT_IMAGE_TEMPLATE + - video_template (str): Format string for video metadata. Default: DEFAULT_VIDEO_TEMPLATE + - separator (str): Separator between caption and metadata. Default: ". " + - enabled (bool): Whether augmentation is enabled. Default: True + """ + + def __init__( + self, input_keys: Optional[list] = None, output_keys: Optional[list] = None, args: Optional[dict] = None + ) -> None: + super().__init__(input_keys, output_keys, args) + + # Configuration with sensible defaults + self.caption_key = args.get("caption_key", "ai_caption") if args else "ai_caption" + self.image_key = args.get("image_key", "images") if args else "images" + self.video_key = args.get("video_key", "video") if args else "video" + self.image_size_key = args.get("image_size_key", "image_size") if args else "image_size" + self.image_template = args.get("image_template", DEFAULT_IMAGE_TEMPLATE) if args else DEFAULT_IMAGE_TEMPLATE + self.video_template = args.get("video_template", DEFAULT_VIDEO_TEMPLATE) if args else DEFAULT_VIDEO_TEMPLATE + self.default_separator = args.get("separator", ". ") if args else ". " + self.enabled = args.get("enabled", True) if args else True + + def __call__(self, data_dict: dict) -> dict | None: + """ + Append resolution (height x width) as text timestamps to the caption. + + Args: + data_dict (dict): Input data dict containing caption and image/video tensor + + Returns: + data_dict (dict): Output dict with augmented caption. + """ + if not self.enabled: + return data_dict + + # Get caption - must exist at this point (set by text transforms) + assert self.caption_key in data_dict, f"caption_key '{self.caption_key}' not found in data_dict." + caption = data_dict[self.caption_key] + + if (not isinstance(caption, str) and not isinstance(caption, dict)) or caption == "": + # This is for unconditional case. + return data_dict + + # Detect image vs video to select template + is_video = self.video_key in data_dict + is_image = self.image_key in data_dict + + if isinstance(caption, str): + # Case 1: Caption is a string. In this case, we create a string template for + # resolution, aspect ratio info and add it + if not is_video and not is_image: + raise ValueError("Neither video_key nor image_key found in data_dict.") + + template = self.video_template if is_video else self.image_template + + # Get dimensions from metadata keys (set by CropToMultiple) + image_size = data_dict.get(self.image_size_key) + height = int(image_size[0]) + width = int(image_size[1]) + + # Format metadata text + metadata_text = template.format(height=height, width=width) + + # Choose separator based on whether caption ends with a period + separator = " " if caption.rstrip().endswith(".") else self.default_separator + + # Update caption + data_dict[self.caption_key] = caption + separator + metadata_text + + elif isinstance(caption, dict): + # Case 2: Caption is a dictionary. This is for the json caption case. + # In this case, we add resolution and aspect ratio in json fields + aspect_ratio = data_dict["__url__"].meta.opts["aspect_ratio"] + height = int(data_dict["image_size"][0]) + width = int(data_dict["image_size"][1]) + data_dict[self.caption_key].update( + { + "resolution": {"H": height, "W": width}, + "aspect_ratio": aspect_ratio, + } + ) + + else: + raise ValueError(f"Unsupported caption type: {type(caption)}") + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/sequence_plan.py b/cosmos_framework/data/vfm/augmentors/sequence_plan.py new file mode 100644 index 0000000..9f0500a --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/sequence_plan.py @@ -0,0 +1,130 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Augmentor for creating sequence plans with random conditional frames. + +Supports two sampling strategies: +- weighted dict (``conditioning_config``): explicit frame-count → probability pairs +- uniform (``uniform_conditioning=True``): k ~ Uniform{0, T_latent-1}, where T_latent + is computed from the actual video length using the VAE temporal compression factor +""" + +import random +from typing import Optional + +import torch + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.data.vfm.sequence_packing import SequencePlan + + +class SequencePlanAugmentor(Augmentor): + """Augmentor that creates SequencePlan with random conditional frames. + + Samples k conditioning frames and writes ``condition_frame_indexes_vision = list(range(k))`` + into the SequencePlan. Downstream packing code reads this field to set condition_mask. + + Args: + input_keys: List of input keys (not used, but required by Augmentor interface). + output_keys: List of output keys (not used, but required by Augmentor interface). + args: Dictionary containing: + - "conditioning_config" (dict[int, float], optional): Weighted distribution + mapping latent-frame counts to unnormalized probabilities. + Example: {0: 0.5, 4: 0.3, 8: 0.2}. Clamped to T_latent-1 at runtime. + - "uniform_conditioning" (bool, default False): When True, samples + k ~ Uniform{0, T_latent-1}. Takes precedence over conditioning_config when + both are set. At least one of uniform_conditioning or conditioning_config + must be provided. + - "temporal_compression_factor" (int, default 4): VAE temporal compression + factor used to convert pixel frame count N to T_latent = 1 + (N-1) // tcf. + """ + + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + + if args is None: + args = {} + + self.conditioning_config = args.get("conditioning_config") + self.uniform_conditioning = args.get("uniform_conditioning", False) + self.temporal_compression_factor = args.get("temporal_compression_factor", 4) + + if self.conditioning_config is None and not self.uniform_conditioning: + raise ValueError("args must provide 'conditioning_config' or set 'uniform_conditioning=True'") + + # Validate and normalize probabilities + if self.conditioning_config is not None: + # Validate keys are non-negative integers + for num_frames, prob in self.conditioning_config.items(): + if not isinstance(num_frames, int) or num_frames < 0: + raise ValueError(f"conditioning_config keys must be non-negative integers, got {num_frames}") + if not isinstance(prob, (int, float)) or prob < 0: + raise ValueError(f"conditioning_config values must be non-negative numbers, got {prob}") + + # Normalize probabilities to sum to 1.0 + total_prob = sum(self.conditioning_config.values()) + if total_prob <= 0: + raise ValueError("conditioning_config probabilities must sum to a positive number") + + self.normalized_config = {k: v / total_prob for k, v in self.conditioning_config.items()} + else: + self.normalized_config = {0: 1.0} + + def __call__(self, data_dict: dict) -> dict: + """Create a SequencePlan with random conditional frames. + + Args: + data_dict: Input data dictionary. Should contain "video" key to determine + the number of frames available. + + Returns: + data_dict: Output dictionary with "sequence_plan" key added. + """ + # Get video to determine available frames + video = data_dict.get("video") + if video is None or (self.conditioning_config is None and not self.uniform_conditioning): + # This is an image batch + sequence_plan = SequencePlan( + has_text=True, # Has text prompt! + has_vision=True, + condition_frame_indexes_vision=[], # No conditioning frames! + ) + data_dict["sequence_plan"] = sequence_plan + return data_dict + + # Determine number of frames + # Video should be a tensor with shape (C, T, H, W) by this point in the pipeline + if isinstance(video, torch.Tensor): + assert video.ndim == 4, "video should be a tensor with shape (C, T, H, W)" + num_frames = video.shape[1] + else: + # If video is not a tensor or dict, we can't determine the exact number + # Use a conservative approach - will be limited by max available frames + num_frames = None + + T_latent = 1 + (num_frames - 1) // self.temporal_compression_factor if num_frames is not None else 1 + + # Sample number of conditional frames + if self.uniform_conditioning: + num_conditional_frames = random.randint(0, max(0, T_latent - 1)) + else: + frames_options = list(self.normalized_config.keys()) + weights = list(self.normalized_config.values()) + num_conditional_frames = random.choices(frames_options, weights=weights, k=1)[0] + num_conditional_frames = min(num_conditional_frames, T_latent - 1) if num_frames is not None else 0 + + # Create condition_frame_indexes_vision list + # Conditional frames are always the first N frames + condition_frame_indexes_vision = list(range(num_conditional_frames)) + + # Create SequencePlan + sequence_plan = SequencePlan( + has_text=True, + has_vision=True, + condition_frame_indexes_vision=condition_frame_indexes_vision, + ) + + # Add sequence plan to data dict + data_dict["sequence_plan"] = sequence_plan + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/sound_sequence_plan.py b/cosmos_framework/data/vfm/augmentors/sound_sequence_plan.py new file mode 100644 index 0000000..c359053 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/sound_sequence_plan.py @@ -0,0 +1,95 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Augmentor that builds a SequencePlan for sound-enabled training. + +This augmentor creates a SequencePlan based on the presence of sound data +in the sample, following the same pattern as Action's ActionTransformPipeline +which builds sequence plans for action-enabled training. + +Placed at the END of the augmentor pipeline (after video/audio extraction +and text transforms) so that all data shapes are known. +""" + +from typing import Optional + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.data.vfm.sound_data_utils import VALID_SOUND_MODES, build_sequence_plan_for_sound + + +class SoundSequencePlanBuilder(Augmentor): + """Builds a SequencePlan for sound-enabled samples. + + Inspects the data dict for sound data and creates an appropriate + SequencePlan. If no sound is present, creates a video-only plan. + + Args: + input_keys: Not used (reads from data_dict directly) + output_keys: Not used + args: Dictionary with: + - mode: Generation mode ("t2vs", "tv2s", "ts2v", "ti2sv"). Default: "t2vs" + - video_key: Key to find video data. Default: "video" + - sound_key: Key to find sound data. Default: "sound" + """ + + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + self.mode = args.get("mode", "t2vs") + self.video_key = args.get("video_key", "video") + self.sound_key = args.get("sound_key", "sound") + + assert self.mode in VALID_SOUND_MODES, f"Invalid mode: {self.mode}. Must be one of {VALID_SOUND_MODES}" + + def __call__(self, data_dict: dict) -> dict | None: + """Add sound fields to the existing SequencePlan. + + Only modifies ``has_sound`` and ``condition_frame_indexes_sound``. + All other fields (vision conditioning, action conditioning, etc.) set + by upstream augmentors are preserved. + + If no upstream plan exists, creates a minimal one with sensible defaults. + """ + video = data_dict.get(self.video_key) + sound = data_dict.get(self.sound_key) + + if video is None: + return None # Can't proceed without video + + if not hasattr(video, "shape"): + return None + + video_length = video.shape[1] # (C, T, H, W) → T + + existing_plan = data_dict.get("sequence_plan") + + if existing_plan is not None: + # Update only the sound fields on the existing plan + if sound is not None and hasattr(sound, "shape"): + sound_plan = build_sequence_plan_for_sound( + mode=self.mode, + video_latent_length=video_length, + sound_latent_length=0, + ) + existing_plan.has_sound = sound_plan.has_sound + existing_plan.condition_frame_indexes_sound = sound_plan.condition_frame_indexes_sound + else: + existing_plan.has_sound = False + existing_plan.condition_frame_indexes_sound = [] + else: + # No upstream plan — build a complete one from scratch + if sound is not None and hasattr(sound, "shape"): + data_dict["sequence_plan"] = build_sequence_plan_for_sound( + mode=self.mode, + video_latent_length=video_length, + sound_latent_length=0, + ) + else: + from cosmos_framework.data.vfm.sequence_packing import SequencePlan + + data_dict["sequence_plan"] = SequencePlan( + has_text=True, + has_vision=True, + has_sound=False, + ) + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/text_tokenizer.py b/cosmos_framework/data/vfm/augmentors/text_tokenizer.py new file mode 100644 index 0000000..2f8196a --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/text_tokenizer.py @@ -0,0 +1,96 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# Augmentor for tokenizing input text + +import json +import random +from typing import Optional + +import torch + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils.lazy_config import instantiate as lazy_instantiate + +_MAX_NUM_TOKENS = 4096 + + +class TextTokenizerTransform(Augmentor): + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + + tokenizer_config = self.args["tokenizer_config"] + self.cfg_dropout_rate = self.args["cfg_dropout_rate"] + self.use_system_prompt = self.args.get("use_system_prompt", False) + + self._processor = lazy_instantiate(tokenizer_config) + + def __call__(self, data_dict: dict) -> dict: + input_caption = data_dict[self.input_keys[0]] + + if isinstance(input_caption, dict): + # Encode dict into a json string. This json string is then passed to the transformer tokenizer. + input_caption = json.dumps(input_caption) + data_dict[self.input_keys[0]] = input_caption + + if self.cfg_dropout_rate > 0: + # If CFG is used, randomly dropout the input caption + # We dropout the input caption by replacing it with an empty string + if random.random() < self.cfg_dropout_rate: + input_caption = "" + data_dict[self.input_keys[0]] = input_caption + + text_ids = self._processor.tokenize_text( + input_caption, + is_video=False, + use_system_prompt=self.use_system_prompt, + ) + text_ids = text_ids[:_MAX_NUM_TOKENS] # truncate the text ids to the maximum number of tokens + # This will take care of wierd edge cases where we generate extremely long captions + data_dict[self.output_keys[0]] = torch.tensor(text_ids) # [N_tokens] + return data_dict + + +_SYSTEM_PROMPT_IMAGE_EDITING = "You are a helpful assistant who will edit images based on the user's instructions." + +_SYSTEM_PROMPT_TRANSFER = "You are a helpful assistant that generates images or videos following the user's instructions and control signals (edge maps, blur, depth, or segmentation)." + +_SYSTEM_PROMPTS = { + "editing": _SYSTEM_PROMPT_IMAGE_EDITING, + "transfer": _SYSTEM_PROMPT_TRANSFER, +} + + +class TextTokenizerTransformForEditing(Augmentor): + """Tokenizer augmentor for interleaved tasks: image editing or transfer (control-conditioned generation). + + Uses a task-specific system prompt. Pass args["task"] = "editing" (default) or "transfer". + """ + + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + + tokenizer_config = self.args["tokenizer_config"] + self.cfg_dropout_rate = self.args.get("cfg_dropout_rate", 0.0) + task = self.args.get("task", "editing") + self._system_prompt = _SYSTEM_PROMPTS.get(task, _SYSTEM_PROMPTS["editing"]) + + self._processor = lazy_instantiate(tokenizer_config) + + def __call__(self, data_dict: dict) -> dict | None: + input_caption = data_dict.get(self.input_keys[0], "") + if self.cfg_dropout_rate > 0 and random.random() < self.cfg_dropout_rate: + input_caption = "" + data_dict[self.input_keys[0]] = input_caption + text_ids = self._processor.tokenize_text(input_caption, system_prompt=self._system_prompt) + data_dict[self.output_keys[0]] = torch.tensor(text_ids) # [N_tokens] + return data_dict + + +class TextTokenizerTransformForTransfer(TextTokenizerTransformForEditing): + """Tokenizer augmentor for transfer (control-conditioned) generation. Uses transfer system prompt.""" + + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + args = dict(args) if args else {} + args["task"] = "transfer" + super().__init__(input_keys, output_keys, args) diff --git a/cosmos_framework/data/vfm/augmentors/text_transforms_for_image.py b/cosmos_framework/data/vfm/augmentors/text_transforms_for_image.py new file mode 100644 index 0000000..51700ea --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/text_transforms_for_image.py @@ -0,0 +1,296 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import json +import random +from typing import Optional + +from imaginaire.datasets.augmentors.v3_text_transforms import pad_and_resize +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.data_sources.data_registration import _CAPTION_EMBEDDING_KEY_MAPPING_IMAGES + +# For the qwen captions, we have 3 variants: short, medium, long +# In addition, for synthetic data, we create prompt embeddings as well. +# There is quite a bit of entropy in the way prompt data is saved. +# Captions are saved as "prompts", while the corresponding embeddings are saved as "original_prompt" +# This part will be cleaned after synthetic data is cleaned to be in the same format as real data. +_AVAILABLE_QWEN_CAPTIONS = ["qwen2p5_7b_short", "qwen2p5_7b_medium", "qwen2p5_7b_long"] +_AVAILABLE_QWEN3_30B_A3B_CAPTIONS = [ + "qwen3_30b_a3b_short", + "qwen3_30b_a3b_descriptive", + "qwen3_30b_a3b_dense", +] +# used for new caption in Nov 2025 +_AVAILABLE_CAPTIONS_V2 = ["caption_short", "caption_medium", "caption_long"] +# used for sft v1 +_AVAILABLE_CAPTIONS_SFT_V1 = [ + "gemini_v1_dense", + "gemini_v2_dense", + "qwen3vl_30B_v1_dense", + "qwen3vl_30B_v2_dense", + "qwen3vl_235B_v1_dense", + "qwen3vl_235B_v2_dense", +] +# used for genplan ablation +# captions are saved as "caption_long" as a JSON string, like {"dense": "xxx", "dense_bbox": "xxx"} +_AVAILABLE_CAPTIONS_GENPLAN = ["dense", "dense_bbox"] +_CAPTION_EMBEDDING_MAPPING = { + "qwen2p5_7b_short": "qwen2p5_7b_short", + "qwen2p5_7b_medium": "qwen2p5_7b_medium", + "qwen2p5_7b_long": "qwen2p5_7b_long", + "prompts": "original_prompt", +} + + +class TextTransformForImage(Augmentor): + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + + def __call__(self, data_dict: dict) -> dict: + r"""Performs camera transformation. + + Args: + data_dict (dict): Input data dict + Returns: + data_dict (dict): Output dict with camera attributes added + """ + + caption_type = self.args["caption_type"] + embedding_key_in_dict = _CAPTION_EMBEDDING_KEY_MAPPING_IMAGES[caption_type] + embedding_type = self.args["embedding_type"] + embedding_input_key_prefix = "" if embedding_type == "t5_xxl" else "umt5_" + + captions_key, embeddings_key = ( + f"captions_{caption_type}", + f"{embedding_input_key_prefix}embeddings_captions_{embedding_key_in_dict}", + ) + decoded_captions_ai = data_dict[captions_key] + decoded_embeddings_ai = data_dict[embeddings_key] + + try: + # Hotfix: Some captions are labeled as "captions" and some are labeled as "caption" + # This issue needs to be fixed in the synthetic data. This is a hack and will be removed + # once the data is cleaned. + caption_key = "captions" if "captions" in decoded_captions_ai else "caption" + embedding_key = "t5_xxl_fp8" if embedding_type == "t5_xxl" else "umt5_xxl" + if caption_type == "qwen2p5_7b_v4": + selected_caption_type = random.choice(_AVAILABLE_QWEN_CAPTIONS) + data_dict["ai_caption"] = decoded_captions_ai[caption_key][selected_caption_type] + t5_embedding = decoded_embeddings_ai[selected_caption_type]["embeddings"][embedding_key] + data_dict["selected_caption_type"] = selected_caption_type + elif caption_type == "prompts": + data_dict["ai_caption"] = decoded_captions_ai["caption"]["prompt"] + t5_embedding = decoded_embeddings_ai[_CAPTION_EMBEDDING_MAPPING[caption_type]]["embeddings"][ + embedding_key + ] + data_dict["selected_caption_type"] = caption_type + else: + assert caption_type == "ai_v3p1", f"Caption type {caption_type} not supported" + if decoded_captions_ai["had_parse_issue"]: + data_dict["ai_caption"] = decoded_captions_ai["captions"]["kosmos_2"] + t5_embedding = decoded_embeddings_ai["kosmos2"]["embeddings"][embedding_key] + else: + data_dict["ai_caption"] = decoded_captions_ai["captions"]["vfc"] + t5_embedding = decoded_embeddings_ai["vfc_fidelity"]["embeddings"][embedding_key] + + out_t5, out_t5_mask = pad_and_resize( + t5_embedding, + self.args["t5_tokens"]["num"], + is_mask_all_ones=self.args["is_mask_all_ones"], + ) + data_dict["t5_text_embeddings"] = out_t5 + data_dict["t5_text_mask"] = out_t5_mask + except Exception as e: + log.warning( + f"TextTransform dataloader error: {data_dict['__url__']}, {data_dict['__key__']}\n error {e}", + rank0_only=False, + ) + return None + + del data_dict[captions_key] + del data_dict[embeddings_key] + + return data_dict + + +class TextTransformForImageWithoutEmbeddings(Augmentor): + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + self.caption_prefix = args.get("caption_prefix", None) if args else None + + def _apply_caption_prefix(self, data_dict: dict) -> None: + """Prepend caption_prefix to ai_caption if configured.""" + if not self.caption_prefix or not isinstance(data_dict.get("ai_caption"), str): + return + original = data_dict["ai_caption"] + data_dict["ai_caption"] = self.caption_prefix + " " + original.lstrip() + log.debug( + f"[caption_prefix] before: {original[:120]!r}... | after: {data_dict['ai_caption'][:120]!r}...", + rank0_only=False, + ) + + def __call__(self, data_dict: dict) -> dict: + r"""Performs text transform without any embedding loading. + This is useful for online computation. + + Args: + data_dict (dict): Input data dict + Returns: + data_dict (dict): Output dict with camera attributes added + """ + + caption_type = self.args["caption_type"] + captions_key = f"captions_{caption_type}" + decoded_captions_ai = data_dict[captions_key] + + train_on_captions = self.args.get("train_on_captions", []) + # if [], will infer based on the caption json + # otherwise it will only use the captions in the list + + try: + # Hotfix: Some captions are labeled as "captions" and some are labeled as "caption" + # This issue needs to be fixed in the synthetic data. This is a hack and will be removed + # once the data is cleaned. + caption_key = "captions" if "captions" in decoded_captions_ai else "caption" + if len(train_on_captions) == 0: + # infer which caption types are there + if caption_type in ("generated_gpt_oss_20b", "generated_gpt_oss_120b"): + selected_caption_type = "caption_long" + if caption_key in decoded_captions_ai: # sharded with sila pipeline + data_dict["ai_caption"] = decoded_captions_ai[caption_key][selected_caption_type] + else: + data_dict["ai_caption"] = decoded_captions_ai[selected_caption_type] + data_dict["selected_caption_type"] = selected_caption_type + elif caption_type == "qwen3_30b_a3b": + selected_caption_type = random.choice(_AVAILABLE_QWEN3_30B_A3B_CAPTIONS) + data_dict["ai_caption"] = decoded_captions_ai[selected_caption_type] + data_dict["selected_caption_type"] = selected_caption_type + elif caption_type == "qwen3_235b_a22b_v0": + # Synthetic scene-text data ingested via + # pipelines/image/text_rendering/ingest_webdataset.py stores captions as a flat + # dict {"caption_short": ..., "caption_long": ...} (no "caption"/"captions" + # nesting), so we index decoded_captions_ai directly. + available = [k for k in _AVAILABLE_CAPTIONS_V2 if k in decoded_captions_ai] + if not available: + raise KeyError( + f"No known caption keys for {caption_type} in {list(decoded_captions_ai.keys())}" + ) + selected_caption_type = random.choice(available) + data_dict["ai_caption"] = decoded_captions_ai[selected_caption_type] + data_dict["selected_caption_type"] = selected_caption_type + elif any( + caption_type in _AVAILABLE_QWEN_CAPTIONS for caption_type in decoded_captions_ai[caption_key].keys() + ): + # qwen2p5_7b_v4 captions + selected_caption_type = random.choice(_AVAILABLE_QWEN_CAPTIONS) + data_dict["ai_caption"] = decoded_captions_ai[caption_key][selected_caption_type] + data_dict["selected_caption_type"] = selected_caption_type + elif caption_type == "cosmos_captioner_v1p1": + selected_caption_type = "caption_cosmos_captioner_image" + if decoded_captions_ai[caption_key].get(selected_caption_type, "") == "": + # xingqianx: a temporary skip as some data is imperfect. + return None # type: ignore + data_dict["ai_caption"] = decoded_captions_ai[caption_key][selected_caption_type] + data_dict["selected_caption_type"] = selected_caption_type + elif caption_type == "cosmos_captioner_v1p1_structured_json": + # this is made by mistake, should be removed in future. + # it is used for cosmos_lab_image_v1_human_sft. Once we fix it, this should be removed. + selected_caption_type = "caption_cosmos_captioner_image_structured_json" + data_dict["ai_caption"] = decoded_captions_ai[caption_key][selected_caption_type] + data_dict["selected_caption_type"] = selected_caption_type + elif any( + caption_type in _AVAILABLE_CAPTIONS_V2 for caption_type in decoded_captions_ai[caption_key].keys() + ): + # v2 captions + selected_caption_type = random.choice(_AVAILABLE_CAPTIONS_V2) + data_dict["ai_caption"] = decoded_captions_ai[caption_key][selected_caption_type] + data_dict["selected_caption_type"] = selected_caption_type + elif caption_type == "prompts": + data_dict["ai_caption"] = decoded_captions_ai["caption"]["prompt"] + data_dict["selected_caption_type"] = caption_type + else: + assert caption_type == "ai_v3p1", f"Caption type {caption_type} not supported" + if decoded_captions_ai["had_parse_issue"]: + data_dict["ai_caption"] = decoded_captions_ai["captions"]["kosmos_2"] + else: + data_dict["ai_caption"] = decoded_captions_ai["captions"]["vfc"] + else: # use the designated captions + # Validate that all specified caption types exist (except genplan types which are nested) + for cap_type in train_on_captions: + if cap_type not in _AVAILABLE_CAPTIONS_GENPLAN: + assert cap_type in decoded_captions_ai[caption_key].keys(), ( + f"Caption type {cap_type} not found in data" + ) + + selected_caption_type = random.choice(train_on_captions) + + if selected_caption_type in _AVAILABLE_CAPTIONS_GENPLAN: + # Genplan captions are nested inside caption_long as a JSON string + caption_long_data = json.loads(decoded_captions_ai[caption_key]["caption_long"]) + data_dict["ai_caption"] = caption_long_data[selected_caption_type] + else: + data_dict["ai_caption"] = decoded_captions_ai[caption_key][selected_caption_type] + data_dict["selected_caption_type"] = selected_caption_type + + except Exception as e: + log.warning( + f"TextTransform dataloader error: {data_dict['__url__']}, {data_dict['__key__']}\n error {e}", + rank0_only=False, + ) + return None + + del data_dict[captions_key] + + self._apply_caption_prefix(data_dict) + return data_dict + + +class TextTransformForImageJsonCaption(Augmentor): + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + self.json_field_dropout_rate = args.get("json_field_dropout_rate", 0.0) if args else 0.0 + + def __call__(self, data_dict: dict) -> dict: + r"""Performs text transform without any embedding loading. + This is useful for online computation. + + Args: + data_dict (dict): Input data dict + Returns: + data_dict (dict): Output dict with camera attributes added + """ + + caption_type = self.args["caption_type"] + captions_key = f"captions_{caption_type}" + + if "cosmos_captioner_v1p1_structured_json" in data_dict[captions_key]: + # this is made by mistake, should be removed in future. + # it is used for cosmos_lab_image_v1_human_sft. Once we fix it, this should be removed. + selected_caption_type = "caption_cosmos_captioner_image_structured_json" + else: + selected_caption_type = "caption_cosmos_captioner_image" + caption_json = data_dict[captions_key]["caption"].get(selected_caption_type, "") + if caption_json == "": + # xingqianx: a temporary skip as some text data is imperfect. + return None # type: ignore + caption_json = json.loads(caption_json) + + # In some erraneous cases, the caption_json is a list + if isinstance(caption_json, list): + caption_json = caption_json[0] + + assert isinstance(caption_json, dict), ( + f"Caption json is not a dict: {caption_json}, url: {data_dict['__url__']}, key: {data_dict['__key__']}" + ) + + # Randomly dropout json keys during training + if self.json_field_dropout_rate > 0: + for key in list(caption_json.keys()): + if random.random() < self.json_field_dropout_rate: + caption_json.pop(key) + + data_dict["ai_caption"] = caption_json + del data_dict[captions_key] + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/text_transforms_for_video.py b/cosmos_framework/data/vfm/augmentors/text_transforms_for_video.py new file mode 100644 index 0000000..9ddd1ae --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/text_transforms_for_video.py @@ -0,0 +1,721 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import json +import random +from typing import Optional + +from imaginaire.datasets.augmentors.v3_text_transforms import pad_and_resize +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log + + +class TextTransformForVideo(Augmentor): + def __init__(self, input_keys: dict, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + + # our caption is saved in json with format: {"": "xxx", "": [{"start_frame": x, "end_frame": x, "": xxx}, ...], "": [{"start_frame":...]} + # our t5 embedding is saved in pickle with format: [{"": array1, "": array2}, ...] + self.captions_key: str = args[ + "captions_key" + ] # s3 folder that saves the captions; this get mapped to the key in data_dict to fetch the caption field + self.embeddings_key: Optional[str] = args[ + "embeddings_key" + ] # s3 folder that saves the embeddings; this get mapped to the key in data_dict to fetch the embedding field + self.caption_windows_key: str = args[ + "caption_windows_key" + ] # key to get the caption windows from the caption field + self.caption_type: str = args["caption_type"] # key of caption type to fetch the caption from caption windows + + self._load_embeddings = self.embeddings_key is not None + + if not self._load_embeddings: + # In this case, we don't load the embeddings + log.info("No embeddings key provided, we will not load embeddings") + self.embedding_caption_type = None + self.t5_tokens_num = None + self.is_mask_all_ones = None + self.embedding_style_mapping = None + else: + self.embedding_caption_type: str = args[ + "embedding_caption_type" + ] # key to get the embedding of a particular caption type from the embedding field + self.t5_tokens_num = args["t5_tokens"]["num"] # number of tokens we cap after padding + self.is_mask_all_ones = args["is_mask_all_ones"] # if true, set mask for t5 to all ones + + self.embedding_style_mapping = { + "long": self.embedding_caption_type, + "short": f"{self.embedding_caption_type}_short", + "medium": f"{self.embedding_caption_type}_medium", + "user": f"{self.embedding_caption_type}_user", + } + + self.caption_probs: dict[str, float] = args[ + "caption_probs" + ] # probabilities for user/short/medium/long captions + self.caption_style_mapping = { + "long": self.caption_type, + "short": f"{self.caption_type}_short", + "medium": f"{self.caption_type}_medium", + "user": f"{self.caption_type}_user", + } + assert self.caption_probs.keys() == self.caption_style_mapping.keys(), ( + "The keys for caption_probs, caption_style_mapping, and embedding_style_mapping should match" + ) + + if self._load_embeddings: + assert self.caption_style_mapping.keys() == self.embedding_style_mapping.keys(), ( + "The keys for caption_style_mapping and embedding_style_mapping should match" + ) + + def __call__(self, data_dict: dict) -> dict: + r"""Performs text transformation. + + Args: + data_dict (dict): Input data dict + Returns: + data_dict (dict): Output dict with captions and t5 embeddings added + """ + + try: + windows = data_dict[self.captions_key][self.caption_windows_key] + n_windows = len(windows) + chunk_index = data_dict["chunk_index"] + + if chunk_index == n_windows: + # This will only happen when the number of captions does not match number of chunks due to re-transcoding the videos. + log.warning( + f"TextTransform dataloader error: Found {data_dict['n_orig_video_frames']} in video but captioning is done with videos of {windows[-1]['end_frame']} frames. This mismatch is due to video re-transcoding.", + rank0_only=False, + ) + chunk_index -= 1 + + selected_caption_window = windows[chunk_index] + except Exception as e: + log.warning( + f"TextTransform dataloader error -- url: {data_dict['__url__']}, key: {data_dict['__key__']}, chunk_index: {data_dict['chunk_index']}\n error {e}", + rank0_only=False, + ) + return None + + sampled_caption_style = None + try: + available_caption_styles = [] + for k in selected_caption_window.keys(): + caption_style = k.replace(self.caption_type, "").replace("_", "") + if caption_style == "": # it is long caption by default + available_caption_styles.append("long") + elif caption_style in self.caption_style_mapping: + available_caption_styles.append(caption_style) + else: + assert caption_style in ["startframe", "endframe"], f"Unsupported caption_type {caption_style}" + + probabilities_for_available_caption_styles = { + k: v for k, v in self.caption_probs.items() if k in available_caption_styles + } + sampled_caption_style = random.choices( + list(probabilities_for_available_caption_styles), + weights=probabilities_for_available_caption_styles.values(), + )[0] + data_dict["ai_caption"] = selected_caption_window[self.caption_style_mapping[sampled_caption_style]] + except Exception as e: + log.warning( + f"TextTransform dataloader error -- url: {data_dict['__url__']}, key: {data_dict['__key__']}, selected_caption_window: {selected_caption_window}\n error {e}", + rank0_only=False, + ) + return None + if data_dict["ai_caption"] == "": + log.warning( + f"TextTransform dataloader error -- empty caption! url: {data_dict['__url__']}, key: {data_dict['__key__']}, selected_caption_window: {selected_caption_window}", + rank0_only=False, + ) + return None + + assert data_dict["ai_caption"] is not None and sampled_caption_style is not None + data_dict["sampled_caption_style"] = sampled_caption_style + + del data_dict[self.captions_key] # delete the field as we have extracted ai_caption from it + + if self._load_embeddings: + ai_caption_embedding_data = data_dict[self.embeddings_key] + try: + if self.embedding_caption_type == "vila_caption": + t5_embedding = ai_caption_embedding_data[chunk_index] + else: + t5_embedding = ai_caption_embedding_data[chunk_index][ + self.embedding_style_mapping[sampled_caption_style] + ] + except Exception as e: + log.warning( + f"TextTransform dataloader error -- url: {data_dict['__url__']}, key: {data_dict['__key__']}, chunk_index: {data_dict['chunk_index']}, n embeddings: {len(ai_caption_embedding_data)}, n captions: {n_windows} \n error {e}", + rank0_only=False, + ) + return None + out_t5, out_t5_mask = pad_and_resize( + t5_embedding, + self.t5_tokens_num, + is_mask_all_ones=self.is_mask_all_ones, + ) + data_dict["t5_text_embeddings"] = out_t5 + data_dict["t5_text_mask"] = out_t5_mask + del data_dict[self.embeddings_key] # delete the field as we have extracted t5 embedding from it + + return data_dict + + +class TextTransformForVideoWithFullFrames(Augmentor): + """ + Pair use with VideoParsingWithFullFrames to get the full frames of the video. + The caption is assumed to be for the entire video frames, rather than TextTransformForVideo + which assumes captions are for a specific chunk of frames. + + Audio captions are handled separately by AudioCaptionAppender, which appends + audio descriptions to the video caption after this augmentor runs. + """ + + def __init__(self, input_keys: dict, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + assert len(input_keys) == 3, "TextTransformForVideoWithFullFrames augmentor only supports three input keys" + self.meta_key = input_keys[0] + self.video_key = input_keys[1] + self.sequence_plan_key = input_keys[2] + self.args = args + self.keep_metas = args.get("keep_metas", False) if args else False + self.caption_prefix = args.get("caption_prefix", None) if args else None + + def _apply_caption_prefix(self, data_dict: dict) -> None: + """Prepend caption_prefix to ai_caption if configured.""" + if not self.caption_prefix or not isinstance(data_dict.get("ai_caption"), str): + return + original = data_dict["ai_caption"] + data_dict["ai_caption"] = self.caption_prefix + " " + original.lstrip() + log.debug( + f"[caption_prefix] before: {original[:120]!r}... | after: {data_dict['ai_caption'][:120]!r}...", + rank0_only=False, + ) + + @staticmethod + def _resolve_multi_chunk_caption(raw_caption: str) -> str: + """Resolve a caption that may be in multi-chunk JSON format. + + Multi-chunk captions are JSON strings encoding a dict of chunks, e.g.: + {"chunk_0_300": {"caption": "...", "start_frame": 0, "end_frame": 300}, ...} + When detected, a chunk is randomly selected and its "caption" text returned. + Plain string captions are returned unchanged. + """ + if not isinstance(raw_caption, str): + return raw_caption + try: + parsed = json.loads(raw_caption) + except (json.JSONDecodeError, TypeError): + return raw_caption + if not isinstance(parsed, dict) or len(parsed) == 0: + return raw_caption + chunk = random.choice(list(parsed.values())) + if isinstance(chunk, dict) and "caption" in chunk: + return chunk["caption"] + return raw_caption + + def __call__(self, data_dict: dict) -> dict: + r"""Performs text transformation. + + Samples a video caption from metadata based on caption_config ratios. + Supports both plain-string captions and multi-chunk JSON captions + (randomly selects one chunk when multiple chunks are present). + Audio captions are handled separately by AudioCaptionAppender. + + Args: + data_dict (dict): Input data dict + Returns: + data_dict (dict): Output dict with captions and t5 embeddings added + """ + caption_config = self.args["caption_config"] + meta_dict = data_dict[self.meta_key] + + for caption_type in caption_config: + assert caption_type in meta_dict, ( + f"Caption type {caption_type} not found in meta_dict (keys = {meta_dict.keys()})" + ) + + # First check if we are doing image to world or video to world + if self.sequence_plan_key in data_dict: + sequence_plan = data_dict[self.sequence_plan_key] + conditioning_frame_indexes_vision = sequence_plan.condition_frame_indexes_vision + if len(conditioning_frame_indexes_vision) > 0: + sampled_caption = self._resolve_multi_chunk_caption(meta_dict["caption_temporal"]) + data_dict["ai_caption"] = sampled_caption + data_dict["sampled_caption_style"] = "caption_temporal" + + self._apply_caption_prefix(data_dict) + if not self.keep_metas: + del data_dict[self.meta_key] + return data_dict + + # Text-to-world: sample from short, medium, long captions + caption_keys = list(caption_config.keys()) + caption_ratios = [caption_config[k]["ratio"] for k in caption_keys] + sampled_caption_type = random.choices(caption_keys, weights=caption_ratios, k=1)[0] + data_dict["ai_caption"] = self._resolve_multi_chunk_caption(meta_dict[sampled_caption_type]) + data_dict["sampled_caption_style"] = sampled_caption_type + + self._apply_caption_prefix(data_dict) + + # Clean up - delete the caption fields that were sampled from + for caption_type in caption_config.keys(): + if caption_type in meta_dict: + del meta_dict[caption_type] + + # Delete metas unless keep_metas=True (set when AudioCaptionAppender runs downstream) + if not self.keep_metas: + del data_dict[self.meta_key] + + return data_dict + + +class TextTransformForVideoTransferFullFrames(Augmentor): + """Read structured captions for the full-frame transfer pipeline. + + Two-level lookup: + + 1. A caption-source key is sampled (with weights) from ``caption_config``. + This key identifies the WebDataset folder / metadata field whose value + is a dict of annotations (e.g. + ``"structured_captions_qwen3-vl-8b-lora-v1.5-merged"``). The sampled + value is looked up first in ``data_dict`` (top-level) and then in + ``meta_dict``. + 2. Inside that caption dict the field ``caption_structured`` is hardcoded + as the JSON-encoded chunked annotation, of the form + ``{"chunk_0_300": {"caption": "", + "start_frame": ..., "end_frame": ...}}``. + + The full-frame pipeline always decodes from the start of the video, so the + first chunk is always selected and its inner JSON-encoded structured payload + is parsed back into a dict before being serialized as ``ai_caption``. + """ + + CAPTION_FIELD = "caption_structured" + + def __init__(self, input_keys: dict, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + assert len(input_keys) >= 1, "TextTransformForVideoTransferFullFrames requires a metadata input key" + self.meta_key = input_keys[0] + self.args = args or {} + self.keep_metas = self.args.get("keep_metas", False) + self.caption_options = self._normalize_caption_config(self.args["caption_config"]) + # This fixes transfer datasets that mix caption chunks with different + # lengths. Each caption source needs its own stride so the sampled video + # stays within the token budget while matching the selected caption. + self.min_stride_key = self.args.get("min_stride_key", "_full_frames_min_stride") + + @staticmethod + def _normalize_caption_config(caption_config: dict | list) -> list[tuple[str, float, dict]]: + if isinstance(caption_config, dict): + options: list[tuple[str, float, dict]] = [] + for caption_key, config in caption_config.items(): + if isinstance(config, dict): + ratio = config.get("ratio", 1.0) + # Keep more than the sampling ratio because source-specific + # settings, like min_stride, are part of how caption/video + # alignment is preserved. + options.append((caption_key, float(ratio), dict(config))) + else: + options.append((caption_key, float(config), {})) + return options + + options = [] + for item in caption_config: + if isinstance(item, str): + options.append((item, 1.0, {})) + elif isinstance(item, dict): + caption_key = item.get("key") or item.get("caption_key") or item.get("caption_type") or item.get("name") + if caption_key is None: + raise ValueError(f"Caption config entry is missing a caption key: {item}") + options.append((caption_key, float(item.get("ratio", 1.0)), dict(item))) + else: + caption_key, ratio = item + options.append((caption_key, float(ratio), {})) + return options + + def _lookup_caption_dict(self, data_dict: dict, meta_dict: dict | None, caption_key: str) -> dict | None: + candidate = data_dict.get(caption_key) + if candidate is None and isinstance(meta_dict, dict): + candidate = meta_dict.get(caption_key) + if isinstance(candidate, dict): + return candidate + return None + + def __call__(self, data_dict: dict) -> dict | None: + meta_dict = data_dict.get(self.meta_key) + + available_options: list[tuple[str, float, dict]] = [] + for key, ratio, option in self.caption_options: + if ratio <= 0: + continue + if self._lookup_caption_dict(data_dict, meta_dict, key) is not None: + available_options.append((key, ratio, option)) + + if not available_options: + log.warning( + f"TextTransformForVideoTransferFullFrames: none of the configured caption keys " + f"{[key for key, _, _ in self.caption_options]} hold a caption dict in metadata/sample keys. " + f"url: {data_dict.get('__url__')}, key: {data_dict.get('__key__')}", + rank0_only=False, + ) + return None + + sampled_caption_key = random.choices( + [key for key, _, _ in available_options], + weights=[ratio for _, ratio, _ in available_options], + k=1, + )[0] + sampled_caption_option = next(option for key, _, option in available_options if key == sampled_caption_key) + caption_dict = self._lookup_caption_dict(data_dict, meta_dict, sampled_caption_key) + if caption_dict is None or self.CAPTION_FIELD not in caption_dict: + log.warning( + f"TextTransformForVideoTransferFullFrames: caption dict for {sampled_caption_key} is missing the " + f"hardcoded {self.CAPTION_FIELD} field. url: {data_dict.get('__url__')}, key: {data_dict.get('__key__')}", + rank0_only=False, + ) + return None + + try: + chunks = json.loads(caption_dict[self.CAPTION_FIELD]) + first_chunk = next(iter(chunks.values())) + structured = json.loads(first_chunk["caption"]) + except Exception as e: + log.warning( + f"TextTransformForVideoTransferFullFrames: failed to decode {sampled_caption_key}.{self.CAPTION_FIELD}. " + f"url: {data_dict.get('__url__')}, key: {data_dict.get('__key__')}, error: {e}", + rank0_only=False, + ) + return None + + data_dict["ai_caption"] = json.dumps(structured) + data_dict["sampled_caption_style"] = sampled_caption_key + if "min_stride" in sampled_caption_option: + # Without this override, 200-frame and 400-frame caption sources + # would share one stride and could either waste context or overflow + # the intended token length. + data_dict[self.min_stride_key] = int(sampled_caption_option["min_stride"]) + + if not self.keep_metas: + data_dict.pop(self.meta_key, None) + for caption_key, _, _ in self.caption_options: + if caption_key in data_dict: + del data_dict[caption_key] + return data_dict + + +class TextTransformForVideoTransferChunkedFrames(TextTransformForVideoTransferFullFrames): + """Read structured captions and sample one chunk for transfer training. + + This keeps the full-frame caption-source sampling behavior, including + per-source options such as ``min_stride``, but emits the sampled chunk's + frame range so the downstream parser can decode the matching RGB/control + frames. + """ + + def __init__(self, input_keys: dict, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + # The parser still needs metadata for fps/resolution after this transform. + self.keep_metas = self.args.get("keep_metas", True) + self.min_num_frames = int(self.args.get("min_num_frames", 5)) + + def __call__(self, data_dict: dict) -> dict | None: + meta_dict = data_dict.get(self.meta_key) + + available_options: list[tuple[str, float, dict]] = [] + for key, ratio, option in self.caption_options: + if ratio <= 0: + continue + if self._lookup_caption_dict(data_dict, meta_dict, key) is not None: + available_options.append((key, ratio, option)) + + if not available_options: + log.warning( + f"TextTransformForVideoTransferChunkedFrames: none of the configured caption keys " + f"{[key for key, _, _ in self.caption_options]} hold a caption dict in metadata/sample keys. " + f"url: {data_dict.get('__url__')}, key: {data_dict.get('__key__')}", + rank0_only=False, + ) + return None + + sampled_caption_key = random.choices( + [key for key, _, _ in available_options], + weights=[ratio for _, ratio, _ in available_options], + k=1, + )[0] + sampled_caption_option = next(option for key, _, option in available_options if key == sampled_caption_key) + caption_dict = self._lookup_caption_dict(data_dict, meta_dict, sampled_caption_key) + if caption_dict is None or self.CAPTION_FIELD not in caption_dict: + log.warning( + f"TextTransformForVideoTransferChunkedFrames: caption dict for {sampled_caption_key} is missing the " + f"hardcoded {self.CAPTION_FIELD} field. url: {data_dict.get('__url__')}, key: {data_dict.get('__key__')}", + rank0_only=False, + ) + return None + + try: + chunks = json.loads(caption_dict[self.CAPTION_FIELD]) + if not isinstance(chunks, dict) or len(chunks) == 0: + log.warning( + f"TextTransformForVideoTransferChunkedFrames: empty chunk dict for {sampled_caption_key}. " + f"url: {data_dict.get('__url__')}, key: {data_dict.get('__key__')}", + rank0_only=False, + ) + return None + + eligible_chunk_keys: list[str] = [] + for chunk_key, chunk in chunks.items(): + try: + start_frame = int(chunk["start_frame"]) + end_frame = int(chunk["end_frame"]) + except (KeyError, TypeError, ValueError): + continue + if end_frame - start_frame >= self.min_num_frames: + eligible_chunk_keys.append(chunk_key) + + if not eligible_chunk_keys: + log.warning( + f"TextTransformForVideoTransferChunkedFrames: no chunks with >= {self.min_num_frames} frames " + f"in {sampled_caption_key}. url: {data_dict.get('__url__')}, key: {data_dict.get('__key__')}", + rank0_only=False, + ) + return None + + sampled_chunk_key = random.choice(eligible_chunk_keys) + sampled_chunk = chunks[sampled_chunk_key] + chunk_start_frame = int(sampled_chunk["start_frame"]) + chunk_end_frame = int(sampled_chunk["end_frame"]) + structured = json.loads(sampled_chunk["caption"]) + except Exception as e: + log.warning( + f"TextTransformForVideoTransferChunkedFrames: failed to decode {sampled_caption_key}.{self.CAPTION_FIELD}. " + f"url: {data_dict.get('__url__')}, key: {data_dict.get('__key__')}, error: {e}", + rank0_only=False, + ) + return None + + data_dict["chunk_start_frame"] = chunk_start_frame + data_dict["chunk_end_frame"] = chunk_end_frame + data_dict["ai_caption"] = json.dumps(structured) + data_dict["sampled_caption_style"] = sampled_caption_key + data_dict["sampled_chunk_key"] = sampled_chunk_key + if "min_stride" in sampled_caption_option: + data_dict[self.min_stride_key] = int(sampled_caption_option["min_stride"]) + + if not self.keep_metas: + data_dict.pop(self.meta_key, None) + for caption_key, _, _ in self.caption_options: + if caption_key in data_dict: + del data_dict[caption_key] + return data_dict + + +class TextTransformForVideoJsonCaption(Augmentor): + """ + This augmentor is used to transform the caption from a json string to a string. + The caption is assumed to be in the format of a json string. + The caption is then transformed to a string by converting the json string to a dictionary and then converting the dictionary to a string. + The caption is then returned as a string. + + When ``meta_dict["caption_audio"]`` is present and non-empty, its contents + are injected into the caption dict under the ``"audio_description"`` key. + This happens after the JSON field dropout so the audio description is + preserved whenever upstream metadata provides it. + """ + + def __init__(self, input_keys: dict, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + assert len(input_keys) >= 2, ( + "TextTransformForVideoJsonCaption augmentor requires at least two input keys: [meta_key, video_key]" + ) + self.meta_key = input_keys[0] + self.video_key = input_keys[1] + self.args = args or {} + self.keep_metas = self.args.get("keep_metas", False) + self.caption_key = self.args.get("caption_key", "caption") + + @staticmethod + def _json_dict_or_none(raw_caption: object) -> dict | None: + if isinstance(raw_caption, dict): + return raw_caption + if not isinstance(raw_caption, str) or len(raw_caption) == 0: + return None + try: + parsed_caption = json.loads(raw_caption) + except (json.JSONDecodeError, TypeError): + return None + return parsed_caption if isinstance(parsed_caption, dict) else None + + @staticmethod + def _frame_count_or_large_bound(meta_dict: dict) -> int: + nb_frames = meta_dict.get("nb_frames") + if isinstance(nb_frames, int) and nb_frames > 0: + return nb_frames + length = meta_dict.get("length") + framerate = meta_dict.get("framerate") + if isinstance(length, (float, int)) and isinstance(framerate, (float, int)) and length > 0 and framerate > 0: + return max(1, int(round(length * framerate))) + # VideoParsingChunkedFrames clamps to the decoder length, so a large end frame is safe. + return 10**9 + + def _find_audio_caption(self, meta_dict: dict) -> str | None: + audio_caption = meta_dict.get("caption_audio") + if isinstance(audio_caption, str) and len(audio_caption) > 0: + return audio_caption + for value in meta_dict.values(): + if isinstance(value, dict): + caption_sound = value.get("caption_sound") + if isinstance(caption_sound, str) and len(caption_sound) > 0: + return caption_sound + return None + + def _parse_legacy_full_video_caption(self, meta_dict: dict) -> dict[str, dict] | None: + """Build one full-video chunk from older caption schemas that do not have ``caption``.""" + caption_json = self._json_dict_or_none(meta_dict.get("caption_structured")) + if caption_json is None: + for caption_key in ( + "caption_rewrite_dense", + "caption_dense", + "caption_descriptive", + "caption_base", + "caption_temporal", + "caption_short", + ): + caption_text = meta_dict.get(caption_key) + if isinstance(caption_text, str) and len(caption_text) > 0: + caption_json = {"description": caption_text} + break + if caption_json is None: + return None + + end_frame = self._frame_count_or_large_bound(meta_dict) + return { + f"chunk_0_{end_frame}": { + "start_frame": 0, + "end_frame": end_frame, + "caption_json": caption_json, + } + } + + def _parse_audio_caption_chunks(self, meta_dict: dict) -> dict[str, dict] | None: + """Build chunk metadata from nested audio-caption metas when visual captions are absent.""" + chunks: dict[str, dict] = {} + for key, value in meta_dict.items(): + if not isinstance(key, str) or not isinstance(value, dict): + continue + caption_sound = value.get("caption_sound") + if not isinstance(caption_sound, str) or len(caption_sound) == 0: + continue + + try: + start_frame, end_frame = [int(part) for part in key.split("_", maxsplit=1)] + except ValueError: + start_frame = value.get("start_frame") + end_frame = value.get("end_frame") + if not isinstance(start_frame, int) or not isinstance(end_frame, int): + continue + + chunks[key] = { + "start_frame": start_frame, + "end_frame": end_frame, + "caption_json": {"audio_description": caption_sound}, + } + + return chunks or None + + def __call__(self, data_dict: dict) -> dict | None: + r"""Performs text transformation. + + Parses the per-chunk caption JSON, randomly samples one chunk, and writes + the chunk's frame range into ``data_dict`` so a downstream + ``VideoParsingChunkedFrames`` can decode only that frame range. When a + non-empty ``caption_audio`` field is present in the metadata, it is + injected into the caption dict under the ``"audio_description"`` key. + + Args: + data_dict (dict): Input data dict + Returns: + data_dict (dict): Output dict with captions and t5 embeddings added + """ + caption_config = self.args["caption_config"] + json_field_dropout_rate = caption_config["json_field_dropout_rate"] + + try: + meta_dict = data_dict[self.meta_key] + raw_caption = meta_dict.get(self.caption_key) + if raw_caption is not None: + caption = self._json_dict_or_none(raw_caption) + if caption is None: + raise ValueError(f"{self.caption_key} is not a JSON object") + else: + # Some sound midtrain shards use older full-video visual captions + # (caption_base/caption_structured/...) instead of the chunked + # ``caption`` field. Prefer those visual captions when present; + # otherwise fall back to nested audio-only chunks. + caption = self._parse_legacy_full_video_caption(meta_dict) + if caption is None: + caption = self._parse_audio_caption_chunks(meta_dict) + if caption is None: + raise KeyError(self.caption_key) + + # Contents of caption + # caption = { + # "chunk_0_300": { + # "caption": "...", + # "start_frame": 0, + # "end_frame": 300, + # }, + # "chunk_300_435": { + # "caption": "...", + # "start_frame": 300, + # "end_frame": 435, + # }, + # } + chunk_keys = list(caption.keys()) + if len(chunk_keys) == 0: + log.warning( + f"TextTransformForVideoJsonCaption: empty caption dict. url: {data_dict.get('__url__')}, key: {data_dict.get('__key__')}", + rank0_only=False, + ) + return None + + sampled_key = random.choice(chunk_keys) + sampled_chunk = caption[sampled_key] + + data_dict["chunk_index"] = chunk_keys.index(sampled_key) + data_dict["chunk_start_frame"] = int(sampled_chunk["start_frame"]) + data_dict["chunk_end_frame"] = int(sampled_chunk["end_frame"]) + + if "caption_json" in sampled_chunk: + caption_json = sampled_chunk["caption_json"] + else: + caption_json = json.loads(sampled_chunk["caption"]) + except Exception as e: + log.warning( + f"TextTransformForVideoJsonCaption dataloader error -- url: {data_dict.get('__url__')}, key: {data_dict.get('__key__')}\n error {e}", + rank0_only=False, + ) + return None + + # Randomly dropout json keys during training + if json_field_dropout_rate > 0: + for key in list(caption_json.keys()): + if random.random() < json_field_dropout_rate: + caption_json.pop(key) + + # Inject audio caption from metas as a new field when available. Added after the field + # dropout above so it is preserved whenever upstream metadata provides it. + audio_caption = self._find_audio_caption(meta_dict) + if isinstance(audio_caption, str) and len(audio_caption) > 0: + caption_json["audio_description"] = audio_caption + + data_dict["ai_caption"] = caption_json + + # Delete metas unless keep_metas=True (set when downstream augmentors still need them, + # e.g. VideoParsingChunkedFrames needs framerate/width/height/nb_frames). + if not self.keep_metas: + del data_dict[self.meta_key] + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/transfer_control_input/__init__.py b/cosmos_framework/data/vfm/augmentors/transfer_control_input/__init__.py new file mode 100644 index 0000000..99d3ee8 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/transfer_control_input/__init__.py @@ -0,0 +1,8 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Transfer control input augmentors (edge, blur, depth, seg) for cosmos3 VFM; copied from transfer2 to avoid cosmos dependency.""" + +from cosmos_framework.data.vfm.augmentors.transfer_control_input.control_input import AddControlInputComb + +__all__ = ["AddControlInputComb"] diff --git a/cosmos_framework/data/vfm/augmentors/transfer_control_input/blur.py b/cosmos_framework/data/vfm/augmentors/transfer_control_input/blur.py new file mode 100644 index 0000000..daacd30 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/transfer_control_input/blur.py @@ -0,0 +1,267 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import random +from typing import Callable, Optional + +import attrs +import cv2 +import numpy as np +import torch + +from cosmos_framework.data.vfm.augmentors.transfer_control_input.fast_blur import BilateralGaussian + + +@attrs.define +class BilateralFilterConfig: + """Configuration for Bilateral filter""" + + use_random: bool = False + # if use_random is False, then optionally define the param values + d: int = 30 + sigma_color: int = 150 + sigma_space: int = 100 + iter: int = 1 + + # if use_random is True, then optionally define the range + d_min: int = 15 + d_max: int = 50 + sigma_color_min: int = 100 + sigma_color_max: int = 300 + sigma_space_min: int = 50 + sigma_space_max: int = 150 + iter_min: int = 1 + iter_max: int = 2 + + # Whether to use GPU kernel (inference only) + use_cuda: bool = False + + +# Blur config default values are tuned for this resolution (longest side). +REFERENCE_RESOLUTION = 720 + + +def _scale_for_resolution(value: float, longest_side: int) -> float: + """Scale a blur parameter from REFERENCE_RESOLUTION to the given longest frame side.""" + if longest_side <= 0: + return value + return value * (longest_side / REFERENCE_RESOLUTION) + + +def _scale_ksize(ksize: int, longest_side: int) -> int: + """Scale kernel size for resolution; result is odd and >= 1.""" + scaled = max(1, int(round(_scale_for_resolution(float(ksize), longest_side)))) + return scaled + 1 if scaled % 2 == 0 else scaled + + +@attrs.define +class GaussianBlurConfig: + """Configuration for Gaussian blur""" + + use_random: bool = False + # if use_random is False, then optionally define the param values + ksize: int = 25 + sigmaX: float = 12.5 + + # if use_random is True, then optionally define the range + ksize_min: int = 21 + ksize_max: int = 29 + sigmaX_min: float = 10.5 + sigmaX_max: float = 14.5 + + +def apply_bilateral_filter( + frames: np.ndarray, + d: int = 9, + sigma_color: float = 75, + sigma_space: float = 75, + iter: int = 1, + bilateral_cuda_module: Optional[Callable] = None, +) -> np.ndarray: + if bilateral_cuda_module is not None: + blurred_image = [] + frames_tensor = torch.from_numpy(frames).cuda() + for _image in frames_tensor.permute(1, 2, 3, 0): + blurred_image.append(bilateral_cuda_module(_image, d // 3, (sigma_color // 2) ** 2, sigma_space**2)) + blurred_image = torch.stack(blurred_image).permute(3, 0, 1, 2) + return blurred_image.cpu().numpy() + + C, T, H, W = frames.shape + blurred_frames = np.empty_like(frames) + + for t in range(T): + frame = np.ascontiguousarray(frames[:, t].transpose(1, 2, 0)) + for _ in range(iter): + frame = cv2.bilateralFilter(frame, d, sigma_color, sigma_space) + if len(frame.shape) == 2: + frame = frame[..., None] + + blurred_frames[:, t] = frame.transpose(2, 0, 1) + + return blurred_frames + + +def _longest_frame_side(frames: np.ndarray) -> int: + """Return the longest spatial dimension (H or W) for CTHW frames.""" + # frames: (C, T, H, W) + return int(max(frames.shape[2], frames.shape[3])) + + +class BilateralFilter: + def __init__(self, config: BilateralFilterConfig) -> None: + self.use_random = config.use_random + self.config = config + assert not (self.use_random and self.config.use_cuda), "Cannot use GPU kernel for training." + self.bilateral_cuda_module = BilateralGaussian() if self.config.use_cuda else None + + def __call__(self, frames: np.ndarray) -> np.ndarray: + config = self.config + longest = _longest_frame_side(frames) + if self.use_random: + d = np.random.randint(config.d_min, config.d_max) + sigma_color = np.random.randint(config.sigma_color_min, config.sigma_color_max) + sigma_space = np.random.randint(config.sigma_space_min, config.sigma_space_max) + iter = np.random.randint(config.iter_min, config.iter_max) + else: + d = config.d + sigma_color = config.sigma_color + sigma_space = config.sigma_space + iter = config.iter + # Scale from reference resolution (720) to current frame size + d = max(1, int(round(_scale_for_resolution(float(d), longest)))) + d = d + 1 if d % 2 == 0 else d # cv2.bilateralFilter requires odd d + sigma_color = max(1.0, _scale_for_resolution(float(sigma_color), longest)) + sigma_space = max(1.0, _scale_for_resolution(float(sigma_space), longest)) + return apply_bilateral_filter(frames, d, sigma_color, sigma_space, iter, self.bilateral_cuda_module) + + +def apply_gaussian_blur(frames: np.ndarray, ksize: int = 5, sigmaX: float = 1.0) -> np.ndarray: + if ksize % 2 == 0: + ksize += 1 # ksize must be odd + + _, T, _, _ = frames.shape + blurred_frames = np.empty_like(frames) + + for t in range(T): + frame = np.ascontiguousarray(frames[:, t].transpose(1, 2, 0)) + frame = cv2.GaussianBlur(frame, (ksize, ksize), sigmaX=sigmaX) + if len(frame.shape) == 2: + frame = frame[..., None] + blurred_frames[:, t] = frame.transpose(2, 0, 1) + + return blurred_frames + + +class GaussianBlur: + def __init__(self, config: GaussianBlurConfig) -> None: + self.use_random = config.use_random + self.config = config + + def __call__(self, frames: np.ndarray) -> np.ndarray: + longest = _longest_frame_side(frames) + if self.use_random: + ksize = np.random.randint(self.config.ksize_min, self.config.ksize_max + 1) + sigmaX = np.random.uniform(self.config.sigmaX_min, self.config.sigmaX_max) + else: + ksize = self.config.ksize + sigmaX = self.config.sigmaX + ksize = _scale_ksize(int(ksize), longest) + sigmaX = max(0.1, _scale_for_resolution(float(sigmaX), longest)) + return apply_gaussian_blur(frames, ksize, sigmaX) + + +@attrs.define +class BlurCombinationConfig: + """Configuration for a combination of blurs with associated probability""" + + # list of choices are: ["gaussian", "bilateral"] + # the corresponding config must be defined for each item in this blur_types list + blur_types: list[str] + probability: float + gaussian_blur: GaussianBlurConfig | None = None + bilateral_filter: BilateralFilterConfig | None = None + + +@attrs.define +class BlurConfig: + """Configuration for blur augmentation with multiple combinations""" + + # probabilities from the list of combinations should add up to 1.0 + blur_combinations: list[BlurCombinationConfig] = [] + + +# For training +random_blur_config = BlurConfig( + blur_combinations=[ + BlurCombinationConfig( + blur_types=["bilateral"], + probability=0.3, + bilateral_filter=BilateralFilterConfig(use_random=True), + ), + BlurCombinationConfig( + blur_types=["gaussian"], + probability=0.5, + gaussian_blur=GaussianBlurConfig(use_random=True), + ), + BlurCombinationConfig( + blur_types=["bilateral", "gaussian"], + probability=0.2, + bilateral_filter=BilateralFilterConfig(use_random=True), + gaussian_blur=GaussianBlurConfig(use_random=True), + ), + ], +) + +# For inference +bilateral_blur_config = BlurConfig( + blur_combinations=[ + BlurCombinationConfig( + blur_types=["bilateral"], + probability=1.0, + bilateral_filter=BilateralFilterConfig(use_random=False), + ), + ], +) + + +class Blur: + def __init__(self, config: BlurConfig | None = None, use_random: bool = True) -> None: + if config is None: + config = random_blur_config if use_random else bilateral_blur_config + probabilities = [combo.probability for combo in config.blur_combinations] + total_prob = sum(probabilities) + assert abs(total_prob - 1.0) < 1e-6, f"Probabilities must sum to 1.0, got {total_prob}" + + self.blur_combinations = config.blur_combinations + self.probabilities = probabilities + self._set_blur_instances() + + def _set_blur_instances(self) -> None: + if not self.blur_combinations: + return + self.blur_combinations_instances = [] + + for blur_combination in self.blur_combinations: + blur_mapping = { + "gaussian": (GaussianBlur, blur_combination.gaussian_blur), + "bilateral": (BilateralFilter, blur_combination.bilateral_filter), + } + + cur_instances = [] + for blur_type in blur_combination.blur_types: + assert blur_type in blur_mapping, f"Unknown {blur_type}. Needs to correct blur_type or blur_mapping." + + blur_class, blur_config = blur_mapping[blur_type] + cur_instances.append(blur_class(blur_config)) + + self.blur_combinations_instances.append(cur_instances) + + assert len(self.blur_combinations_instances) == len(self.blur_combinations), ( + "Number of blur_combinations_instances needs to match number of blur_combinations." + ) + + def __call__(self, frames: np.ndarray) -> np.ndarray: + blur_instances = random.choices(self.blur_combinations_instances, weights=self.probabilities, k=1)[0] + for ins in blur_instances: + frames = ins(frames) + return frames diff --git a/cosmos_framework/data/vfm/augmentors/transfer_control_input/control_input.py b/cosmos_framework/data/vfm/augmentors/transfer_control_input/control_input.py new file mode 100644 index 0000000..3b46a82 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/transfer_control_input/control_input.py @@ -0,0 +1,660 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import random +from typing import Optional, Union + +import cv2 +import numpy as np +import torch +import torchvision.transforms.functional as transforms_F +from pycocotools import mask as mask_utils + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.augmentors.transfer_control_input.blur import Blur, BlurConfig +from cosmos_framework.data.vfm.augmentors.transfer_control_input.seg import ( + decode_partial_rle_width1, + segmentation_color_mask, +) + +# Constants for segmentation color processing +# These parameters control the color-based mask extraction process in AddControlInputSeg + +# Color quantization bin size for grouping similar colors together +# Range: 1-100 (smaller values = more granular color detection, larger values = more color grouping) +# Typical range: 10-50, where 25 provides good balance between precision and grouping +_BIN_SIZE = 25 + +# Maximum number of unique colors to examine for mask generation (to limit computation time) +# Range: 10-500 (smaller values = faster processing, larger values = more thorough color search) +# Typical range: 50-200, where 100 balances thoroughness with performance +_MAX_UNIQUE_COLORS = 100 + +# Color distance tolerance for considering pixels as the same color +# Range: 1-100 (smaller values = stricter color matching, larger values = more lenient matching) +# Typical range: 10-60, where 30 provides good tolerance for natural color variations +_COLOR_TOLERANCE = 30 + +# RGB value threshold below which a color is considered "black" and filtered out +# Range: 0-100 (smaller values = stricter black detection, larger values = more colors considered black) +# Typical range: 20-80, where 50 effectively filters out dark/black regions +_BLACK_THRESHOLD = 50 + + +def _maybe_torch_to_numpy(frames: Union[torch.Tensor, list]) -> np.ndarray: + try: + return frames.numpy() + except AttributeError: + return np.array(frames) + + +class AddControlInputEdge(Augmentor): + """ + Add control input to the data dictionary. control input are expanded to 3-channels + steps to add new items: modify this file, configs/conditioner.py, conditioner.py + """ + + def __init__( + self, + input_keys: list, + output_keys: Optional[list] = ["control_input_edge"], + args: Optional[dict] = None, + use_random: Optional[bool] = True, + preset_strength: Optional[str] = "medium", + edge_t_lower: Optional[int] = None, + edge_t_upper: Optional[int] = None, + **kwargs, + ) -> None: + super().__init__(input_keys, output_keys, args) + self.use_random = use_random + self.preset_strength = preset_strength + self.t_lower = edge_t_lower + self.t_upper = edge_t_upper + + def __call__(self, data_dict: dict) -> dict: + if "control_input_edge" in data_dict: + return data_dict + key_img = self.input_keys[0] + key_out = self.output_keys[0] + frames = data_dict[key_img] + # log.info(f"Adding control input edge. Input key: {key_img}, Output key: {key_out}. Use random: {self.use_random}, Preset strength: {self.preset_strength}") + # Get lower and upper threshold for canny edge detection. + if self.use_random: # always on for training, always off for inference + if self.t_lower is not None and self.t_upper is not None: + # Use provided t_lower and t_upper values + t_lower = self.t_lower + t_upper = self.t_upper + else: + # Generate random values as before + t_lower = np.random.randint(20, 100) # Get a random lower thre + t_diff = np.random.randint(50, 150) # Get a random diff between lower and upper + t_upper = t_lower + t_diff # The upper thre is lower added by the diff + else: + if self.preset_strength == "none" or self.preset_strength == "very_low": + t_lower, t_upper = 20, 50 + elif self.preset_strength == "low": + t_lower, t_upper = 50, 100 + elif self.preset_strength == "medium": + t_lower, t_upper = 100, 200 + elif self.preset_strength == "high": + t_lower, t_upper = 200, 300 + elif self.preset_strength == "very_high": + t_lower, t_upper = 300, 400 + else: + raise ValueError(f"Preset {self.preset_strength} not recognized.") + + frames = _maybe_torch_to_numpy(frames).astype(np.uint8) + is_image = len(frames.shape) < 4 + + # Compute the canny edge map by the two thresholds. + if is_image: + edge_maps = cv2.Canny(frames, t_lower, t_upper)[None, None] + else: + edge_maps = [ + cv2.Canny(img, t_lower, t_upper) for img in frames.transpose((1, 2, 3, 0)) + ] # (C, T, H, W) -> (T, H, W) + edge_maps = np.stack(edge_maps)[None] + edge_maps = torch.from_numpy(edge_maps).expand(3, -1, -1, -1) # [3,T,H,W] + if is_image: + edge_maps = edge_maps[:, 0] + data_dict[key_out] = edge_maps + return data_dict + + +class AddControlInputBlur(Augmentor): + """ + Main class for adding blurred input to the data dictionary. + self.output_keys[0] indicates the types of blur added to the input. + For example, control_input_gaussian_guided indicates that both Gaussian and Guided filters are applied + """ + + def __init__( + self, + input_keys: list, # [key_load, key_img] + output_keys: Optional[list] = ["control_input_blur"], + args: Optional[dict] = None, # not used + use_random: bool = True, # whether to use random parameters + blur_config: BlurConfig | None = None, + downup_preset: str | int = "medium", # preset strength for downup factor + min_downup_factor: int = 4, # minimum downup factor + max_downup_factor: int = 16, # maximum downup factor + downsize_before_blur: bool = True, # whether to downsize before applying blur and then upsize or downup after blur + blur_downsize_factor: list[int] = list(range(1, 5)), # downscale factor for blur + resize_cuda: bool = False, # whether to do resizing on GPU, the result is still moved back to CPU for compatibility. + **kwargs, + ) -> None: + super().__init__(input_keys, output_keys, args) + self.use_random = use_random + downup_preset_values = { + "none": 1, + "very_low": min_downup_factor, + "low": min_downup_factor, + "medium": (min_downup_factor + max_downup_factor) // 2, + "high": max_downup_factor, + "very_high": max_downup_factor, + } + blur_downup_preset_values = { + "none": 1, + "very_low": 1, + "low": 4, + "medium": 2, + "high": 1, + "very_high": 4, + } + self.blur = Blur(config=blur_config, use_random=use_random) + + self.preset_strength = downup_preset + self.downup_preset = downup_preset if isinstance(downup_preset, int) else downup_preset_values[downup_preset] + self.downsize_before_blur = downsize_before_blur + self.min_downup_factor = min_downup_factor + self.max_downup_factor = max_downup_factor + self.blur_downsize_factor = blur_downsize_factor + self.blur_downup_preset = blur_downup_preset_values[downup_preset] + self.resize_cuda = resize_cuda + assert not (self.use_random and self.resize_cuda), "Cannot use resize on GPU during training." + + def _load_frame(self, data_dict: dict) -> tuple[np.ndarray, bool]: + key_img = self.input_keys[0] + frames = data_dict[key_img] + frames = _maybe_torch_to_numpy(frames) + is_image = False + if len(frames.shape) < 4: + frames = frames.transpose((2, 0, 1))[:, None] + is_image = True + return frames, is_image + + def __call__(self, data_dict: dict) -> dict: + if "control_input_blur" in data_dict: + # already processed + data_dict[self.output_keys[0]] = data_dict["control_input_blur"] + return data_dict + + key_out = self.output_keys[0] + + frames, is_image = self._load_frame(data_dict) + if self.preset_strength == "none": + if is_image: + frames = frames[:, 0] + data_dict[key_out] = torch.from_numpy(frames) + return data_dict + C, T, H, W = frames.shape + + # --- 1. Downscale Before Blur --- + if self.use_random: + downscale_factor = random.choice(self.blur_downsize_factor) + else: + downscale_factor = self.blur_downup_preset + + if self.downsize_before_blur: + new_W, new_H = W // downscale_factor, H // downscale_factor + downscaled = np.empty((C, T, new_H, new_W), dtype=frames.dtype) + + for t in range(T): + frame = np.ascontiguousarray(frames[:, t].transpose(1, 2, 0)) + resized = cv2.resize(frame, (new_W, new_H), interpolation=cv2.INTER_AREA) + if len(resized.shape) == 2: + resized = resized[..., None] + downscaled[:, t] = resized.transpose(2, 0, 1) + + frames = downscaled + # -------------------------- + + # --- 2. Apply Blur --- + frames = self.blur(frames) + # -------------------------- + + # --- 3. Upscale After Blur --- + if self.downsize_before_blur: + upscaled = np.empty((C, T, H, W), dtype=frames.dtype) + for t in range(T): + frame = np.ascontiguousarray(frames[:, t].transpose(1, 2, 0)) + resized = cv2.resize(frame, (W, H), interpolation=cv2.INTER_LINEAR) + if len(resized.shape) == 2: + resized = resized[..., None] + upscaled[:, t] = resized.transpose(2, 0, 1) + frames = upscaled + + # --- 4. Final Downup Augmentation --- + if self.use_random: + scale_factor = random.randint(self.min_downup_factor, self.max_downup_factor + 1) + else: + scale_factor = self.downup_preset + + final_W, final_H = int(W / scale_factor), int(H / scale_factor) + + final_frames = np.empty_like(frames) + for t in range(T): + frame = np.ascontiguousarray(frames[:, t].transpose(1, 2, 0)) + small = cv2.resize(frame, (final_W, final_H), interpolation=cv2.INTER_CUBIC) + large = cv2.resize(small, (W, H), interpolation=cv2.INTER_CUBIC) + if len(large.shape) == 2: + large = large[..., None] + final_frames[:, t] = large.transpose(2, 0, 1) + + if is_image: + final_frames = final_frames[:, 0] + data_dict[key_out] = torch.from_numpy(final_frames) + + return data_dict + + +class AddControlInputDepth(Augmentor): + """ + Add control input to the data dictionary. control input are expanded to 3-channels + steps to add new items: modify this file, configs/conditioner.py, conditioner.py + """ + + def __init__( + self, + input_keys: list, + output_keys: Optional[list] = ["control_input_depth"], + args: Optional[dict] = None, + **kwargs, + ) -> None: + super().__init__(input_keys, output_keys, args) + + def __call__(self, data_dict: dict) -> dict: + if "control_input_depth" in data_dict: + # already processed + return data_dict + + key_out = self.output_keys[0] + depth = data_dict["depth"] + + frames = data_dict["video"] + _, T, H, W = frames.shape + depth = transforms_F.resize( + depth, + size=(H, W), + interpolation=transforms_F.InterpolationMode.BILINEAR, + ) + data_dict[key_out] = depth + return data_dict + + +class AddControlInputSeg(Augmentor): + """ + Add control input to the data dictionary. control input are expanded to 3-channels + steps to add new items: modify this file, configs/conditioner.py, conditioner.py + """ + + def __init__( + self, + input_keys: list, + output_keys: Optional[list] = ["control_input_seg"], + thres_mb_python_decode: Optional[int] = 256, # required: <= 512 for 7b + use_fixed_color_list: bool = False, + num_masks_max: int = 100, + random_sample_num_masks: bool = True, + min_mask_size: float = 0.2, + args: Optional[dict] = None, + **kwargs, + ) -> None: + """ + Args: + thres_mb_python_decode: int, threshold of memory usage for python decode, in MB + use_fixed_color_list: bool, if True, use predefined colors for segmentation masks. If False, generate random colors for segmentation masks. + num_masks_max: int, maximum number of masks to sample + random_sample_num_masks: bool, if True, sample number of masks randomly. If False, sample all masks in the data. + min_mask_size: float, minimum size of the mask area, in fraction of the entire frame. + """ + super().__init__(input_keys, output_keys, args) + self.use_fixed_color_list = use_fixed_color_list + self.num_masks_max = num_masks_max + self.thres_mb_python_decode = thres_mb_python_decode + self.random_sample_num_masks = random_sample_num_masks + self.min_mask_size = min_mask_size + + def get_masks(self, data_dict: dict, num_masks: int = 1) -> tuple[torch.Tensor, bool]: + """ + Get a single mask from the data dictionary. + segmentation: list of dicts + phrase: str + segmentation_mask_rle: dict + data: dict + size: [N, 1] + counts: bytes + mask_shape: [T, H, W] + """ + frames = data_dict["video"] + _, T, H, W = frames.shape + + if not isinstance(data_dict["segmentation"], dict) and num_masks == 1: + # this video is a color-coded segmentation mask, where each color corresponds to a different object + # we need to extract the binary mask for a single object from the video + seg_video = data_dict["segmentation"] + seg_video = transforms_F.resize( + seg_video, + size=(H, W), + interpolation=transforms_F.InterpolationMode.NEAREST, + ) + # Get the first frame of the segmentation video + first_frame = seg_video[:, 0, :, :] + # Get a list of unique colors from the first frame and calculate mask size for each unique color + unique_colors = (first_frame // _BIN_SIZE).view(3, -1).permute(1, 0).unique( + dim=0 + ) * _BIN_SIZE # [N_colors,3] + # Randomly shuffle unique colors and take first N colors + perm = torch.randperm(len(unique_colors)) + unique_colors = unique_colors[perm] + unique_colors = unique_colors[:_MAX_UNIQUE_COLORS] # check up to max colors to save time + mask_sizes = [] + for color in unique_colors: + color_diff = first_frame.to(torch.float32) - color[:, None, None] + color_dists = torch.sqrt(torch.sum(color_diff**2, dim=0)) + mask = color_dists < _COLOR_TOLERANCE + mask_size = mask.sum() / (H * W) # Size as fraction of frame + mask_sizes.append(mask_size) + + # Only keep colors that produce masks >= min_mask_size of frame and not black + valid_color_indices = [ + i + for i, size in enumerate(mask_sizes) + if size >= self.min_mask_size and (unique_colors[i] > _BLACK_THRESHOLD).sum() > 0 + ] + if len(valid_color_indices) == 0: + # If no masks are large enough, return all ones + log.critical("No masks are large enough, returning all ones") + all_masks = np.ones((num_masks, T, H, W)).astype(bool) + return torch.from_numpy(all_masks), False # [num_masks,T,H,W] + else: + # Randomly select one of the valid large masks + valid_color_idx = valid_color_indices[np.random.randint(len(valid_color_indices))] + target_color = unique_colors[valid_color_idx] + # Create binary mask where True means within tolerance of target color + color_diff = seg_video.to(torch.float32) - target_color[:, None, None, None] + color_dists = torch.sqrt(torch.sum(color_diff**2, dim=0, keepdim=True)) # [1,T,H,W] + mask = (color_dists < _COLOR_TOLERANCE).to(torch.bool) # [1,T,H,W] + return mask, True + frame_indices = data_dict["frame_indices"] + frame_start, frame_end = frame_indices[0], frame_indices[-1] + 1 + is_continuous_frame_indices = (frame_end - frame_start) == T + assert len(frame_indices) == T, ( + f"frame_indices length {len(frame_indices)} != T {T}, likely due to video decoder using different fps, i.e. sample with stride. Need to return frame indices from video decoder." + ) + + all_masks = np.ones((num_masks, T, H, W)).astype(bool) # [num_masks,T,H,W] + + # sample number of masks + mask_ids = np.arange(len(data_dict["segmentation"])).tolist() + if len(data_dict["segmentation"]) == 0 or num_masks == 0: + return torch.from_numpy(all_masks), False # [num_masks,T,H,W] + if num_masks == 1: # Try up to 16 masks to find a large enough mask + mask_ids_select = np.random.choice(mask_ids, min(len(mask_ids), 16), replace=False) + else: + mask_ids_select = np.random.choice(mask_ids, num_masks, replace=False) + + for idx, mid in enumerate(mask_ids_select): + mask = data_dict["segmentation"][mid] + if type(mask) != dict: # data has sharding issue, skip this mask + return torch.from_numpy(all_masks), False # [num_masks,T,H,W] + shape = mask["segmentation_mask_rle"]["mask_shape"] + num_byte_per_mb = 1024 * 1024 + # total number of elements in uint8 (1 byte) / num_byte_per_mb + if mask["segmentation_mask_rle"]["data"]["size"][0] / num_byte_per_mb > self.thres_mb_python_decode: + # Switch to python decode if the mask is too large to avoid out of shared memory + if is_continuous_frame_indices and ( + T * shape[1] * shape[2] / num_byte_per_mb <= self.thres_mb_python_decode + ): + log.critical( + f"Using python decode for mask of shape {shape}, Continuous frame indices, frame_start: {frame_start}, frame_end: {frame_end}" + ) + rle = decode_partial_rle_width1( + mask["segmentation_mask_rle"]["data"], + frame_start * shape[1] * shape[2], + frame_end * shape[1] * shape[2], + ) + partial_shape = (frame_end - frame_start, shape[1], shape[2]) + rle = rle.reshape(partial_shape) * 255 + rle = np.stack( + [cv2.resize(_image_np, (W, H), interpolation=cv2.INTER_NEAREST) for _image_np in rle] + ) + else: # need to call decode_partial_rle_width1 multiple times + # It takes too much time to decode the mask, so we skip it and select another modality instead + log.critical(f"Skipping python decode for mask of shape {shape}") + return torch.from_numpy(all_masks), False # [num_masks,T,H,W] + else: + rle = mask_utils.decode(mask["segmentation_mask_rle"]["data"]) + rle = rle.reshape(shape) * 255 + # Select the frames that are in the video + if len(rle) < frame_end: # Pad the mask if it is shorter than original video + rle = np.vstack([rle, [rle[-1]] * (frame_end - len(rle))]) + rle = np.stack([cv2.resize(rle[i], (W, H), interpolation=cv2.INTER_NEAREST) for i in frame_indices]) + if num_masks == 1: # if we only need one mask and the current mask is large enough, return it + if (rle > 0).sum() / rle.size >= self.min_mask_size: + # log.critical(f"Found a large enough mask with size {(rle > 0).sum() / rle.size}") + all_masks[0] = rle.astype(bool) + break + elif idx == len(mask_ids_select) - 1: + log.critical("No large enough mask found, returning all ones") + else: # if we need multiple masks, return all masks + all_masks[idx] = rle.astype(bool) + del rle + return torch.from_numpy(all_masks), True # [num_masks,T,H,W] + + def __call__(self, data_dict: dict) -> dict: + if "control_input_seg" in data_dict: + # already processed + return data_dict + + key_out = self.output_keys[0] + if not isinstance(data_dict["segmentation"], dict): + # already have a color-coded segmentation mask video, directly use it + seg = data_dict["segmentation"] + seg = transforms_F.resize( + seg, + size=data_dict["video"].shape[-2:], + interpolation=transforms_F.InterpolationMode.NEAREST, + ) + data_dict[key_out] = seg + return data_dict + + # sample number of masks + if self.random_sample_num_masks: + num_masks = np.random.randint(0, min(self.num_masks_max + 1, len(data_dict["segmentation"]) + 1)) + else: + num_masks = len(data_dict["segmentation"]) + + all_masks, success = self.get_masks(data_dict, num_masks) + + if not success: + data_dict["preprocess_failed"] = True + del all_masks # free memory + return data_dict + + key_out = self.output_keys[0] + # control_input_seg is the colored segmentation mask, value in [0,255], shape (3, T, H, W) + data_dict[key_out] = torch.from_numpy( + segmentation_color_mask(all_masks, self.use_fixed_color_list) + ) # [3,T,H,W] + if num_masks > 0: + data_dict[key_out + "_mask"] = all_masks[random.randint(0, num_masks - 1)].clone()[None] # [1,T,H,W] + del all_masks # free memory + return data_dict + + +class AddControlInputIdentity(Augmentor): + def __init__( + self, + input_keys: list, + output_keys: Optional[list] = ["control_input_identity"], + args: Optional[dict] = None, + **kwargs, + ) -> None: + super().__init__(input_keys, output_keys, args) + + def __call__(self, data_dict: dict) -> dict: + key_img = self.input_keys[0] + key_out = self.output_keys[0] + frames = _maybe_torch_to_numpy(data_dict[key_img]) # CTHW for video, HWC for image + is_image = len(frames.shape) < 4 + if is_image: + frames = frames.transpose((2, 0, 1)) + data_dict[key_out] = torch.from_numpy(frames).clone() # [C,T,H,W] for video, [C,H,W] for image + return data_dict + + +class AddControlInputHdmapBbox(Augmentor): + """ + Add control input to the data dictionary. control input are expanded to 3-channels + steps to add new items: modify this file, configs/conditioner.py, conditioner.py + """ + + def __init__( + self, + input_keys: list, + output_keys: Optional[list] = ["control_input_hdmap_bbox"], + args: Optional[dict] = None, + use_random: Optional[bool] = True, + preset_strength: Optional[str] = "medium", + **kwargs, + ) -> None: + super().__init__(input_keys, output_keys, args) + self.use_random = use_random + self.preset_strength = preset_strength + + def __call__(self, data_dict: dict) -> dict: + if "control_input_hdmap_bbox" in data_dict: + return data_dict + key_input = self.input_keys[0] + key_out = self.output_keys[0] + data_dict[key_out] = data_dict[key_input] + return data_dict + + +CTRL_HINT_KEYS = { + "control_input_edge": AddControlInputEdge, + "control_input_blur": AddControlInputBlur, + "control_input_depth": AddControlInputDepth, + "control_input_seg": AddControlInputSeg, + "control_input_inpaint": AddControlInputIdentity, + "control_input_hdmap_bbox": AddControlInputHdmapBbox, +} + + +class AddControlInputComb(Augmentor): + """ + Add control input to the data dictionary. control input are expanded to 3-channels + steps to add new items: modify this file, configs/conditioner.py, conditioner.py + """ + + def __init__( + self, + input_keys: list, + output_keys: Optional[list] = None, + args: Optional[dict] = None, + use_random: bool = True, + control_input_type: str = "edge_blur_depth_seg", + use_control_mask_prob: float = 0.0, + num_control_inputs_prob: list[float] = [1.0, 0.0, 0.0, 0.0], + num_control_inputs: int | None = None, + **kwargs, + ) -> None: + super().__init__(input_keys, output_keys, args) + self.use_random = use_random + self.control_hint_keys = ["control_input_" + key for key in control_input_type.split("_")] + self.use_control_mask_prob = use_control_mask_prob + self.num_control_inputs_prob = num_control_inputs_prob[: len(self.control_hint_keys)] + self.num_control_inputs = num_control_inputs + self.comb = {} + for output_key, class_name in CTRL_HINT_KEYS.items(): + aug = class_name( + input_keys=input_keys, output_keys=[output_key], args=args, use_random=use_random, **kwargs + ) + self.comb[output_key] = aug + + def __call__(self, data_dict: dict) -> dict: + success = False + if self.use_random: + # Randomly select a number of control inputs (or use fixed num_control_inputs when set) + num_keys_prob = self.num_control_inputs_prob + ctrl_hint_keys = self.control_hint_keys + if self.num_control_inputs is not None: + num_keys = max(1, min(self.num_control_inputs, len(ctrl_hint_keys))) + else: + num_keys = random.choices(range(len(ctrl_hint_keys)), weights=num_keys_prob, k=1)[0] + 1 + output_keys = np.random.choice(ctrl_hint_keys, size=num_keys, replace=False) + # output_keys = np.random.choice(["control_input_edge", "control_input_blur", "control_input_depth"], size=num_keys, replace=False) + zero_input = torch.zeros_like(data_dict[self.input_keys[0]]) # [C,T,H,W] + zero_mask = torch.zeros(*data_dict[self.input_keys[0]][:1].shape, dtype=torch.bool) # [1,T,H,W] + ones_mask = torch.ones(*data_dict[self.input_keys[0]][:1].shape, dtype=torch.bool) # [1,T,H,W] + use_control_mask = random.random() < self.use_control_mask_prob + for cur_key in ctrl_hint_keys: + cur_mask_key = cur_key + "_mask" + if cur_key in output_keys: + data_dict["preprocess_failed"] = False + data_dict = self.comb[cur_key](data_dict) + # log.critical(f"self.use_control_mask_prob: {self.use_control_mask_prob}") + if use_control_mask or cur_key == "control_input_inpaint": + # Get mask for the control input + if cur_mask_key not in data_dict: + data_dict[cur_mask_key], success = self.comb["control_input_seg"].get_masks( + data_dict, num_masks=1 + ) + else: + data_dict[cur_mask_key] = ones_mask + + # If preprocess failed or cannot get inpaint mask, use control_input_edge instead + if data_dict["preprocess_failed"] or (cur_key == "control_input_inpaint" and not success): + data_dict[cur_key] = zero_input + data_dict[cur_mask_key] = zero_mask + if num_keys == 1 and "control_input_edge" in ctrl_hint_keys: + new_key = "control_input_edge" + log.critical(f"Preprocess failed for {cur_key}, using {new_key} instead") + if new_key in data_dict: + del data_dict[new_key] + data_dict = self.comb[new_key](data_dict) + data_dict[new_key + "_mask"] = ones_mask + else: + data_dict[cur_key] = zero_input + data_dict[cur_mask_key] = zero_mask + + # Free memory: remove unused depth/segmentation to avoid OOM later + if "control_input_depth" not in output_keys and "depth" in data_dict: + del data_dict["depth"] + if "control_input_seg" not in output_keys and "segmentation" in data_dict: + del data_dict["segmentation"] + if "segmentation" in data_dict and isinstance(data_dict["segmentation"], dict): + del data_dict["segmentation"] + + if "control_input_inpaint" in output_keys and success: # Post-process the inpaint mask + inpaint_mask_key = "control_input_inpaint_mask" + if random.random() < 0.5: # randomly negate the mask + data_dict[inpaint_mask_key] = ~data_dict[inpaint_mask_key] + # Make sure the inpaint mask does not overlap with other masks + for cur_key in ctrl_hint_keys: + cur_mask_key = cur_key + "_mask" + if cur_mask_key == inpaint_mask_key: + continue + if torch.all(data_dict[cur_mask_key]) or torch.all(~data_dict[cur_mask_key]): # dummy mask + continue + # Remove overlap by zeroing overlapping regions in mask1 + overlap = data_dict[cur_mask_key] & data_dict[inpaint_mask_key] + if overlap.any(): + data_dict[inpaint_mask_key] = data_dict[inpaint_mask_key] & ~overlap + + else: + for k, v in self.comb.items(): + data_dict = v(data_dict) + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/transfer_control_input/fast_blur.py b/cosmos_framework/data/vfm/augmentors/transfer_control_input/fast_blur.py new file mode 100644 index 0000000..202cf93 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/transfer_control_input/fast_blur.py @@ -0,0 +1,92 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import ctypes +from ctypes import POINTER, c_float, c_int, c_ubyte, sizeof + +import torch + + +class BilateralGaussian: + class NppiSize(ctypes.Structure): + _fields_ = [("width", c_int), ("height", c_int)] + + class NppiPoint(ctypes.Structure): + _fields_ = [("x", c_int), ("y", c_int)] + + def __init__(self): + self.npp_i_lib = self._load_npp_library() + self._setup_buffer_size_function() + self._setup_bilateral_function() + + def _load_npp_library(self): + return ctypes.CDLL("libnppif.so") + + def _setup_buffer_size_function(self): + self.get_buffer_size_func = self.npp_i_lib.nppiFilterCannyBorderGetBufferSize + self.get_buffer_size_func.restype = c_int + self.get_buffer_size_func.argtypes = [BilateralGaussian.NppiSize, POINTER(c_int)] # oSizeROI # bufferSize + + def _setup_bilateral_function(self): + self.bilateral_function = self.npp_i_lib.nppiFilterBilateralGaussBorder_8u_C3R + self.bilateral_function.restype = c_int + self.bilateral_function.argtypes = [ + POINTER(c_ubyte), # pSrc + c_int, # nSrcStep + BilateralGaussian.NppiSize, # oSrcSize + BilateralGaussian.NppiPoint, # oSrcOffset + POINTER(c_ubyte), # pDst + c_int, # nDstStep + BilateralGaussian.NppiSize, # oSizeROI + c_int, # nRadius + c_int, # nStepBetweenSrcPixels + c_float, # nValSquareSigma + c_float, # nPosSquareSigma + c_int, # eBorderType + ] + + def _prepare_input(self, image_tensor): + if not image_tensor.is_cuda: + image_tensor = image_tensor.cuda() + if image_tensor.dtype != torch.uint8: + image_tensor = (image_tensor * 255).byte() + return image_tensor + + def _get_buffer_size(self, roi): + buffer_size = c_int(0) + status = self.get_buffer_size_func(roi, ctypes.byref(buffer_size)) + if status != 0: + raise RuntimeError(f"Failed to get buffer size, status: {status}") + return buffer_size.value + + def __call__(self, image_tensor, radius=30, color_sigma_square=150 * 150, sigma_space_square=100 * 100): + # Prepare input + image_tensor = self._prepare_input(image_tensor) + + height, width, channels = image_tensor.shape + output = torch.empty_like(image_tensor) + + src_ptr = ctypes.cast(image_tensor.data_ptr(), POINTER(c_ubyte)) + dst_ptr = ctypes.cast(output.data_ptr(), POINTER(c_ubyte)) + + roi = BilateralGaussian.NppiSize(width, height) + + status = self.bilateral_function( + src_ptr, + width * channels * sizeof(c_ubyte), + BilateralGaussian.NppiSize(width, height), + BilateralGaussian.NppiPoint(0, 0), + dst_ptr, + width * channels * sizeof(c_ubyte), + roi, + c_int(radius), + 1, # step size + c_float(color_sigma_square), + c_float(sigma_space_square), + 2, # border replicate + ) + + if status != 0: + raise RuntimeError(f"NPP Canny edge detection failed with status {status}") + + return output diff --git a/cosmos_framework/data/vfm/augmentors/transfer_control_input/seg.py b/cosmos_framework/data/vfm/augmentors/transfer_control_input/seg.py new file mode 100644 index 0000000..d0ff9c4 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/transfer_control_input/seg.py @@ -0,0 +1,171 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import random + +import matplotlib.colors as mcolors +import numpy as np + +# Array of 23 highly distinguishable colors in RGB format +PREDEFINED_COLORS_SEGMENTATION = np.array( + [ + [255, 0, 0], # Red + [0, 255, 0], # Green + [0, 0, 255], # Blue + [255, 255, 0], # Yellow + [0, 255, 255], # Cyan + [255, 0, 255], # Magenta + [255, 140, 0], # Dark Orange + [255, 105, 180], # Hot Pink + [0, 0, 139], # Dark Blue + [0, 128, 128], # Teal + [75, 0, 130], # Indigo + [128, 0, 128], # Purple + [255, 69, 0], # Red-Orange + [34, 139, 34], # Forest Green + [128, 128, 0], # Olive + [70, 130, 180], # Steel Blue + [255, 215, 0], # Gold + [255, 222, 173], # Navajo White + [144, 238, 144], # Light Green + [255, 99, 71], # Tomato + [221, 160, 221], # Plum + [0, 255, 127], # Spring Green + [255, 255, 255], # White + ] +) + + +def generate_distinct_colors() -> np.ndarray: + """ + Generate `n` visually distinguishable and randomized colors. + + Returns: + np.ndarray, (3) + """ + # Randomize hue, saturation, and lightness within a range + hue = random.uniform(0, 1) # Full spectrum of hues + saturation = random.uniform(0.1, 1) # Vibrant colors + lightness = random.uniform(0.2, 1.0) # Avoid too dark + + r, g, b = mcolors.hsv_to_rgb((hue, saturation, lightness)) + return (np.array([r, g, b]) * 255).astype(np.uint8) + + +def segmentation_color_mask(segmentation_mask: np.ndarray, use_fixed_color_list: bool = False) -> np.ndarray: + """ + Convert segmentation mask to color mask + Args: + segmentation_mask: np.ndarray, shape (num_masks, T, H, W) + Returns: + np.ndarray, shape (3, T, H, W), with each mask converted to a color mask, value [0,255] + """ + + num_masks, T, H, W = segmentation_mask.shape + segmentation_mask_sorted = [segmentation_mask[i] for i in range(num_masks)] + # Sort the segmentation mask by the number of non-zero pixels, from most to least + segmentation_mask_sorted = sorted(segmentation_mask_sorted, key=lambda x: np.count_nonzero(x), reverse=True) + + output = np.zeros((3, T, H, W), dtype=np.uint8) + if use_fixed_color_list: + predefined_colors_permuted = PREDEFINED_COLORS_SEGMENTATION[ + np.random.permutation(len(PREDEFINED_COLORS_SEGMENTATION)) + ] + else: + predefined_colors_permuted = [generate_distinct_colors() for _ in range(num_masks)] + # index the segmentation mask from last channel to first channel, i start from num_masks-1 to 0 + for i in range(num_masks): + mask = segmentation_mask_sorted[i] + color = predefined_colors_permuted[i % len(predefined_colors_permuted)] + + # Create boolean mask and use it for assignment + bool_mask = mask > 0 + for c in range(3): + output[c][bool_mask] = color[c] + + return output + + +def decode_partial_rle_width1(rle_obj: dict, start_row: int, end_row: int) -> np.ndarray: + """ + Decode a partial RLE encoded mask with width = 1. In SAM2 output, the video mask (num_frame, height, width) are reshaped to (total_size, 1). + Sometimes the video mask could be large, e.g. 1001x1080x1092 shape and it takes >1GB memory if using pycocotools, resulting in segmentation faults when training with multiple GPUs and data workers. + This function is used to decode the mask for a subset of frames to reduce memory usage. + + Args: + rle_obj (dict): RLE object containing: + - 'size': A list [height, width=1] indicating the dimensions of the mask. + - 'counts': A bytes or string object containing the RLE encoded data. + start_row (int): The starting row (inclusive). It's computed from frame_start * height * width. + end_row (int): The ending row (exclusive). It's computed from frame_end * height * width. + + Returns: + numpy.ndarray: Decoded binary mask for the specified rows as a 1D numpy array. + """ + height, width = rle_obj["size"] + + # Validate row range + if width != 1: + raise ValueError("This function is optimized for width=1.") + if start_row < 0 or end_row > height or start_row >= end_row: + raise ValueError("Invalid row range specified.") + + # Decode the RLE counts + counts = rle_obj["counts"] + if isinstance(counts, str): + counts = np.frombuffer(counts.encode("ascii"), dtype=np.uint8) + elif isinstance(counts, bytes): + counts = np.frombuffer(counts, dtype=np.uint8) + else: + raise ValueError("Unsupported format for counts. Must be str or bytes.") + + # Interpret counts as a sequence of run lengths + run_lengths = [] + current_val = 0 + i = 0 + while i < len(counts): + x = int(0) + k = 0 + more = True + while more: + c = int(counts[i]) - 48 + x |= (c & 0x1F) << (5 * k) + more = (c & 0x20) != 0 + i += 1 + k += 1 + if not more and (c & 0x10): + x |= -1 << (5 * k) + if len(run_lengths) > 2: + x += run_lengths[-2] + + run_lengths.append(x) + current_val += x + if current_val > end_row: + break + # Initialize the partial mask + idx_start = start_row + idx_end = end_row + partial_mask = np.zeros(idx_end - idx_start, dtype=np.uint8) + partial_height = end_row - start_row + idx = 0 # Current global index + for i, run in enumerate(run_lengths): + run_start = idx + run_end = idx + run + if run_end <= idx_start: + # Skip runs entirely before the region + idx = run_end + continue + if run_start >= idx_end: + # Stop decoding once we pass the region + break + + # Calculate overlap with the target region + start = max(run_start, idx_start) + end = min(run_end, idx_end) + if start < end: + partial_start = start - idx_start + partial_end = end - idx_start + partial_mask[partial_start:partial_end] = i % 2 + + idx = run_end + return partial_mask.reshape((partial_height, 1), order="F") diff --git a/cosmos_framework/data/vfm/augmentors/transfer_control_transform.py b/cosmos_framework/data/vfm/augmentors/transfer_control_transform.py new file mode 100644 index 0000000..74fb523 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/transfer_control_transform.py @@ -0,0 +1,317 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +""" +Augmentors for transfer (control-conditioned) image and video generation in the cosmos3 VFM pipeline. + +Transfer training conditions the model on control signals (edge, blur, depth, or segmentation) +to generate images or videos, aligned with cosmos_framework/transfer2. This module provides: + +- **TransferToTrainingFormat**: Converts (control_input, target) into the joint dataloader format + with SequencePlan (condition frame + generated frame), for both image and video outputs. + +- **VideoTransferSampleFrame**: For video→image transfer: samples a single frame index consistently + across control and video tensors, producing image-sized tensors from 4D video inputs. +- **AddControlFromVideoComb**: Uses AddControlInputComb (in transfer_control_input) to compute one of edge/blur/depth/seg + from video or precomputed fields and writes the chosen control to data_dict["control_input"]. +- **SampleResolution**: Samples a resolution from a list and sets data_dict["_res_size_map"] so downstream + resize/padding use that resolution (used to combine multiple resolutions in one dataloader). +""" + +from __future__ import annotations + +import random +from typing import cast + +import torch +import torchvision.transforms.functional as transforms_F + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.augmentors.transfer_control_input import AddControlInputComb +from cosmos_framework.data.vfm.sequence_packing import SequencePlan +from cosmos_framework.data.vfm.utils import VIDEO_RES_SIZE_INFO + + +class SampleResolution(Augmentor): + """Sample one resolution from a list and set data_dict['_res_size_map'] for downstream resize/padding. + + When used before ResizeLargestSideAspectPreserving and ReflectionPadding, those augmentors will + use obtain_augmentation_size(), which reads _res_size_map when present. This allows one dataloader + to produce samples at different resolutions (e.g. 480 and 720) by sampling per sample. + + resolutions_weights: Optional sampling weights for each resolution (same length as resolutions). + Weights are used by random.choices and need not sum to 1. If None, sampling is uniform. + """ + + def __init__( + self, + input_keys: list, + output_keys: list | None = None, + args: dict | None = None, + resolutions: list[str] | None = None, + resolutions_weights: list[float] | None = None, + **kwargs, + ) -> None: + super().__init__(input_keys, output_keys, args) + self.resolutions = list(resolutions) if resolutions else [] + assert len(self.resolutions) > 0, "SampleResolution requires at least one resolution." + for r in self.resolutions: + assert r in VIDEO_RES_SIZE_INFO, f"Unknown resolution {r}; known: {list(VIDEO_RES_SIZE_INFO.keys())}" + self.resolutions_weights = resolutions_weights + if self.resolutions_weights is not None: + assert len(self.resolutions_weights) == len(self.resolutions), ( + "resolutions_weights must have same length as resolutions." + ) + + def __call__(self, data_dict: dict) -> dict: + if self.resolutions_weights is not None: + res = random.choices(self.resolutions, weights=self.resolutions_weights, k=1)[0] + else: + res = random.choice(self.resolutions) + data_dict["_res_size_map"] = VIDEO_RES_SIZE_INFO[res] + return data_dict + + +class TransferToTrainingFormat(Augmentor): + """Convert (control_input, target) into joint-dataloader training format with SequencePlan. + + Reads data_dict["control_input"] and data_dict["video"] (target). Normalizes control to + mean/std 0.5, then writes [control_tensor, target_tensor] into data_dict[output_media_key] + ("images" for image transfer, "video" for video transfer). Also sets num_frames, + dataset_name, fps, ai_caption, selected_caption_type, sequence_plan, and image_size. + + Supports both image (3D: C,H,W) and video (4D: C,T,H,W); for image output, 4D tensors + are sliced to the first frame. Same output structure as ImageEditingToTrainingFormat. + """ + + def __init__( + self, + input_keys: list | None = None, + mean: float = 0.5, + std: float = 0.5, + output_media_key: str = "images", + conditioning_config: dict[int, float] | None = None, + share_vision_temporal_positions: bool = True, + args: dict | None = None, + ) -> None: + super().__init__(input_keys or [], None, args) + self.mean = mean + self.std = std + self.output_media_key = output_media_key + self.conditioning_config = conditioning_config + self.share_vision_temporal_positions = share_vision_temporal_positions + + if self.conditioning_config is not None: + for num_frames, prob in self.conditioning_config.items(): + if not isinstance(num_frames, int) or num_frames < 0: + raise ValueError(f"conditioning_config keys must be non-negative integers, got {num_frames}") + if not isinstance(prob, (int, float)) or prob < 0: + raise ValueError(f"conditioning_config values must be non-negative numbers, got {prob}") + total_prob = sum(self.conditioning_config.values()) + if total_prob <= 0: + raise ValueError("conditioning_config probabilities must sum to a positive number") + self.normalized_conditioning_config = {k: v / total_prob for k, v in self.conditioning_config.items()} + else: + self.normalized_conditioning_config = None + + def _normalize_tensor(self, x: torch.Tensor) -> torch.Tensor: + """Normalize channel-wise to given mean/std. Accepts values in [0,1] or [0,255] (auto-detected).""" + if x.dtype == torch.uint8 or x.max() > 1.0: + x = x.float() / 255.0 + return transforms_F.normalize(x, mean=[self.mean] * 3, std=[self.std] * 3) + + def __call__(self, data_dict: dict) -> dict | None: + control_norm = data_dict.get("control_input") + target_norm = data_dict.get("video") + + if control_norm is None or target_norm is None: + log.warning( + f"TransferToTrainingFormat: missing control or target (video): {data_dict.get('__key__', 'unknown')}", + rank0_only=False, + ) + return None + + try: + if control_norm.dim() == 2: + control_norm = control_norm.unsqueeze(0).expand(3, -1, -1) # [3,H,W] + # is_video = control.dim() == 4 and isinstance(target_norm, torch.Tensor) and target_norm.dim() == 4 + if self.output_media_key == "video": + # Video: (C, T, H, W) each; normalize per frame + # control_norm = self._normalize_tensor(control.float()) + num_frames = control_norm.shape[1] + data_dict["video"] = [control_norm, target_norm] + data_dict["num_frames"] = num_frames + data_dict["dataset_name"] = "video_transfer" + data_dict["fps"] = data_dict.get("fps", 24.0) + else: + # Image: (C, H, W) + if target_norm.dim() == 4: + target_norm = target_norm[:, 0] + if control_norm.dim() == 4: + control_norm = control_norm[:, 0] + # control_norm = self._normalize_tensor(control.float()) + data_dict["images"] = [control_norm, target_norm] + data_dict["num_frames"] = 2 + data_dict["dataset_name"] = "image_transfer" + data_dict["fps"] = 30.0 + data_dict.setdefault("ai_caption", "") + data_dict.setdefault("selected_caption_type", "transfer_caption") + + num_condition_frames = 0 + if self.normalized_conditioning_config is not None: + frames_options = list(self.normalized_conditioning_config.keys()) + weights = list(self.normalized_conditioning_config.values()) + num_condition_frames = random.choices(frames_options, weights=weights, k=1)[0] + if self.output_media_key == "video" and target_norm.dim() == 4: + max_cond = target_norm.shape[1] - 1 + num_condition_frames = min(num_condition_frames, max_cond) + + if num_condition_frames > 0 and target_norm.shape[1] > 1: + condition_frames_indexes = list(range(num_condition_frames)) + else: + condition_frames_indexes = [] + + data_dict["sequence_plan"] = SequencePlan( + has_text=True, + has_vision=True, + condition_frame_indexes_vision=condition_frames_indexes, + # ControlNet-style transfer: control item and target item are + # spatio-temporally aligned (same source video, frame-synced). + # Forces shared temporal mRoPE grid across both items so the + # model sees control_t=k and target_t=k as the same time index. + share_vision_temporal_positions=self.share_vision_temporal_positions, + ) + except Exception as e: + log.warning( + f"TransferToTrainingFormat error: {data_dict.get('__key__', 'unknown')}, {e}", + rank0_only=False, + ) + return None + + # duplicate image_size for each vision input/output + data_dict["image_size"] = [data_dict["image_size"]] * len(data_dict[self.output_media_key]) + + return data_dict + + +class VideoTransferSampleFrame(Augmentor): + """Sample a single frame index from video tensors for image→image transfer. + + For each key in input_keys (default ["control_input", "video"]), resolves the + tensor (e.g. unwraps data_dict["video"]["video"]). Picks one temporal index t + (random if random_frame=True, else 0) and for every 4D tensor (C, T, H, W) + replaces it in-place with the slice at t, yielding (C, 1, H, W). 3D tensors + are left unchanged. All keys must be present; returns None if any is missing. + """ + + def __init__( + self, + input_keys: list | None = None, + args: dict | None = None, + random_frame: bool = True, + ) -> None: + self.input_keys = input_keys or ["control_input", "video"] + super().__init__(self.input_keys, None, args) + self.random_frame = random_frame + + def _get_tensor(self, data_dict: dict, key: str) -> torch.Tensor | None: + """Return the tensor for key; if key is 'video' and value is a dict, return value['video'].""" + val = data_dict.get(key) + if val is None: + return None + if isinstance(val, dict) and key == "video": + return val.get("video") + return val + + def __call__(self, data_dict: dict) -> dict | None: + # Resolve tensors; find T from first 4D tensor. Require all keys present. + tensors: list[tuple[str, torch.Tensor]] = [] + T: int | None = None + for key in self.input_keys: + raw = self._get_tensor(data_dict, key) + if raw is None or not isinstance(raw, torch.Tensor): + return None + tensor = cast(torch.Tensor, raw) + if tensor.dim() == 4: + if T is None: + T = tensor.shape[1] + if T == 0: + return None + tensors.append((key, tensor)) + else: + tensors.append((key, tensor)) + + if T is None: + # No 4D tensor; nothing to sample + return data_dict + + t_idx = random.randint(0, T - 1) if self.random_frame else 0 + + for key, tensor in tensors: + if tensor.dim() == 4: + sampled = tensor[:, t_idx : t_idx + 1] + else: + sampled = tensor + data_dict[key] = sampled + + return data_dict + + +class AddControlFromVideoComb(Augmentor): + """Compute one control signal from video via AddControlInputComb and set control_input. + + Delegates to AddControlInputComb (edge/blur computed from video; depth/seg from data_dict + when present). After the comb runs, selects the first non-zero control among + control_input_edge, control_input_blur, control_input_depth, control_input_seg, + writes it to data_dict["control_input"], and removes the temporary control keys. + + Args: + control_input_type: e.g. "edge_blur", "edge_blur_depth_seg" (which controls to consider). + num_control_inputs_prob: Probability distribution over number of combined controls; + this wrapper uses only the single chosen control. + """ + + CONTROL_KEYS = ("control_input_edge", "control_input_blur", "control_input_depth", "control_input_seg") + + def __init__( + self, + input_keys: list, + output_keys: list | None = None, + args: dict | None = None, + control_input_type: str = "edge_blur_depth_seg", + use_random: bool = True, + num_control_inputs_prob: tuple[float, ...] = (1.0, 0.0, 0.0, 0.0), + num_control_inputs: int | None = None, + **kwargs, + ) -> None: + super().__init__(input_keys, output_keys or ["control_input"], args) + self._comb = AddControlInputComb( + input_keys=input_keys, + output_keys=None, + use_random=use_random, + control_input_type=control_input_type, + num_control_inputs_prob=list(num_control_inputs_prob), + num_control_inputs=num_control_inputs, + **kwargs, + ) + + def __call__(self, data_dict: dict) -> dict | None: + data_dict = self._comb(data_dict) + if data_dict is None: + return None + # Pick first control key that exists and has non-zero data (comb sets unchosen to zeros). + for key in self.CONTROL_KEYS: + if key in data_dict: + t = data_dict[key] + if isinstance(t, torch.Tensor) and t.numel() > 0 and t.abs().sum() > 0: + data_dict["control_input"] = t + break + else: + # No break: no valid control found (e.g. all chosen controls failed or are zero). + log.warning("AddControlFromVideoComb: no non-zero control found", rank0_only=False) + return None + for key in self.CONTROL_KEYS: + data_dict.pop(key, None) + data_dict.pop(key + "_mask", None) + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/video_parsing.py b/cosmos_framework/data/vfm/augmentors/video_parsing.py new file mode 100644 index 0000000..cfaa934 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/video_parsing.py @@ -0,0 +1,841 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import random +from typing import Optional + +import numpy as np +import omegaconf +import torch +from einops import rearrange +from torchcodec.decoders import AudioDecoder, VideoDecoder +from torchvision.transforms.v2 import Resize, UniformTemporalSubsample + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.data.imaginaire.webdataset.augmentors.image.misc import obtain_augmentation_size +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.utils import VIDEO_RES_SIZE_INFO + +# Map dataset_resolution_type to resolution tier key in VIDEO_RES_SIZE_INFO +_DATASET_RESOLUTION_TIER: dict[str, str] = {"gt480p": "480", "gt720p": "720", "gt1080p": "1080"} + +_MIN_FPS = 10 +_MAX_FPS = 60 + + +class VideoParsing(Augmentor): + """ + This augmentor is used to parse the video bytes and get the video frames. + the return dict is back-compatible with old datasets, which video decoding happens in the decoder stage. + + Now uses torchcodec instead of decord for video decoding, with optional audio extraction. + """ + + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + assert len(input_keys) == 2, "VideoParsing augmentor only supports two input keys" + self.meta_key = input_keys[0] + self.video_key = input_keys[1] + + self.key_for_caption = args["key_for_caption"] + assert self.key_for_caption in [ + "t2w_windows", + "i2w_windows_later_frames", + ], "key_for_caption must be either t2w_windows or i2w_windows_later_frames" + self.min_duration = args["min_duration"] + self.min_fps = args["min_fps"] + self.max_fps = args["max_fps"] + self.num_frames = args["num_video_frames"] + self.use_native_fps = args["use_native_fps"] # orginal fps if (total_frames // self.num_frames == 1). + # a list of allowed num_multiplers (how many frames are skipped) + # default is 1 - 100 which allows virtually any num_multipler possible + self.allowed_num_multiplers = args.get("allowed_num_multiplers", list(range(1, 100))) + log.info(f"allowed_num_multiplers in video_parsing with use_native_fps: {self.allowed_num_multiplers}") + self.use_original_fps = args["use_original_fps"] # use original fps without sampling + + # Dynamic FPS mode: sample stride from valid range based on video properties + self.use_dynamic_fps = args.get("use_dynamic_fps", False) + # low_fps_bias: 0.0 = favor original FPS (stride=1), 0.5 = uniform, 1.0 = favor slow-mo (high stride) + self.low_fps_bias = args.get("low_fps_bias", 0.5) + assert 0.0 <= self.low_fps_bias <= 1.0, f"low_fps_bias must be in [0, 1], got {self.low_fps_bias}" + + # Validate mutually exclusive modes + mode_count = sum([self.use_dynamic_fps, self.use_native_fps, self.use_original_fps]) + assert mode_count <= 1, ( + f"Only one FPS mode can be enabled at a time. Got: " + f"use_dynamic_fps={self.use_dynamic_fps}, " + f"use_native_fps={self.use_native_fps}, " + f"use_original_fps={self.use_original_fps}" + ) + + if self.use_dynamic_fps: + log.info( + f"use_dynamic_fps mode enabled: stride will be sampled from valid range per video " + f"with low_fps_bias={self.low_fps_bias} (0.0=favor original FPS, 0.5=uniform, 1.0=favor slow-mo)" + ) + + if self.use_native_fps or self.use_original_fps: + assert self.num_frames > 0, "num_frames must be greater than 0 when use_native_fps is True" + if self.use_dynamic_fps: + assert self.num_frames > 0, "num_frames must be greater than 0 when use_dynamic_fps is True" + if self.num_frames > 0: + self.sampler = UniformTemporalSubsample(self.num_frames) + self.video_decode_num_threads = args.get("video_decode_num_threads", 1) + + # Audio extraction parameters + self.extract_audio = args.get("extract_audio", False) + self.audio_sample_rate = args.get("audio_sample_rate", 44100) + self.seek_mode = args.get("seek_mode", "exact") + + def _extract_audio_chunk( + self, video_bytes: bytes, video_fps: float, frame_indices: list[int] + ) -> torch.Tensor | None: # returns [C,N_audio] or None + """ + Extract audio chunk corresponding to the given frame indices. + + Args: + video_bytes: Raw video bytes + video_fps: Video frames per second + frame_indices: List of frame indices being extracted + + Returns: + Audio tensor of shape (C, N) or None if audio extraction fails + """ + try: + # Create audio decoder + audio_decoder = AudioDecoder(video_bytes) + + # Calculate time range for audio corresponding to video frames + time_start = frame_indices[0] / video_fps + time_end = (frame_indices[-1] + 1) / video_fps # +1 to include the last frame's duration + + # Get audio samples for the specific time range + audio_metadata = audio_decoder.metadata + orig_sample_rate = audio_metadata.sample_rate + + audio_samples = audio_decoder.get_samples_played_in_range(start_seconds=time_start, stop_seconds=time_end) + audio_chunk = audio_samples.data # [C,N_orig] + + # Resample if needed + if orig_sample_rate != self.audio_sample_rate: + import librosa + + audio_np = audio_chunk.numpy() + resampled_audio_np = librosa.resample( + audio_np, orig_sr=orig_sample_rate, target_sr=self.audio_sample_rate, axis=-1 + ) + audio_chunk = torch.from_numpy(resampled_audio_np) # [C,N_resampled] + + # Clean up audio decoder + del audio_decoder + + return audio_chunk + + except Exception as e: + log.warning(f"Failed to extract audio: {e}", rank0_only=False) + return None + + def _sample_stride_with_bias(self, max_stride: int) -> int: + """Sample a stride from [1, max_stride] with bias controlled by low_fps_bias. + + Args: + max_stride: Maximum valid stride value. + + Returns: + Sampled stride value. + + The bias controls the probability distribution: + - low_fps_bias=0.0: Favor stride=1 (original FPS) + - low_fps_bias=0.5: Uniform distribution + - low_fps_bias=1.0: Favor high strides (slow-mo / lower FPS) + """ + if max_stride == 1: + return 1 + + # Linear interpolation from (1 - bias) to bias, clamped to min 0.01 + strides = np.arange(1, max_stride + 1) + weights = np.linspace(1 - self.low_fps_bias, self.low_fps_bias, max_stride) + weights = np.maximum(weights, 0.01) + probs = weights / weights.sum() + + return int(np.random.choice(strides, p=probs)) + + def __call__(self, data_dict: dict) -> dict | None: + try: + meta_dict = data_dict[self.meta_key] + video = data_dict[self.video_key] + except Exception as e: + log.warning( + f"Cannot find video. url: {data_dict['__url__']}, key: {data_dict['__key__']}", rank0_only=False + ) + return None + + if not isinstance(video, bytes): + return data_dict + + video_info = { + "fps": meta_dict["framerate"], + "n_orig_video_frames": meta_dict["nb_frames"], + } + + if video_info["fps"] < self.min_fps: + log.warning(f"Video FPS {video_info['fps']} is less than min_fps {self.min_fps}", rank0_only=False) + return None + if video_info["fps"] > self.max_fps: + log.warning(f"Video FPS {video_info['fps']} is greater than max_fps {self.max_fps}", rank0_only=False) + return None + + options: list = list((i, item) for i, item in enumerate(meta_dict[self.key_for_caption])) + + # Skip the last window if possible. + # All windows except the last are 5 seconds long. The last window has a duration in the range [2.5s, 7.5), which is less preferred. + if len(options) > 1: + options = options[:-1] + + # shuffle options + random.shuffle(options) + video_frames = None + dynamic_conditioning_fps = None # Track conditioning FPS for dynamic mode + for chunk_index, option in options: + start_frame = option["start_frame"] + end_frame = option["end_frame"] + if (end_frame - start_frame) < self.min_duration * video_info["fps"]: + continue + + if self.use_native_fps or self.use_original_fps or self.use_dynamic_fps: + if (end_frame - start_frame) < self.num_frames: + continue + + # Create video decoder with torchcodec (directly from bytes) + video_decoder = VideoDecoder( + video, seek_mode=self.seek_mode, num_ffmpeg_threads=self.video_decode_num_threads + ) + + if self.use_dynamic_fps or self.use_native_fps or self.use_original_fps: + # Shared: Handle alpamayo - skip first 5 frames + if "alpamayo" in data_dict["__url__"].root: + start_frame += 5 + if (end_frame - start_frame) < self.num_frames: + continue + + total_frames = end_frame - start_frame + + # Compute num_multiplier based on mode + if self.use_dynamic_fps: + # Dynamic FPS mode: compute valid strides and sample with bias + max_stride = total_frames // self.num_frames + if max_stride < 1: + # Not enough frames even for stride=1, skip this chunk + continue + + # Sample stride with low_fps_bias controlling the distribution + num_multiplier = self._sample_stride_with_bias(max_stride) + + # Compute conditioning FPS based on sampled stride + dynamic_conditioning_fps = video_info["fps"] / num_multiplier + + fps_mode_desc = ( + "original_fps (contiguous)" if num_multiplier == 1 else f"subsampled (stride={num_multiplier})" + ) + log.info( + f"Dynamic FPS mode: video_fps={video_info['fps']}, total_frames={total_frames}, " + f"max_stride={max_stride}, sampled_stride={num_multiplier}, " + f"conditioning_fps={dynamic_conditioning_fps:.2f}, mode={fps_mode_desc}, " + f"low_fps_bias={self.low_fps_bias}", + rank0_only=False, + ) + elif self.use_native_fps: + # take mid self.num_frames frames from start frame to end frame. + # always try lower fps if possible. + num_multiplier = total_frames // self.num_frames + if num_multiplier not in self.allowed_num_multiplers: + log.debug( + f"Skipping chunk (native_fps): stride not allowed. num_multiplier={num_multiplier}, allowed={self.allowed_num_multiplers}" + ) + continue + else: # self.use_original_fps + # Original FPS mode: no frame skipping + num_multiplier = 1 + + # Shared: Check if we have enough frames for the selected stride + expected_length = self.num_frames * num_multiplier + if total_frames < expected_length: + log.info( + f"Skipping chunk: not enough frames for stride. total_frames={total_frames}, expected={expected_length}, num_multiplier={num_multiplier}", + rank0_only=False, + ) + continue + + # Shared: Select frames from the center of the window + _start_frame = start_frame + (total_frames - expected_length) // 2 + _end_frame = _start_frame + expected_length + frame_indices = list(range(_start_frame, _end_frame, num_multiplier)) + assert len(frame_indices) == self.num_frames, "frame_indices length is not equal to num_frames" + + # Decode frames with torchcodec + frame_batch = video_decoder.get_frames_at(frame_indices) + video_frames = frame_batch.data # [T,C,H,W] + video_frames = video_frames.permute(1, 0, 2, 3) # [C,T,H,W] + + # Clean up video decoder + del video_decoder + + # Extract audio if requested + audio_chunk = None + if self.extract_audio: + audio_chunk = self._extract_audio_chunk(video, video_info["fps"], frame_indices) # [C,N_audio] + + break + + else: + frame_indices = list(range(start_frame, end_frame)) + num_multiplier = 1 # No frame skipping in this block of code. + + # online hot-fix for alpamayo data. Skip the first 5 frames as there is chance that the first five frames contain black frames. + if "alpamayo" in data_dict["__url__"].root: + assert len(frame_indices) >= 5, ( + "Getting less than 5 frames for alpamayo videos. There is no way to skip the first five frames." + ) + frame_indices = frame_indices[5:] + start_frame += 5 + + # Decode frames with torchcodec + try: + frame_batch = video_decoder.get_frames_at(frame_indices) + except Exception as e: + # Some segmentation videos for Transfer are not long enough as the target video, skip them. + log.warning( + f"Video is not long enough, return None. url: {data_dict['__url__']}, key: {data_dict['__key__']}, error: {e}, start_frame: {start_frame}, end_frame: {end_frame}, frame_indices: {frame_indices}", + rank0_only=False, + ) + return None + video_frames = frame_batch.data # [T,C,H,W] + video_frames = video_frames.permute(1, 0, 2, 3) # [C,T,H,W] + + # Clean up video decoder + del video_decoder + + # Extract audio if requested + audio_chunk = None + if self.extract_audio: + audio_chunk = self._extract_audio_chunk(video, video_info["fps"], frame_indices) # [C,N_audio] + + break + + if video_frames is None: + log.warning( + f"No valid video frames found, return None. url: {data_dict['__url__']}, key: {data_dict['__key__']}", + rank0_only=False, + ) + return None + + video_info["chunk_index"] = chunk_index + video_info["frame_start"] = start_frame + video_info["frame_end"] = end_frame + video_info["num_frames"] = end_frame - start_frame # type: ignore + if self.num_frames > 0 and not (self.use_dynamic_fps or self.use_native_fps or self.use_original_fps): + # Uniform temporal subsampling mode (default when no FPS mode is enabled) + video_frames = rearrange( + self.sampler(rearrange(video_frames, "c t h w -> t c h w")), "t c h w -> c t h w" + ) # [C,T_sub,H,W] where T_sub = self.num_frames + num_multiplier = ( + end_frame - start_frame + ) / self.num_frames # Specifically for the uniform temporal subsampling case. + + video_info["video"] = video_frames + video_info["num_multiplier"] = num_multiplier # Store the frame skipping multiplier + + + # 1. Our video parser stores the original video FPS of the video. + # 2. We have multiple modes of frame selection -- consecutive chunk of frames or subsampled frames. + # Here's what we do in each case: + # + # A. Dynamic FPS mode (use_dynamic_fps=True): + # - We compute max possible stride based on total_frames // num_frames. + # - We sample a stride uniformly from [1, max_stride]. + # - We compute conditioning_fps = native_fps / stride. + # - This gives us a diverse range of effective FPS values. + # + # B. Consecutive chunk of frames (use_original_fps=True): + # - We use the stored FPS and the number of frames in the video. + # - We calculate the duration in seconds using the above two values. + # - conditioning_fps = native_fps (num_multiplier=1) + # + # C. Subsampled frames (use_native_fps=True or uniform subsampling): + # - We check the skipping_rate (1 / num_multiplier) in case of subsampling. + # - We adjust the conditioning FPS by the skipping_rate (faithful to original video's motion). + # - conditioning_fps = native_fps / num_multiplier + # - We calculate the duration in seconds using the adjusted conditioning FPS and the number of frames. + if dynamic_conditioning_fps is not None: + # Dynamic FPS mode: use the pre-computed conditioning FPS + video_info["conditioning_fps"] = dynamic_conditioning_fps + else: + # Other modes: compute effective FPS from stride + video_info["conditioning_fps"] = ( + video_info["fps"] / num_multiplier + ) # Effective FPS for RoPE modulation and text timestamps + + # Add audio if extracted + if audio_chunk is not None: + video_info["audio"] = audio_chunk + video_info["audio_sample_rate"] = self.audio_sample_rate + + # update data_dict, make it back-compatible with old datasets, which video decoding happens in the decoder stage. + data_dict[self.video_key] = video_info + + return data_dict + + +class VideoParsingWithFullFrames(Augmentor): + """ + This augmentor is used to parse the video bytes and get the video frames. + The caption is assumed to be for the entire video frames, rather than VideoParsing which assume captions are for a specific chunk of frames + """ + + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + assert len(input_keys) == 2, "VideoParsingWithFullFrames augmentor only supports two input keys" + self.meta_key = input_keys[0] + self.video_key = input_keys[1] + self.args = args + + # Dynamic FPS mode options + # If use_dynamic_fps=True, then we sample fps from a valid range of values. + # If use_dynamic_fps=False, then we use the original fps of the video (no frame skipping). + self.use_dynamic_fps = args.get("use_dynamic_fps", False) + # low_fps_bias: 0.0 = favor original FPS (stride=1), 0.5 = uniform, 1.0 = favor slow-mo (high stride) + self.max_stride = args.get("max_stride", 3) + self.min_stride = args.get("min_stride", 1) + assert self.max_stride >= self.min_stride, ( + f"max_stride ({self.max_stride}) must be >= min_stride ({self.min_stride})" + ) + self.min_fps = args.get("min_fps", _MIN_FPS) + self.max_fps = args.get("max_fps", _MAX_FPS) + if self.use_dynamic_fps: + log.info(f"use_dynamic_fps mode enabled: stride will be sampled from valid range per video ") + + self.video_decode_num_threads = args.get("video_decode_num_threads", 1) + self.seek_mode = args.get("seek_mode", "exact") + + self.size = args.get("size", None) + self.perform_resize = self.size is not None + + # Audio extraction parameters + self.extract_audio = args.get("extract_audio", False) + self.audio_sample_rate = args.get("audio_sample_rate", 48000) + # When True, emit placeholder sound=None and audio_sample_rate + # without extracting audio. Keeps output keys consistent across + # datasets that share the same dataloader (some with audio, some + # without). + self.emit_placeholder_sound = args.get("emit_placeholder_sound", False) + + # Resolution filter: when not "all", skip samples whose (width, height) are below the + # minimum for this aspect ratio in VIDEO_RES_SIZE_INFO[tier]. + self.dataset_resolution_type = args.get("dataset_resolution_type", "all") + self.resolution_tier = _DATASET_RESOLUTION_TIER.get(self.dataset_resolution_type) + + def _sample_stride_with_bias(self, max_stride: int, min_stride: int = 1) -> int: + """Sample a stride from [min_stride, max_stride] with bias controlled by low_fps_bias. + + Args: + max_stride: Maximum valid stride value. + min_stride: Minimum valid stride value. + + Returns: + Sampled stride value. + max_stride=3, min_stride=1, probs = [0.86681333, 0.11731043, 0.01587624] + These values are chosen to approximately match our old ablations. + TODO @pchattopadhy: Do ablations with this scheme + """ + assert max_stride >= min_stride, f"max_stride ({max_stride}) must be >= min_stride ({min_stride})" + if max_stride == min_stride: + return min_stride + + # Samples native fps stride mostly and picks low fps with some probability. + strides = np.arange(min_stride, max_stride + 1) + weights = np.exp(-2 * strides) + probs = weights / weights.sum() + return int(np.random.choice(strides, p=probs)) + + def _validate_and_probe(self, video: Optional[bytes], meta_dict: dict, data_dict: dict) -> bool: + """Validate video bytes, back-fill missing metadata via probing, and + enforce fps/resolution filters. + Returns True if the video is valid, False otherwise. + """ + + if not isinstance(video, bytes): + raise ValueError(f"Video is not bytes. url: {data_dict['__url__']}, key: {data_dict['__key__']}") + + if len(video) == 0: + log.warning( + f"Empty video bytes. url: {data_dict['__url__']}, key: {data_dict['__key__']}", rank0_only=False + ) + return False + + # Back-fill missing metadata keys (width, height, framerate, nb_frames) by probing the + # video stream header. Also probe when the sidecar framerate looks abnormal to verify + # against the actual video stream. + _needs_probe = any(k not in meta_dict for k in ("width", "height", "framerate", "nb_frames")) + _metadata_fps = meta_dict.get("framerate", 0) + _fps_suspicious = _metadata_fps > _MAX_FPS or _metadata_fps < _MIN_FPS + _needs_probe = _needs_probe or _fps_suspicious + if _needs_probe: + _probe = VideoDecoder(video, seek_mode=self.seek_mode) + meta_dict.setdefault("width", _probe.metadata.width) + meta_dict.setdefault("height", _probe.metadata.height) + meta_dict.setdefault("nb_frames", _probe.metadata.num_frames) + meta_dict["framerate"] = _probe.metadata.average_fps + del _probe + + # Skip videos with framerates outside [min_fps, max_fps] + if meta_dict["framerate"] > self.max_fps: + log.warning( + f"Skipping video with framerate {meta_dict['framerate']} > max_fps {self.max_fps}. " + f"url: {data_dict['__url__']}, key: {data_dict['__key__']}", + rank0_only=False, + ) + return False + if meta_dict["framerate"] < self.min_fps: + log.warning( + f"Skipping video with framerate {meta_dict['framerate']} < min_fps {self.min_fps}. " + f"url: {data_dict['__url__']}, key: {data_dict['__key__']}", + rank0_only=False, + ) + return False + + # Resolution check: skip sample if (width, height) are below the minimum for this aspect ratio + width = meta_dict["width"] + height = meta_dict["height"] + aspect_ratio: str | None = None + + if "__url__" in data_dict: + aspect_ratio = data_dict["__url__"].meta.opts["aspect_ratio"] + + # If the resolution of the video is smaller than the minimum resolution for the aspect ratio, skip the sample. This will ensure that we do not upsample any video. + if self.resolution_tier is not None: + min_w, min_h = VIDEO_RES_SIZE_INFO[self.resolution_tier][aspect_ratio] + if width < min_w and height < min_h: + return False + + return True + + def __call__(self, data_dict: dict) -> dict | None: + + # if in future we need to train with batch size > 1, need to pad frames + try: + meta_dict = data_dict[self.meta_key] + video = data_dict[self.video_key] + except Exception as e: + log.warning( + f"Cannot find video. url: {data_dict['__url__']}, key: {data_dict['__key__']}", rank0_only=False + ) + return None + + if not self._validate_and_probe(video, meta_dict, data_dict): + return None + + # Resize video frames if size is specified. This computes a scaling ratio that fits the + # video within the target size bounds while preserving the original aspect ratio. + # The resize transform is applied during decoding via VideoDecoder's transforms parameter. + if self.perform_resize: + img_size = obtain_augmentation_size(data_dict, {"size": self.size}) + assert isinstance(img_size, (tuple, omegaconf.listconfig.ListConfig)), ( + f"Arg size in resize should be a tuple, get {type(img_size)}, {img_size}" + ) + img_w, img_h = img_size + orig_w, orig_h = meta_dict["width"], meta_dict["height"] + + # Compute uniform scaling ratio to fit video within target bounds (aspect-ratio preserving) + scaling_ratio = min((img_w / orig_w), (img_h / orig_h)) + target_size = (int(scaling_ratio * orig_h + 0.5), int(scaling_ratio * orig_w + 0.5)) + + assert target_size[0] <= img_h and target_size[1] <= img_w, ( + f"Resize error. orig {(orig_w, orig_h)} desire {img_size} compute {target_size}" + ) + transform = [Resize(target_size)] + else: + transform = None + + # Adding try-expcept because some of the data is bad and video decoding call fail. + try: + video_decoder = VideoDecoder( + video, + seek_mode=self.seek_mode, + num_ffmpeg_threads=self.video_decode_num_threads, + transforms=transform, + ) + num_video_frames = len(video_decoder) + + stride = self._sample_stride_with_bias(self.max_stride, self.min_stride) + frame_indices = np.arange(0, num_video_frames, stride).tolist() + + # VAE compress temporal by 4x, with 1 as condition + # thus the max_video_frames must be 1 + 4N + num_video_frames = min(len(frame_indices), self.args.get("max_num_frames", 1000)) + N = (num_video_frames - 1) // 4 + num_video_frames = 1 + 4 * N + frame_indices = frame_indices[0:num_video_frames] + + frame_batch = video_decoder.get_frames_at(frame_indices) + video_frames = frame_batch.data # [T,C,H,W] + video_frames = video_frames.permute(1, 0, 2, 3) # [C,T,H,W] (T = num_video_frames) + + del video_decoder + except Exception as e: + log.warning( + f"Failed to decode video. url: {data_dict['__url__']}, key: {data_dict['__key__']}, error: {e}", + rank0_only=False, + ) + return None + + video_info = { + "frame_start": frame_indices[0], + "frame_end": frame_indices[-1], + "num_frames": len(frame_indices), + "video": video_frames, + "fps": meta_dict["framerate"], + "conditioning_fps": meta_dict["framerate"] / stride, + "n_orig_video_frames": num_video_frames, + } + + # Extract audio for the same time range as the video frames + if self.extract_audio: + audio_chunk = self._extract_audio_chunk( + video_bytes=video, video_fps=meta_dict["framerate"], frame_indices=frame_indices + ) + if audio_chunk is not None: + video_info["sound"] = audio_chunk + else: + video_info["sound"] = None + # Always include audio_sample_rate when extract_audio is enabled, + # even if audio extraction failed, so the collate function has a + # consistent set of keys across all samples in the batch. + video_info["audio_sample_rate"] = self.audio_sample_rate + elif self.emit_placeholder_sound: + video_info["sound"] = None + video_info["audio_sample_rate"] = self.audio_sample_rate + + data_dict[self.video_key] = video_info + + return data_dict + + def _extract_audio_chunk( + self, video_bytes: bytes, video_fps: float, frame_indices: list[int] + ) -> torch.Tensor | None: # returns [C,N_audio] or None + """Load audio from the clip, resample, and truncate to match video duration. + + Args: + video_bytes: Raw video bytes + video_fps: Video frames per second, used to compute video duration for truncation. + frame_indices: Frame indices extracted from the video. + + Returns: + Audio tensor of shape (C, N) or None if extraction fails. + """ + try: + # Quick check: probe container for audio streams before AudioDecoder init. + # AudioDecoder is slow when no audio stream exists. We use torchcodec._core + # (internal API) to read container metadata without setting up a decode pipeline. + # If this breaks on a future torchcodec upgrade, remove this block — AudioDecoder + # will still work, just slower on videos without audio. + try: + from torchcodec._core import create_from_bytes, get_container_metadata + + _handle = create_from_bytes(video_bytes) + _meta = get_container_metadata(_handle) + _has_audio = _meta.best_audio_stream_index is not None + del _handle, _meta + if not _has_audio: + return None + except (ImportError, AttributeError): + pass # Fall through to AudioDecoder if _core API is unavailable + + audio_decoder = AudioDecoder(video_bytes) + all_samples = audio_decoder.get_all_samples() + audio = all_samples.data # [C,N_orig] + orig_sr = all_samples.sample_rate + del audio_decoder, all_samples + + if orig_sr != self.audio_sample_rate: + import librosa + + audio = torch.from_numpy( + librosa.resample(audio.numpy(), orig_sr=orig_sr, target_sr=self.audio_sample_rate, axis=-1) + ) # [C,N_resampled] + + # Truncate audio to match the extracted video frame duration. + if len(frame_indices) > 0 and video_fps > 0: + video_duration = (frame_indices[-1] + 1) / video_fps + max_audio_samples = int(video_duration * self.audio_sample_rate) + if audio.shape[-1] > max_audio_samples: + audio = audio[:, :max_audio_samples] # [C,N_truncated] + + return audio.clone() # [C,N_audio] + + except Exception as e: + log.warning(f"Failed to extract audio: {e}", rank0_only=False) + return None + + +class VideoParsingChunkedFrames(VideoParsingWithFullFrames): + """ + This augmentor is used to parse the video bytes and get the video frames for a chunk of frames. + In the new scheme, we process + - Full frames if num_frames < 400 + - If num_frames >= 400, we caption only for the first n frame chunk + In this case, the video extraction needs to only extract the first n frame chunk + + Additionally, in robotics and AV data, we do multi-chunk captioning. + In this case, we need to sample a chunk uniformly at random and extract the video frames only for that chunk. + + The chunk's frame range is supplied by an upstream ``TextTransformForVideoJsonCaption`` + augmentor via ``data_dict["chunk_start_frame"]`` and ``data_dict["chunk_end_frame"]``. + Only frames in ``[chunk_start_frame, chunk_end_frame)`` (and the matching audio range) + are decoded. + """ + + def __init__(self, input_keys: list, output_keys: Optional[list] = None, args: Optional[dict] = None) -> None: + super().__init__(input_keys, output_keys, args) + + def __call__(self, data_dict: dict) -> dict | None: + + # if in future we need to train with batch size > 1, need to pad frames + try: + meta_dict = data_dict[self.meta_key] + video = data_dict[self.video_key] + except Exception as e: + log.warning( + f"Cannot find video. url: {data_dict['__url__']}, key: {data_dict['__key__']}", rank0_only=False + ) + return None + + if not self._validate_and_probe(video, meta_dict, data_dict): + return None + + # The chunk frame range must be supplied by an upstream caption-parsing augmentor + # (e.g. TextTransformForVideoJsonCaption). + if "chunk_start_frame" not in data_dict or "chunk_end_frame" not in data_dict: + log.warning( + f"VideoParsingChunkedFrames: missing chunk_start_frame/chunk_end_frame in data_dict. " + f"url: {data_dict['__url__']}, key: {data_dict['__key__']}", + rank0_only=False, + ) + return None + chunk_start = int(data_dict["chunk_start_frame"]) + chunk_end = int(data_dict["chunk_end_frame"]) + + # Resize video frames if size is specified. This computes a scaling ratio that fits the + # video within the target size bounds while preserving the original aspect ratio. + # The resize transform is applied during decoding via VideoDecoder's transforms parameter. + if self.perform_resize: + img_size = obtain_augmentation_size(data_dict, {"size": self.size}) + assert isinstance(img_size, (tuple, omegaconf.listconfig.ListConfig)), ( + f"Arg size in resize should be a tuple, get {type(img_size)}, {img_size}" + ) + img_w, img_h = img_size + orig_w, orig_h = meta_dict["width"], meta_dict["height"] + + # Compute uniform scaling ratio to fit video within target bounds (aspect-ratio preserving) + scaling_ratio = min((img_w / orig_w), (img_h / orig_h)) + target_size = (int(scaling_ratio * orig_h + 0.5), int(scaling_ratio * orig_w + 0.5)) + + assert target_size[0] <= img_h and target_size[1] <= img_w, ( + f"Resize error. orig {(orig_w, orig_h)} desire {img_size} compute {target_size}" + ) + transform = [Resize(target_size)] + else: + transform = None + + # Adding try-expcept because some of the data is bad and video decoding call fail. + try: + video_decoder = VideoDecoder( + video, + seek_mode=self.seek_mode, + num_ffmpeg_threads=self.video_decode_num_threads, + transforms=transform, + ) + decoder_len = len(video_decoder) + + # Clamp the chunk range to what the decoder actually has. + chunk_start_clamped = max(0, min(chunk_start, decoder_len)) + chunk_end_clamped = max(chunk_start_clamped, min(chunk_end, decoder_len)) + if chunk_end_clamped <= chunk_start_clamped: + log.warning( + f"VideoParsingChunkedFrames: empty chunk after clamping. " + f"chunk=[{chunk_start},{chunk_end}), decoder_len={decoder_len}, " + f"url: {data_dict['__url__']}, key: {data_dict['__key__']}", + rank0_only=False, + ) + del video_decoder + return None + + stride = self._sample_stride_with_bias(self.max_stride, self.min_stride) + frame_indices = np.arange(chunk_start_clamped, chunk_end_clamped, stride).tolist() + + # VAE compress temporal by 4x, with 1 as condition + # thus the max_video_frames must be 1 + 4N + num_video_frames = min(len(frame_indices), self.args.get("max_num_frames", 1000)) + N = (num_video_frames - 1) // 4 + num_video_frames = 1 + 4 * N + if num_video_frames < 1: + log.warning( + f"VideoParsingChunkedFrames: chunk too short for stride. " + f"chunk=[{chunk_start_clamped},{chunk_end_clamped}), stride={stride}, " + f"url: {data_dict['__url__']}, key: {data_dict['__key__']}", + rank0_only=False, + ) + del video_decoder + return None + frame_indices = frame_indices[0:num_video_frames] + if len(frame_indices) == 0: + del video_decoder + return None + + frame_batch = video_decoder.get_frames_at(frame_indices) + video_frames = frame_batch.data # [T,C,H,W] + video_frames = video_frames.permute(1, 0, 2, 3) # [C,T,H,W] (T = num_video_frames) + + del video_decoder + except Exception as e: + log.warning( + f"Failed to decode video. url: {data_dict['__url__']}, key: {data_dict['__key__']}, error: {e}", + rank0_only=False, + ) + return None + + video_info = { + "frame_start": frame_indices[0], + "frame_end": frame_indices[-1], + "num_frames": len(frame_indices), + "video": video_frames, + "fps": meta_dict["framerate"], + "conditioning_fps": meta_dict["framerate"] / stride, + "n_orig_video_frames": num_video_frames, + } + + # Extract audio for the same time range as the chunk's video frames. + if self.extract_audio: + audio_chunk = self._extract_audio_chunk( + video_bytes=video, video_fps=meta_dict["framerate"], frame_indices=frame_indices + ) + if audio_chunk is not None: + video_info["sound"] = audio_chunk + else: + video_info["sound"] = None + # Always include audio_sample_rate when extract_audio is enabled, + # even if audio extraction failed, so the collate function has a + # consistent set of keys across all samples in the batch. + video_info["audio_sample_rate"] = self.audio_sample_rate + elif self.emit_placeholder_sound: + video_info["sound"] = None + video_info["audio_sample_rate"] = self.audio_sample_rate + + data_dict[self.video_key] = video_info + + # Cleanup: this augmentor is the last consumer of metas in the json-caption pipeline. + # Also drop the chunk range markers now that the chunk has been decoded. + data_dict.pop(self.meta_key, None) + data_dict.pop("chunk_start_frame", None) + data_dict.pop("chunk_end_frame", None) + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/vlm/__init__.py b/cosmos_framework/data/vfm/augmentors/vlm/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/data/vfm/augmentors/vlm/bytes_to_media.py b/cosmos_framework/data/vfm/augmentors/vlm/bytes_to_media.py new file mode 100644 index 0000000..3f61341 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/bytes_to_media.py @@ -0,0 +1,213 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Augmentors for handling video loading from pickled bytes. +Copied from projects/cosmos/reason1/datasets/augmentors/bytes_to_media.py +Changes: + 1: fully support start frame end frame, s.t. we could remove the projects/cosmos/reason1/datasets/augmentors/bytes_to_media.py class for predict2 video support + 2: add processor in init, as we need to read the processing config during the video decoding process +""" + +import io +import pickle as pkl +from typing import Dict, Optional + +from PIL import Image, UnidentifiedImageError + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.vlm.video_decoder_qwen import _video_decoder_qwen_func +from cosmos_framework.data.vfm.processors.qwen3vl_processor import Qwen3VLProcessor +from cosmos_framework.utils.vfm.video_preprocess import tensor_to_pil_images + + +class BytesToMedia(Augmentor): + """ + Converts PKL bytes stored in a data dictionary into media. + + Handles input formats for the specified input key: + A dictionary mapping media names (str) to bytes objects. + + The output format is a dictionary mapping names to their respective decoded objects: + Input dict[str, bytes] -> Output dict[str, torch.Tensor | PIL.Image] + + Corrupted or non-decodable bytes are skipped with a warning. + """ + + def __init__( + self, + input_key: str = "media", + output_key: str = "media", + min_fps_thres: int = 4, + max_fps_thres: int = 60, + target_fps: float = 4.0, + min_video_token_length: int = 16, + max_video_token_length: int = 8192, + num_threads: int = 0, + random_augmentation: bool = False, + is_input_pickle_byptes: bool = True, + use_start_frame_end_frame: bool = False, + frame_count_random_range: Optional[list[int]] = None, + processor: Qwen3VLProcessor = None, + ) -> None: + """ + Args: + input_key (str): Key in the data_dict containing video/image data. + output_key (str): Key to store the resulting video frame tensors or PIL images. + min_fps_thres (int): Minimum FPS threshold for video decoding. + max_fps_thres (int): Maximum FPS threshold for video decoding. + target_fps (float): Target FPS for video decoding. + min_video_token_length (int): Minimum token length for video decoding. + max_video_token_length (int): Maximum token length for video decoding. + num_threads (int): Number of threads for video decoding. + random_augmentation (bool): Whether to apply random augmentation during decoding. + is_input_pickle_byptes (bool): Whether the input key is in the data_dict instead of pkl files. (For cosmos predict2 videos) + use_start_frame_end_frame (bool): Whether to use start_frame and end_frame to decode the video. (For cosmos predict2 videos) + frame_count_random_range (list[int], optional): Random frame count range. Defaults to None. + """ + self.input_key = input_key + self.output_key = output_key + self.video_decoder_params = { + "min_fps_thres": min_fps_thres, + "max_fps_thres": max_fps_thres, + "target_fps": target_fps, + "min_video_token_length": min_video_token_length, + "max_video_token_length": max_video_token_length, + "num_threads": num_threads, + "random_augmentation": random_augmentation, + "frame_count_random_range": frame_count_random_range, + } + self.is_input_pickle_byptes = is_input_pickle_byptes + self.use_start_frame_end_frame = use_start_frame_end_frame + self.processor = processor + + def _bytes_to_video_frames( + self, video_bytes: bytes, identifier: str = "video", start_frame: int = None, end_frame: int = None + ) -> Optional[Dict]: + """Converts video bytes to video frame tensors using the video decoder.""" + try: + result = _video_decoder_qwen_func( + key=f"{identifier}.mp4", # Add .mp4 extension for the decoder + data=video_bytes, + processor=self.processor, + start_frame=start_frame, + end_frame=end_frame, + **self.video_decoder_params, + ) + result["videos"] = tensor_to_pil_images(result["videos"]) # 3,T,H,W -> list of PIL images + if result is not None: + return result + else: + log.warning(f"Skipping item '{identifier}': Video decoder returned None.") + return None + except Exception as e: + log.warning(f"Skipping item '{identifier}': Error decoding video bytes: {e}") + return None + + def _perhaps_unpickle_image_bytes(self, image_bytes: bytes) -> bytes: + """Unpickles the image bytes if it's double-pickled.""" + if image_bytes[:3] == b"\x80\x04\x95": + nested_data = pkl.loads(image_bytes) + if isinstance(nested_data, dict) and "image" in nested_data: + image_bytes = nested_data["image"] + else: + image_bytes = nested_data + return image_bytes + + def _bytes_to_pil(self, image_bytes: bytes, identifier: str = "image") -> Optional[Image.Image]: + """Converts a single bytes object to a PIL Image.""" + image_bytes = self._perhaps_unpickle_image_bytes(image_bytes) + try: + with io.BytesIO(image_bytes) as stream: + img = Image.open(stream) + img.load() # Verify the image data + return img.convert("RGB") # Convert to standard RGB format + except UnidentifiedImageError: + log.warning(f"Skipping item '{identifier}': Cannot identify image file from bytes.") + except Exception as e: + log.warning(f"Skipping item '{identifier}': Error decoding image bytes: {e}") + return None + + def __call__(self, data_dict: Dict) -> Dict: + """ + Processes the data_dict to convert video/image bytes to their respective formats. + + Args: + data_dict (Dict): The input data dictionary. + + Returns: + Dict: The modified data dictionary with video frame tensors and/or PIL images. + """ + input_key = self.input_key + output_key = self.output_key + + if input_key not in data_dict: + log.debug( + f"Input key '{input_key}' not found in data_dict. Skipping BytesToMedia. Available keys: {data_dict.keys()}" + ) + return data_dict + + raw = data_dict[input_key] + if self.is_input_pickle_byptes: + if isinstance(raw, bytes): + # Old webdataset (<1.0.2): .pkl files not auto-decoded, raw bytes arrive here + data = pkl.loads(raw) + elif isinstance(raw, dict): + # New webdataset (>=1.0.2): basichandlers runs as default post-handler, + # auto-decoding .media.pkl before this augmentor — use directly + data = raw + else: + raise ValueError(f"Input key '{input_key}' has unexpected type {type(raw)}; expected bytes or dict.") + else: + data = raw + output_data = {} + + if isinstance(data, dict): + for name, item in data.items(): + if isinstance(item, bytes): + # Determine if this is video or image based on the key name + if ("video" in name.lower() or ".mp4" in name.lower()) and not self.use_start_frame_end_frame: + # Decode as video + result = self._bytes_to_video_frames(item, identifier=f"{input_key}['{name}']") + if result: + output_data[name] = result + elif ("video" in name.lower() or ".mp4" in name.lower()) and self.use_start_frame_end_frame: + assert "start_frame" in data_dict.keys() and "end_frame" in data_dict.keys(), ( + f"start_frame and end_frame are not in data_dict.keys(): {data_dict.keys()}" + ) + start_frame = data_dict["start_frame"] + end_frame = data_dict["end_frame"] + result = self._bytes_to_video_frames( + item, identifier=f"{input_key}['{name}']", start_frame=start_frame, end_frame=end_frame + ) + if result: + output_data[name] = result + + elif ( + "image" in name.lower() + or ".jpg" in name.lower() + or ".jpeg" in name.lower() + or ".png" in name.lower() + ): + # Decode as image + result = self._bytes_to_pil(item, identifier=f"{input_key}['{name}']") + if result: + output_data[name] = result + else: + log.warning( + f"Skipping item with key '{name}' in '{input_key}': Key does not contain 'video', '.mp4', '.jpg', '.jpeg', '.png', or 'image'." + ) + else: + log.warning(f"Skipping item with key '{name}' in '{input_key}': Expected bytes, got {type(item)}.") + else: + raise ValueError( + f"Input key '{input_key}' has unsupported type {type(data)}. " + f"Expected dict[str, bytes] for video/image data." + ) + + # Add the processed data and optionally remove the input key + data_dict[output_key] = output_data + if input_key != output_key: + del data_dict[input_key] + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/vlm/filter_output_key.py b/cosmos_framework/data/vfm/augmentors/vlm/filter_output_key.py new file mode 100644 index 0000000..409d558 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/filter_output_key.py @@ -0,0 +1,58 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Augmentations to remove keys from the output data_dict""" + +from typing import Dict, List, Optional + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log + + +class FilterOutputKey(Augmentor): + """ + Keep a subset of keys in the output data_dict + """ + + def __init__( + self, + input_keys: List = [], + output_keys: Optional[list] = [ + "__key__", + "__url__", + "dialog_str", + "input_ids", + "token_mask", + "attention_mask", + "pixel_values_videos", + "video_grid_thw", + "second_per_grid_ts", + "raw_video", # for debugging + "pixel_values", + "image_grid_thw", + "raw_image", # for debugging + # For collate_fn + "pad_token_id", + "ignore_index", + "labels", + ], + text_only: bool = False, + args: Optional[dict] = None, + ) -> None: + self.output_keys = output_keys + self.text_only = text_only + + def __call__(self, data_dict: Dict) -> Dict: + data_dict = {k: data_dict[k] for k in self.output_keys if k in data_dict} + + has_media = "pixel_values" in data_dict or "pixel_values_videos" in data_dict + has_text = "input_ids" in data_dict and "labels" in data_dict + is_valid_data = has_media or has_text + if not self.text_only and not is_valid_data: + log.critical( + f"No media input in data_dict: {data_dict.keys()} | __url__: {data_dict['__url__']} | __key__: {data_dict['__key__']} | dialog_str: {data_dict.get('dialog_str', '')} | does not contain pixel_values or pixel_values_videos", + rank0_only=False, + ) + return None + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/vlm/filter_seq_length.py b/cosmos_framework/data/vfm/augmentors/vlm/filter_seq_length.py new file mode 100644 index 0000000..99e8a46 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/filter_seq_length.py @@ -0,0 +1,53 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Augmentations to remove keys from the output data_dict""" + +from typing import Dict, List, Optional + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.processors.qwen3vl_processor import Qwen3VLProcessor + + +class FilterSeqLength(Augmentor): + """ + Check the sequence length of the input data_dict and filter out the samples that are too long (TODO: Instead of removing them, we can truncate the input ids, but need to make sure the image tokens are not truncated) + """ + + def __init__( + self, + input_keys: List = ["input_ids"], + output_keys: Optional[list] = ["input_ids"], + max_token_length: int = 24000, + processor: Qwen3VLProcessor = None, + ) -> None: + self.max_token_length = max_token_length + self.processor = processor + + def __call__(self, data_dict: Dict) -> Dict: + input_ids = data_dict["input_ids"] + if input_ids.shape[-1] > self.max_token_length: + # check if there is pixel values or pixel value videos in the remaining tokens, if not truncate the input ids + input_ids_extra = input_ids[self.max_token_length :] + has_video_tokens = sum(input_ids_extra == self.processor.video_token_id) > 0 + has_image_tokens = sum(input_ids_extra == self.processor.image_token_id) > 0 + if not has_video_tokens and not has_image_tokens: + log.debug( + f"Truncating input_ids from {input_ids.shape[-1]} to {self.max_token_length} because there are no video or image tokens in the remaining tokens | __url__: path={data_dict['__url__'].path} root={data_dict['__url__'].root} | __key__: {data_dict['__key__']} | dialog_str: {data_dict.get('dialog_str', '')}" + ) + data_dict["input_ids"] = data_dict["input_ids"][: self.max_token_length] + data_dict["token_mask"] = data_dict["token_mask"][: self.max_token_length] + data_dict["attention_mask"] = data_dict["attention_mask"][: self.max_token_length] + data_dict["labels"] = data_dict["labels"][: self.max_token_length] + return data_dict + + if input_ids.shape[-1] > self.max_token_length: + msg = f"Input ids length {input_ids.shape[-1]} is greater than max token length {self.max_token_length} | __url__: path={data_dict['__url__'].path} root={data_dict['__url__'].root} | __key__: {data_dict['__key__']} | dialog_str: {data_dict.get('dialog_str', '')}" + if "pixel_values" in data_dict: + msg += f" | pixel_values: {data_dict['pixel_values'].shape}" + if "pixel_values_videos" in data_dict: + msg += f" | pixel_values_videos: {data_dict['pixel_values_videos'].shape}" + log.critical(msg, rank0_only=False) + return None + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/vlm/floating_number_format.py b/cosmos_framework/data/vfm/augmentors/vlm/floating_number_format.py new file mode 100644 index 0000000..2aff566 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/floating_number_format.py @@ -0,0 +1,74 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import re +from typing import Dict + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor + + +def format_floating_number(text: str, decimal_places: int) -> str: + """ + Format floating point numbers in text according to the specified format. + + Args: + text: Input text containing floating point numbers + floating_number_format: Format string like '.2f', '2.2f', etc. + + Returns: + Text with floating point numbers formatted according to the format string + """ + # Pattern to match floating point numbers (including integers that could be floats) + # Matches: integers, decimals like 123.456, scientific notation, etc. + pattern = r"-?\d+\.?\d*(?:[eE][+-]?\d+)?" + + def replace_float(match: re.Match) -> str: + try: + num = float(match.group()) + # Format the number using the provided format string + # Handle format strings like '.2f' or '2.2f' + formatted = f"{num:.{decimal_places}f}".rstrip("0").rstrip(".") if decimal_places > 0 else str(int(num)) + return formatted + except (ValueError, TypeError): + # If conversion fails, return the original match + return match.group() + + # Replace all floating point numbers in the text + formatted_text = re.sub(pattern, replace_float, text) + return formatted_text + + +class FloatingNumberFormat(Augmentor): + def __init__( + self, + input_key: str = "conversation", + decimal_places: int = 2, + urls_needs_format: list = [], + processor=None, + ) -> None: + """ + Args: + input_keys (list): List of input keys. + """ + self.input_key = input_key + self.decimal_places = decimal_places + self.urls_needs_format = urls_needs_format + + def __call__(self, data_dict: Dict) -> Dict: + url = data_dict["__url__"] + if not any(url_pattern in url.root for url_pattern in self.urls_needs_format): + return data_dict + + for item in data_dict[self.input_key]: + if item["role"] == "user": + for content in item["content"]: + if content["type"] == "text": + content["text"] = format_floating_number(content["text"], self.decimal_places) + elif item["role"] == "assistant": + if isinstance(item["content"], list): + assert len(item["content"]) == 1 + assert item["content"][0]["type"] == "text" + item["content"] = format_floating_number(item["content"][0]["text"], self.decimal_places) + else: + item["content"] = format_floating_number(item["content"], self.decimal_places) + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/vlm/format_describe_anything.py b/cosmos_framework/data/vfm/augmentors/vlm/format_describe_anything.py new file mode 100644 index 0000000..8ca392e --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/format_describe_anything.py @@ -0,0 +1,273 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Augmentors for image captions with SoM prompting. +Copied from projects/cosmos/reason1/datasets/augmentors/format_describe_anything.py +Changes: + 1. Unify system prompt to 'You are a helpful assistant.' + 2. Move task requirements from system prompts to the end of user prompts. +""" + +import json +import random +from typing import Dict, List, Literal + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.data.vfm.augmentors.vlm.timestamp import markdown_to_list + + +# reorder dict entries +def reorder_dict_entries(conversation_data: List[Dict]) -> List[Dict]: + key_order = ["subject_id", "category", "caption"] + output_dict = {} + for key in key_order: + if key in conversation_data: + output_dict[key] = conversation_data[key] + return output_dict + + +def list_to_markdown(conversation_data: List[Dict]) -> str: + conversation_data = [reorder_dict_entries(item) for item in conversation_data] + json_string = json.dumps(conversation_data, indent=2) + return f"```json\n{json_string}\n```".strip() + + +def augment_assistant_message( + assistant_message: List[Dict], + output_format: Literal[ + "dense_image_caption_json_per_subject", + "dense_image_caption_plain_per_subject", + "caption_one_object", + "location_and_caption_json_one_category", + "location_and_caption_plain_one_category", + ], +): + if output_format == "dense_image_caption_json_per_subject": + output_message = list_to_markdown(assistant_message) + return output_message + elif output_format == "dense_image_caption_plain_per_subject": + output_message = "" + for item in assistant_message: + output_message += f"subject_id = <{item['subject_id']}> category = <{item['category']}> {item['caption']}\n" + return output_message + + elif output_format == "location_and_caption_json_one_category": + # remove category + assistant_message = [ + {"subject_id": item["subject_id"], "caption": item["caption"]} for item in assistant_message + ] + output_message = list_to_markdown(assistant_message) + return output_message + elif output_format == "location_and_caption_plain_one_category": + output_message = "" + for item in assistant_message: + output_message += f"subject_id = <{item['subject_id']}> {item['caption']}\n" + return output_message + + elif output_format == "caption_one_object": + return f"{assistant_message[0]['caption']}" + else: + raise ValueError(f"Invalid output format: {output_format}") + + +def augment_user_prompt( + assistant_message: List[dict], + output_format: Literal[ + "dense_image_caption_json_per_subject", + "dense_image_caption_plain_per_subject", + "caption_one_object", + "location_and_caption_json_one_category", + "location_and_caption_plain_one_category", + ], +): + if ( + output_format == "dense_image_caption_json_per_subject" + or output_format == "dense_image_caption_plain_per_subject" + ): + if random.random() < 0.5: + user_prompt = random.choice( + [ + "Caption the notable attributes in the provided image.", + "Describe the notable attributes in the provided image.", + "Summarize the notable attributes in the provided image.", + ] + ) + if random.random() < 0.5: + user_prompt = "Please " + user_prompt.lower() + else: + user_prompt = random.choice( + [ + "Can you caption the notable attributes in the provided image?", + "Can you describe the notable attributes in the provided image?", + "Can you summarize the notable attributes in the provided image?", + ] + ) + if output_format == "dense_image_caption_json_per_subject": + user_prompt += """ List and describe all marked subjects in the image with their categories and detailed captions using the following format: +```json +[ +{ +"subject_id": , +"category": , +"caption": , +}, +{ +"subject_id": , +"category": , +"caption": , +}, +] +``` +""" + else: + user_prompt += " Please provide captions of the tracked objects in the images using the following format: \nsubject_id = category = caption of event 1.\nsubject_id = category = caption of event 2.\n" + elif output_format == "caption_one_object": + event = assistant_message[0] + user_prompt = random.choice( + [ + f"What happen to the object with ID <{event['subject_id']}>?", + f"Describe the object with ID <{event['subject_id']}>?", + f"Provide a caption of the object with ID <{event['subject_id']}>?", + ] + ) + elif ( + output_format == "location_and_caption_json_one_category" + or output_format == "location_and_caption_plain_one_category" + ): + event = assistant_message[0] + + user_prompt = random.choice( + [ + f"Caption the attribute of the object with category <{event['category']}>.", + f"Please describe the attribute of the object with category <{event['category']}>.", + f"Please caption the attribute of the object with category <{event['category']}>.", + f"Summarize the attribute of the object with category <{event['category']}>.", + ] + ) + if output_format == "location_and_caption_json_one_category": + user_prompt += """ Find all marked subjects that belong to and describe them in detail using the following format: +```json +[ +{ + "subject_id": , + "caption": +}, +{ + "subject_id": , + "caption": +}, +] +```""" + else: + user_prompt += """ Find all marked subjects that belong to and describe them in detail using the following format: +subject_id = caption of event 1. +subject_id = caption of event 2.""" + else: + raise ValueError(f"Invalid output format: {output_format}") + return user_prompt + + +class FormatDescribeAnything(Augmentor): + def __init__( + self, + input_key: list = "media", + output_format: Literal[ + "dense_image_caption_per_subject", + "caption_one_object", + "location_and_caption_one_category", + "random", + ] = "random", + urls_needs_timestamp: list = ["tl_plm_sav_20250714"], + ) -> None: + """ + Args: + input_keys (list): List of input keys. + """ + self.input_key = input_key + self.output_format = output_format + self.urls_needs_timestamp = urls_needs_timestamp + + def __call__(self, data_dict: Dict) -> Dict: + url = data_dict["__url__"] + if not any(url_pattern in url.root for url_pattern in self.urls_needs_timestamp): + return data_dict + + if self.output_format == "random": + output_format = random.choice( + [ + "dense_image_caption_per_subject", + "caption_one_object", + "location_and_caption_one_category", + ] + ) + else: + output_format = self.output_format + + if output_format == "dense_image_caption_per_subject": + output_format = random.choice( + ["dense_image_caption_json_per_subject", "dense_image_caption_plain_per_subject"] + ) + elif output_format == "location_and_caption_one_category": + output_format = random.choice( + ["location_and_caption_json_one_category", "location_and_caption_plain_one_category"] + ) + + # find the assistant message and parse into a list of dictionaries + for item in data_dict["conversation"]: + if item["role"] == "assistant": + """ + content dict: + ```json + [ + { + "subject_id": "4", + "category": "doughnut", + "caption": "This doughnut has a golden-brown exterior and a light, airy inside, evenly covered with a shiny, clear sugar glaze." + }, + { + "subject_id": "5", + "category": "tray", + "caption": "This stainless steel tray is rectangular with rounded edges and includes a sequence of symmetrical cut-outs shaped like simplified flowers or four-leaf clovers." + }, + ] + ``` + """ + assistant_message = markdown_to_list(item["content"]) + if assistant_message is None: + return None # skip this sample + break + # sort assistant_message by object id + assistant_message = sorted(assistant_message, key=lambda x: int(x["subject_id"])) + + # if temporal localization or caption, sample one event + if output_format in ["caption_one_object"]: + assistant_message = random.sample(assistant_message, 1) + elif output_format in ["location_and_caption_json_one_category", "location_and_caption_plain_one_category"]: + available_category = list( + set([assistant_message_i["category"] for assistant_message_i in assistant_message]) + ) + # sample one subject id + category = random.choice(available_category) + assistant_message = [ + assistant_message_i + for assistant_message_i in assistant_message + if assistant_message_i["category"] == category + ] + + # process conversation + conversation = data_dict["conversation"] + for item in conversation: + if item["role"] == "system": + item["content"] = "You are a helpful assistant." + elif item["role"] == "user": + for content in item["content"]: + if content["type"] == "text": + content["text"] = augment_user_prompt(assistant_message, output_format) + if content["text"] is None: # parse error + return None + elif item["role"] == "assistant": + assistant_message = augment_assistant_message(assistant_message, output_format) + item["content"] = assistant_message + data_dict["conversation"] = conversation + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/vlm/nvlm_data_to_conversation.py b/cosmos_framework/data/vfm/augmentors/vlm/nvlm_data_to_conversation.py new file mode 100644 index 0000000..0526c46 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/nvlm_data_to_conversation.py @@ -0,0 +1,256 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Visual-Text Transformations or Augmentations.""" + +import json +import random +from typing import Dict, Optional + +import numpy as np +from PIL import ImageDraw + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log + + +def convert_conversation_role(messages): + """ + The original messages can be in the following format: + [ + {"from": "human", "value": "Hello, how are you?"}, + {"from": "gpt", "value": "I'm good, thank you!"}, + ] + or + [ + {"role": "user", "content": "Hello, how are you?"}, + {"role": "gpt", "content": "I'm good, thank you!"}, + ] + + The target format is: + [ + {"role": "user", "content": "Hello, how are you?"}, + {"role": "assistant", "content": "I'm good, thank you!"}, + ] + """ + role_mapping = { + "human": "user", + "gpt": "assistant", + "user": "user", + "assistant": "assistant", + "label": "assistant", + } + messages_converted = [] + for message in messages: + assert "from" in message or "role" in message, f"Invalid message: {message}" + assert "value" in message or "content" in message, f"Invalid message: {message}" + role = message["from"] if "from" in message else message["role"] + content = message["value"] if "value" in message else message["content"] + role = role_mapping[role] + messages_converted.append({"role": role, "content": content}) + return messages_converted + + +class NVLMImageDataConversation(Augmentor): + """ + This augmentor is used to convert the nvlm data to a conversation format. + It will take the data_dict with the following keys: + { + "data_class": str, + "images": List[PIL.Image.Image], + "text": str, + "words_boxes": Optional[List[List[int]]], + "words_text": Optional[List[str]], + "similarity_matrix": Optional[List[List[float]]], + } + and convert it to a dictionary with the following keys: + { + "conversation": List[Dict], # Can be taken by TokenizeData augmentors shared with all datasets + "media": Dict, + } + + The dataclass includes: + - SimilarityInterleavedWebdataset + - CaptioningWebdataset + - MultiChoiceVQAWebdataset + - VQAWebdataset + - OCRWebdataset + - TextOCRWebdataset + + SimilarityInterleavedWebdataset will come with the conversations + CaptioningWebdataset will come with the caption + MultiChoiceVQAWebdataset will come with the question and choices + VQAWebdataset will come with the question and answer + OCRWebdataset will come with the text and words_boxes + TextOCRWebdataset will come with the text and words_boxes + """ + + def __init__( + self, + input_keys: list = ["data_class", "images", "text", "words_boxes", "words_text"], + output_keys: Optional[list] = ["text"], + media_type: str = "image", + media_key_in_data_dict: str = "images", + ) -> None: + super().__init__(input_keys, output_keys, None) + self.media_type = media_type + self.media_key_in_data_dict = media_key_in_data_dict + + self.user_prompt_list = json.load( + open("projects/cosmos3/vlm/datasets/augmentors/user_prompt_caption_general.json", "r") + ) + self.user_prompt_ocr_list = json.load( + open("projects/cosmos3/vlm/datasets/augmentors/user_prompt_ocr.json", "r") + ) + + def __call__(self, data_dict: Dict) -> Dict: + try: + return self.try_parse(data_dict) + except Exception as e: + log.warning( + f"Error parsing data_dict: {e} | data_dict: {data_dict.keys()} | __url__: {data_dict['__url__']}" + ) + return None + + def try_parse(self, data_dict: Dict) -> Dict: + """ + The output data_dict will has key "conversation" and "media" + for the "conversation" key, it will be list of dict + + data['conversation'] = [ + {"role": "system", "content": "**"}, + { + "role": "user", + "content": [ + {"type": media_type, media_type: media_dict_key}, + {"type": "text", "text": user_prompt}, + ], + }, + {"role": "assistant", "content": caption}, + ] + """ + data_class = data_dict["data_class"] + media_dict = {} + user_content_list = [] + for media_id, media in enumerate(data_dict[self.media_key_in_data_dict]): + media_dict_key = f"{self.media_type}_{media_id}" + user_content_list.append({"type": self.media_type, self.media_type: media_dict_key}) + media_dict[media_dict_key] = media + + if data_class == "SimilarityInterleavedWebdataset": + messages = data_dict["texts"] + messages = convert_conversation_role(messages) + # Insert the user_content_list to the user content + for message_id, message in enumerate(messages): + if message["role"] == "user": # Add to the first user message + messages[message_id]["content"] = user_content_list + [ + {"type": "text", "text": messages[message_id]["content"]} + ] + break + + elif data_class == "CaptioningWebdataset": + raw_captions = data_dict["caption"] + user_prompt = random.choice(self.user_prompt_list) + user_content_list.append({"type": "text", "text": user_prompt}) + messages = [ + {"role": "user", "content": user_content_list}, + {"role": "assistant", "content": f"{raw_captions}"}, + ] + + elif data_class == "MultiChoiceVQAWebdataset": + if data_dict["correct_choice_idx"] == -1: + answer = data_dict["choices"] + user_prompt = "\nAnswer the question using a single word or phrase." + else: + answer = data_dict["correct_choice_idx"] + user_prompt = "\nAnswer with the option's letter from the given choices directly." + user_content_list.append( + {"type": "text", "text": f"{data_dict['context']} {data_dict['choices']}. {user_prompt}"} + ) + messages = [ + {"role": "user", "content": user_content_list}, + {"role": "assistant", "content": f"{answer}"}, + ] + elif data_class == "VQAWebdataset": + user_prompt = "\nAnswer the question using a single word or phrase." + answer = data_dict["answers"] + if isinstance(answer, list): + # random sample one answer + answer = random.choice(answer) + user_content_list.append({"type": "text", "text": f"{data_dict['context']} {user_prompt}"}) + messages = [ + {"role": "user", "content": user_content_list}, + {"role": "assistant", "content": f"{answer}"}, + ] + elif data_class == "OCRWebdataset": + user_prompt = random.choice(self.user_prompt_ocr_list) + if ( + "words_boxes" in data_dict + and "words_text" in data_dict + and isinstance(data_dict["words_boxes"], list) + and isinstance(data_dict["words_text"], list) + and len(data_dict["words_boxes"]) == len(data_dict["words_text"]) + ): + boxes = data_dict["words_boxes"] + text = data_dict["words_text"] + # random sample one box and text + index = random.randint(0, len(boxes) - 1) + box = boxes[index] + text = text[index] + user_prompt = ( + user_prompt + f"\nbox: {box}; original image size: {np.array(media_dict[media_dict_key]).shape}" + ) + assert len(media_dict) == 1, ( + f"media_dict: {media_dict} | user_prompt: {user_prompt} | __url__: {data_dict['__url__']}" + ) + # Draw the box on the image + image = media_dict[media_dict_key] + + log.info( + f"box: {box} | text: {text} | media_dict_key: {media_dict_key} | __url__: {data_dict['__url__']} | image shape: {np.array(image).shape}" + ) + if len(box) == 4: + draw = ImageDraw.Draw(image) + draw.rectangle(box, outline="red", width=2) + media_dict[media_dict_key] = image + + reply = text + elif "words_text" in data_dict: + reply = data_dict["words_text"] + else: + reply = data_dict["text"] + user_content_list.append({"type": "text", "text": user_prompt}) + messages = [ + {"role": "user", "content": user_content_list}, + {"role": "assistant", "content": f"{reply}"}, + ] + else: + log.warning(f"Invalid data class: {data_class}") + return None + + # Remove image tag in the user content if any + def remove_image_tag(text): + text = text.replace("\n", "").replace("", "").replace("", "") + return text + + for message_id in range(len(messages)): + if messages[message_id]["role"] == "user" and isinstance(messages[message_id]["content"], list): + for content_id in range(len(messages[message_id]["content"])): + if messages[message_id]["content"][content_id]["type"] == "text": + text = messages[message_id]["content"][content_id]["text"] + text = remove_image_tag(text) + messages[message_id]["content"][content_id]["text"] = text + elif messages[message_id]["role"] == "user" and isinstance(messages[message_id]["content"], str): + messages[message_id]["content"] = remove_image_tag(messages[message_id]["content"]) + + # Make sure the assistant text content is a string not float or int + if messages[message_id]["role"] == "assistant" and not isinstance(messages[message_id]["content"], list): + if isinstance(messages[message_id]["content"], dict): + if messages[message_id]["content"]["type"] == "text": + messages[message_id]["content"]["text"] = f"{messages[message_id]['content']['text']}" + else: + messages[message_id]["content"] = f"{messages[message_id]['content']}" + + data_dict["conversation"] = messages + data_dict["media"] = media_dict + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/vlm/nvlm_data_unify.py b/cosmos_framework/data/vfm/augmentors/vlm/nvlm_data_unify.py new file mode 100644 index 0000000..eb029eb --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/nvlm_data_unify.py @@ -0,0 +1,120 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Visual-Text Transformations or Augmentations.""" + +import io +from typing import Dict, Optional + +from PIL import Image + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.augmentors.vlm.nvlm_sample_loaders_and_part_filters import ( + get_data_class, + get_part_filter, + get_sample_loader, +) + + +class NVLMImageDataUnify(Augmentor): + """ + This augmentor is used to unify the data format of the nvlm data. + It will take the raw nvlm data tar and convert it to a dictionary with the following keys: + { + "__url__": str, + "__key__": str, + "data_class": str, + "images": List[PIL.Image.Image], + "text": str, + "words_boxes": Optional[List[List[int]]], + "words_text": Optional[List[str]], + "similarity_matrix": Optional[List[List[float]]], + } + """ + + def __init__( + self, + input_keys: list = ["raw_nvlm"], + output_keys: Optional[list] = [], + args: Optional[dict] = None, + data_path_prefix: list[str] = [ + "cosmos_framework/ar/v2/nvlm/", + ], # prefix of the data in s3 + ) -> None: + super().__init__(input_keys, output_keys, args) + self.data_path_prefix = data_path_prefix + + def convert_image(self, img): + try: + if isinstance(img, bytes): + img = Image.open(io.BytesIO(img)).convert("RGB") + elif isinstance(img, Image.Image): + img = img.convert("RGB") + pass # Image is already in PIL format + elif isinstance(img, list): + for i in range(len(img)): + img[i], success = self.convert_image(img[i]) + if not success: + return Image.new("RGB", (256, 256), (0, 0, 0)), False + return img, True + else: + raise ValueError(f"Invalid image type: {type(img)}") + + success = True + except Exception as e: + log.warning(f"Error processing image: {e}. Creating an empty black image.", rank0_only=False) + img = Image.new("RGB", (256, 256), (0, 0, 0)) # Creates a 256x256 black image + success = False + return img, success + + def __call__(self, data_dict: Dict) -> Dict: + url = data_dict["__url__"] + data_path = "/".join(url.path.split("/")[:-1]) # remove the last part of the path + sample_loader = get_sample_loader(data_path) + part_filter = get_part_filter(data_path) + data_class = get_data_class(data_path) + assert sample_loader is not None and part_filter is not None and data_class is not None, ( + f"sample_loader({sample_loader}) or part_filter({part_filter}) or data_class({data_class}) is not found for {data_path}" + ) + + raw = {"__url__": url, "__key__": data_dict["__key__"]} + output = {"__url__": url, "__key__": data_dict["__key__"]} + for k, v in data_dict.items(): + ext = k.split(".")[-1] + if part_filter(ext): + raw[ext] = v + try: + output_converted = sample_loader(raw) + # Here output_converted will be a dictionary with the following keys: + # { + # "__key__": str, + # "image": PIL.Image.Image, + # "images": List[PIL.Image.Image], + # "text": str, + # "words_boxes": Optional + # "words_text": Optional + # "similarity_matrix": Optional + # } + except Exception as e: + log.warning( + f"Error in sample_loader: {e}, sample_loader: {sample_loader}, data_path: {data_path}, raw: {raw.keys()}, original_data_dict: {data_dict.keys()}, __url__: {url}, __key__: {data_dict['__key__']}" + ) + return None + + output.update(output_converted) + if "image" not in output_converted and "images" not in output_converted: + success = False + log.warning(f"image not found in {output_converted.keys()}") + if "image" in output_converted: # Single image case + img, success = self.convert_image(output["image"]) + output["images"] = [img] # What should be the format for the iamges + elif "images" in output_converted: + output["images"] = output_converted["images"] + output["images"], success = self.convert_image(output["images"]) + if not success: + log.warning(f"image conversion failed for {data_dict['__key__']} url: {url} | Skip this data") + return None + output["data_class"] = data_class + + return output diff --git a/cosmos_framework/data/vfm/augmentors/vlm/nvlm_sample_loaders_and_part_filters.py b/cosmos_framework/data/vfm/augmentors/vlm/nvlm_sample_loaders_and_part_filters.py new file mode 100644 index 0000000..fabe0c3 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/nvlm_sample_loaders_and_part_filters.py @@ -0,0 +1,2815 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# Combined Sample Loaders +# Auto-generated script combining all sample_loader.py files (Dont edit this file! Edit the projects/cosmos/reasoning/v1/scripts/create_sample_loader_and_part_filter_file.py instead) + +import io + +import torch +from PIL import Image + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.data_sources.vlm.nvlm import data_path_mapping + +# This file was automatically generated by `nvgpt4 data prepare`. + +# import torch + + +def sample_loader_0(raw: dict) -> dict: # Note: Images are already decoded to tensors + + if "text" in raw: + caption = raw["text"] + else: + caption = raw["json"]["caption"] + return dict( + __key__=raw["__key__"], + image=raw["jpg"], # expected type: torch.Tensor + caption=caption, # expected type: str + ) + + +def part_filter_0(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg", "text") + + +# This file was automatically generated by `energon prepare`. + + + +def sample_loader_1(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_1(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `energon prepare`. + + + +def sample_loader_2(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_2(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `energon prepare`. + + + +def sample_loader_3(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_3(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `energon prepare`. + + + +def sample_loader_4(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_4(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_5(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_5(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_6(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + key = raw["__key__"] + if "docvqa" in key: + context = json_item["question"] + answers = json_item["answers"] + image = raw["jpg"] + answer_weights = json_item["answer_weights"] + elif "textvqa" in key or "lrv_instruct" in key: + context = json_item["question"] + answers = json_item["answer"] + image = raw["jpg"] + answer_weights = None + elif "stvqa" in key: + context = json_item["question"] + answers = json_item["answers"] + image = raw["jpg"] + answer_weights = [1.0] * len(json_item["answers"]) + elif "chartqa" in key: + context = json_item["query"] + answers = json_item["label"] + image = raw["png"] + answer_weights = None + elif "screenqa" in key: + image = raw["jpg"] + context = json_item["question"] + answers = json_item["ground_truth"] + answer_weights = [1.0] * len(json_item["ground_truth"]) + elif "HME100K" in key: + image = raw["jpg"] + context = "Please write out the expression of the formula in the image using LaTeX format." + answers = json_item["latex_formula"] + answer_weights = None + else: # scale, textbook + image = raw["jpg"] + context = json_item["question"] + answers = json_item["answer"] + answer_weights = None + + return dict( + __key__=key, + image=image, + context=context, + answers=answers, + answer_weights=answer_weights, + ) + + +def part_filter_6(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg", "png") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_7(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + + return dict( + __key__=raw["__key__"], + image=raw["jpg"], # expected type: torch.Tensor + context=j["question_string"], # expected type: str + answers=j["answer"], # expected type: typing.Union[typing.List[str], NoneType], default: None + answer_weights=None, # expected type: typing.Union[torch.Tensor, NoneType], default: None + ) + + +def part_filter_7(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_8(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + + return dict( + __key__=raw["__key__"], + image=raw["jpg"], # expected type: torch.Tensor + context=j["question"], # expected type: str + answers=str(j["answer"]), # expected type: typing.Optional[typing.List[str]], default: None + answer_weights=None, # expected type: typing.Optional[torch.Tensor], default: None + ) + + +def part_filter_8(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_9(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + + return dict( + __key__=raw["__key__"], + image=raw["jpg"], # expected type: torch.Tensor + context=j["question"].strip(), # expected type: str + answers=j["gt_answer"].strip(), # expected type: typing.Union[typing.List[str], NoneType], default: None + answer_weights=None, # expected type: typing.Union[torch.Tensor, NoneType], default: None + ) + + +def part_filter_9(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_10(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + return dict( + image=raw["jpg"], # expected type: torch.Tensor + context=j["question"], # expected type: str + answers=j["answer"], # expected type: typing.Optional[typing.List[str]], default: None + answer_weights=None, # expected type: typing.Optional[torch.Tensor], default: None + ) + + +def part_filter_10(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_11(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + + return dict( + __key__=raw["__key__"], + image=raw["jpg"], + context=j["question"], + answers=j["answer"], + answer_weights=None, + ) + + +def part_filter_11(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_12(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + return dict( + __key__=raw["__key__"], + image=raw["jpg"], # expected type: torch.Tensor + context=j["question"], # expected type: str + answers=j["answer"], # expected type: typing.Optional[typing.List[str]], default: None + answer_weights=None, # expected type: typing.Optional[torch.Tensor], default: None + ) + + +def part_filter_12(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_13(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + key = raw["__key__"] + + if "geoqa_plus" in key or "tqa" in key: + return dict( + __key__=raw["__key__"], + image=raw["jpg"], + context=json_item["question"], + choices=json_item["choices"], + correct_choice_idx=json_item["correct_answer_index"], + ) + elif "geometry3k" in key: + return dict( + __key__=raw["__key__"], + image=raw["jpg"], + context=json_item["question"], + choices=json_item["choices"], + correct_choice_idx=ord(json_item["answer"].lower()) - 97, + ) + else: # science_qa, ai2d + image_key = "png" if "png" in raw else "jpg" + if image_key not in raw: + log.warning(f"Image key {image_key} not found in with raw keys: {raw.keys()}") + return dict( + __key__=raw["__key__"], # science_qa_sample_{idx} + image=raw[image_key], # expected type: torch.Tensor + context=json_item["question"], # expected type: str + choices=json_item["choices"], # expected type: typing.Union[typing.List[str], NoneType], default: None + correct_choice_idx=json_item["correct_choice_index"], + ) + + +def part_filter_13(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "png", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_14(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + return dict( + __key__=raw["__key__"], # arxiv_qa_sample_{idx} + image=raw["jpg"], # expected type: torch.Tensor + context=json_item["question"], # expected type: str + choices=json_item["options"], # expected type: typing.Union[typing.List[str], NoneType], default: None + correct_choice_idx=json_item["correct_choice_index"], + ) + + +def part_filter_14(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_15(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + if json_item["question_type"] == "multi_choice": + correct_choice_idx = json_item["choices"].index(json_item["answer"]) + return dict( + __key__=raw["__key__"], + image=raw["jpg"], + context=json_item["question"], + choices=json_item["choices"], + correct_choice_idx=correct_choice_idx, + ) + else: + # A temporary hack for non multi-choice samples. + # If correct_choice_idx=-1, we should route it to the VQAWebdataset dataloading method. + # (74.7% free-text questions, 25.3% multi-choice questions) + return dict( + __key__=raw["__key__"], + image=raw["jpg"], + context=json_item["question"], + choices=[json_item["answer"]], + correct_choice_idx=-1, + ) + + +def part_filter_15(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_16(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + + return dict( + __key__="llava-{}".format(raw["__key__"]), images=[raw["jpg"]], texts=j["conversations"], similarity_matrix=None + ) + + +def part_filter_16(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_17(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_17(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_18(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_18(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_19(raw: dict) -> dict: # Note: Images are already decoded to tensors + + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, # expected type: torch.Tensor + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_19(part: str) -> bool: + + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_20(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + + return dict( + __key__="llava-{}".format(raw["__key__"]), images=[raw["jpg"]], texts=j["conversations"], similarity_matrix=None + ) + + +def part_filter_20(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_21(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + + return dict( + __key__="llava-{}".format(raw["__key__"]), images=[raw["png"]], texts=j["conversations"], similarity_matrix=None + ) + + +def part_filter_21(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "png") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_22(raw: dict) -> dict: # Note: Images are already decoded to tensors + + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, # expected type: torch.Tensor + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_22(part: str) -> bool: + + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_23(raw: dict) -> dict: # Note: Images are already decoded to tensors + + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, # expected type: torch.Tensor + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_23(part: str) -> bool: + + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_24(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_24(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_25(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + + return dict( + __key__="llava-{}".format(raw["__key__"]), images=[raw["jpg"]], texts=j["conversations"], similarity_matrix=None + ) + + +def part_filter_25(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_26(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + + return dict( + __key__="llava-{}".format(raw["__key__"]), images=[raw["jpg"]], texts=j["conversations"], similarity_matrix=None + ) + + +def part_filter_26(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_27(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_27(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_28(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_28(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_29(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_29(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_30(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_30(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_31(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_31(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_32(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_32(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_33(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_33(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_34(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_34(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_35(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_35(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_36(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_36(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_37(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_37(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_38(raw: dict) -> dict: + j = raw["json"] + + if "ReCTs" in raw["__key__"]: + return dict( + __key__=raw["__key__"], + image=raw["jpg"], + text="", + words_boxes=j["quads_1k_normalized"], + words_text=j["texts"], + ) + else: # coco-text-multi, textocr-multi + return dict( + __key__=raw["__key__"], + image=raw["jpg"], + text="", + words_boxes=j["bboxes_1k_normalized"], + words_text=j["texts"], + ) + + +def part_filter_38(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_39(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + return dict( + image=raw["jpg"], # expected type: torch.Tensor + text=" ".join(j["lines"]["text"]), # expected type: str + ) + + +def part_filter_39(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_40(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_40(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_41(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_41(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_42(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_42(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_43(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_43(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_44(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_44(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_45(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_45(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_46(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_46(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_47(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_47(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_48(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_48(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_49(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_49(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_50(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_50(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_51(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + for i, turn in enumerate(json_item["conversations"]): + if i > 0 and turn["from"] == "human" and "" in turn["value"]: + turn["value"] = turn["value"].replace("\n", "") + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_51(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_52(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_52(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_53(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + # for i, turn in enumerate(json_item['conversations']): + # if i > 0 and turn['from'] == 'human' and '' in turn['value']: + # turn['value'] = turn['value'].replace("\n", "") + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_53(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_54(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_54(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_55(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_55(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_56(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_56(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_57(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_57(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_58(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_58(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_59(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_59(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_60(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_60(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_61(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_61(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_62(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_62(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_63(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_63(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_64(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_64(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_65(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_65(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_66(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_66(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_67(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_67(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("img", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_68(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_68(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_69(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_69(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_70(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_70(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_71(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_71(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_72(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + images = [raw["jpg"]] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_72(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_73(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_73(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "img") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_74(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + img = Image.open(io.BytesIO(raw["img"])) + images = [img] + + return dict( + __key__="llava-{}".format(raw["__key__"]), + images=images, + texts=json_item["conversations"], + similarity_matrix=None, + ) + + +def part_filter_74(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "img") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_75(raw: dict) -> dict: # Note: Images are already decoded to tensors + + if "text" in raw: + caption = raw["text"] + else: + caption = raw["json"]["caption"] + return dict( + __key__=raw["__key__"], + image=raw["jpg"], # expected type: torch.Tensor + caption=caption, # expected type: str + ) + + +def part_filter_75(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg", "text") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_76(raw: dict) -> dict: # Note: Images are already decoded to tensors + + if "text" in raw: + caption = raw["text"] + else: + caption = raw["json"]["caption"] + return dict( + __key__=raw["__key__"], + image=raw["jpg"], # expected type: torch.Tensor + caption=caption, # expected type: str + ) + + +def part_filter_76(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg", "text") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_77(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + total = len(json_item["conversations"]) // 2 + idx = random.randrange(total) # noqa: F821 + human = json_item["conversations"][idx * 2] + out = json_item["conversations"][idx * 2 + 1] + + return dict( + __key__=raw["__key__"], + image=raw["jpg"], + context=human["value"].replace("\n", ""), + answers=out["value"], + answer_weights=None, + ) + + +def part_filter_77(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_78(raw: dict) -> dict: # Note: Images are already decoded to tensors + json_item = raw["json"] + + total = len(json_item["conversations"]) // 2 + idx = random.randrange(total) # noqa: F821 + human = json_item["conversations"][idx * 2] + out = json_item["conversations"][idx * 2 + 1] + + return dict( + __key__=raw["__key__"], + image=raw["jpg"], + context=human["value"].replace("\n", ""), + answers=out["value"], + answer_weights=None, + ) + + +def part_filter_78(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + + + +def sample_loader_79(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + if "answer" in j: + answers = [a[0] for a in j["answer"][0]] + answer_weights = torch.Tensor([float(a[1]) for a in j["answer"][0]]) + else: + answers = None + answer_weights = None + + return dict( + __key__=raw["__key__"], + image=raw["jpg"], # expected type: torch.Tensor + context=j["question"], # expected type: str + answers=answers, # expected type: typing.List[str] + answer_weights=answer_weights, # expected type: typing.Union[torch.Tensor, NoneType] + ) + + +def part_filter_79(part: str) -> bool: + # Filter for parts required by the sample_loader + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_80(raw: dict) -> dict: # Note: Images are already decoded to tensors + + return dict( + __key__=raw["__key__"], + image=raw["jpg"], # expected type: torch.Tensor + context=raw["json"]["question"], # expected type: str + answers=raw["json"]["answer"], # expected type: typing.Union[typing.List[str], NoneType], default: None + answer_weights=None, # expected type: typing.Union[torch.Tensor, NoneType], default: None + ) + + +def part_filter_80(part: str) -> bool: + + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_81(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + return dict( + image=raw["jpg"], # expected type: torch.Tensor + text=" ".join(j["lines"]["text"]), # expected type: str + ) + + +def part_filter_81(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_82(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + + return dict( + __key__=raw["__key__"], + image=raw["jpg"], + text=j["text"], + words_boxes=j["bbox_1k_normalized"], + ) + + +def part_filter_82(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_83(raw: dict) -> dict: + j = raw["json"] + + return dict( + __key__=raw["__key__"], + image=raw["jpg"], + text=j["text"], + words_boxes=j["bbox_1k_normalized"], + ) + + +def part_filter_83(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_84(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + + return dict( + __key__=raw["__key__"], + image=raw["jpg"], + text=j["text"], + words_boxes=j["bbox_1k_normalized"], + ) + + +def part_filter_84(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_85(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + + return dict(__key__=raw["__key__"], image=raw["jpg"], text=j["text"], words_boxes=j["quad_1k_normalized"]) + + +def part_filter_85(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_86(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + + return dict( + __key__=raw["__key__"], + image=raw["jpg"], + text=j["text"], + words_boxes=j["bbox_1k_normalized"], + ) + + +def part_filter_86(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_87(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + + quad = j["quad"] + quad = [val for point in quad for val in point] + + return dict( + image=raw["jpg"], # expected type: torch.Tensor + text=j["text"], # expected type: str + words_boxes=quad, # expected type: typing.Optional[torch.Tensor], default: None + ) + + +def part_filter_87(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_88(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + return dict( + __key__=raw["__key__"], + image=raw["jpg"], + text="", + words_boxes=j["bboxes_1k_normalized"], + words_text=j["texts"], + ) + + +def part_filter_88(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_89(raw: dict) -> dict: + j = raw["json"] + return dict( + __key__=raw["__key__"], + image=raw["jpg"], + text="", + words_boxes=j["bboxes_1k_normalized"], + words_text=j["texts"], + ) + + +def part_filter_89(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("jpg", "json") + + +# This file was automatically generated by `nvgpt4 data prepare`. + + + +def sample_loader_90(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + return dict( + __key__=raw["__key__"], + image=raw["jpg"], + text="", + words_boxes=j["quads_1k_normalized"], + words_text=j["texts"], + ) + + +def part_filter_90(part: str) -> bool: + + # E.g. if your dataset contains jpeg, txt and json, but you won't use json, + # remove it from the list, such that it is not decoded. If you need all, keep as is + return part in ("json", "jpg") + + + + +def sample_loader_91(raw: dict) -> dict: # Note: Images are already decoded to tensors + j = raw["json"] + if "answer" in j: + answers = [a[0] for a in j["answer"][0]] + answer_weights = torch.Tensor([float(a[1]) for a in j["answer"][0]]) + else: + answers = None + answer_weights = None + + return dict( + __key__=raw["__key__"], + image=raw["jpg"], # expected type: torch.Tensor + context=j["question"], # expected type: str + answers=answers, # expected type: typing.List[str] + answer_weights=answer_weights, # expected type: typing.Union[torch.Tensor, NoneType] + ) + + +def part_filter_91(part: str) -> bool: + # Filter for parts required by the sample_loader + return part in ("jpg", "json") + + +# Dataset -> Sample Loader Mapping +dataset_loader_mapping = { + "coco_train_val_restval": { + "sample_loader": "sample_loader_0", + "part_filter": "part_filter_0", + "data_class": "CaptioningWebdataset", + "data_weight": 0.01, + }, + "extended-sci/data/merged/CoT": { + "sample_loader": "sample_loader_1", + "part_filter": "part_filter_1", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.006, + }, + "extended-sci/data/merged/single-choice": { + "sample_loader": "sample_loader_2", + "part_filter": "part_filter_2", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.004, + }, + "extended-sci/data/extended-sci-3/CoT": { + "sample_loader": "sample_loader_3", + "part_filter": "part_filter_3", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.0006, + }, + "extended-sci/data/extended-sci-3/single-choice": { + "sample_loader": "sample_loader_4", + "part_filter": "part_filter_4", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.0004, + }, + "nvlm/wdai/data/SceMQA_processed": { + "sample_loader": "sample_loader_5", + "part_filter": "part_filter_5", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.0006, + }, + "nvlm/wdai/data/vqa_collection_doc_text_st_chart_scale_textbook_LRV_Screen": { + "sample_loader": "sample_loader_6", + "part_filter": "part_filter_6", + "data_class": "VQAWebdataset", + "data_weight": 0.08, + }, + "nvlm/wdai/data/plotqa/processed": { + "sample_loader": "sample_loader_7", + "part_filter": "part_filter_7", + "data_class": "VQAWebdataset", + "data_weight": 0.095, + }, + "nvlm/wdai/data/clevr-math/processed": { + "sample_loader": "sample_loader_8", + "part_filter": "part_filter_8", + "data_class": "VQAWebdataset", + "data_weight": 0.02, + }, + "nvlm/wdai/data/MMC-Instruction/processed": { + "sample_loader": "sample_loader_9", + "part_filter": "part_filter_9", + "data_class": "VQAWebdataset", + "data_weight": 0.07, + }, + "nvlm/wdai/data/ocrvqa/processed": { + "sample_loader": "sample_loader_10", + "part_filter": "part_filter_10", + "data_class": "VQAWebdataset", + "data_weight": 0.06, + }, + "nvlm/wdai/data/dude/processed": { + "sample_loader": "sample_loader_11", + "part_filter": "part_filter_11", + "data_class": "VQAWebdataset", + "data_weight": 0.01, + }, + "nvlm/wdai/data/VisualMRC/processed": { + "sample_loader": "sample_loader_12", + "part_filter": "part_filter_12", + "data_class": "VQAWebdataset", + "data_weight": 0.015, + }, + "nvlm/wdai/data/mcvqa_collection_scienceqa_ai2d_geoqaplus_geometry3k_tqa": { + "sample_loader": "sample_loader_13", + "part_filter": "part_filter_13", + "data_class": "MultiChoiceVQAWebdataset", + "data_weight": 0.025, + }, + "nvlm/wdai/data/arxiv_qa/processed": { + "sample_loader": "sample_loader_14", + "part_filter": "part_filter_14", + "data_class": "MultiChoiceVQAWebdataset", + "data_weight": 0.02, + }, + "nvlm/wdai/data/tabmwp/processed": { + "sample_loader": "sample_loader_15", + "part_filter": "part_filter_15", + "data_class": "MultiChoiceVQAWebdataset", + "data_weight": 0.015, + }, + "nvlm/wdai/data/ocr_vqa_aug/processed": { + "sample_loader": "sample_loader_16", + "part_filter": "part_filter_16", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.055, + }, + "nvlm/wdai/data/dvqa_full/processed": { + "sample_loader": "sample_loader_17", + "part_filter": "part_filter_17", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.055, + }, + "nvlm/wdai/data/LLaVA-v1.5_shuffle/no_refcoco_vg_ocrvqa": { + "sample_loader": "sample_loader_18", + "part_filter": "part_filter_18", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.085, + }, + "vqa/more_data/infographics_vqa/processed/train": { + "sample_loader": "sample_loader_19", + "part_filter": "part_filter_19", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.01, + }, + "nvlm/wdai/data/sharegpt4o/processed": { + "sample_loader": "sample_loader_20", + "part_filter": "part_filter_20", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.02, + }, + "nvlm/wdai/data/sparse_ocr_data/merged": { + "sample_loader": "sample_loader_21", + "part_filter": "part_filter_21", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.045, + }, + "nvlm/nayeonl/data/blendv4/MetaMathQA/processed/train_text_image": { + "sample_loader": "sample_loader_22", + "part_filter": "part_filter_22", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.004, + }, + "nvlm/nayeonl/data/blendv4/gsm8k/processed/train_text_image": { + "sample_loader": "sample_loader_23", + "part_filter": "part_filter_23", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.003, + }, + "nvlm/wdai/data/docmatix/processed": { + "sample_loader": "sample_loader_24", + "part_filter": "part_filter_24", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.1, + }, + "nvlm/wdai/data/bentham_hw_squad/processed": { + "sample_loader": "sample_loader_25", + "part_filter": "part_filter_25", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.01, + }, + "nvlm/wdai/data/WikiTableQA/processed": { + "sample_loader": "sample_loader_26", + "part_filter": "part_filter_26", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.003, + }, + "nvlm/wdai/data/figureqa/processed": { + "sample_loader": "sample_loader_27", + "part_filter": "part_filter_27", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.01, + }, + "nvlm/wdai/data/llava-onevision/ai2d_combined_processed": { + "sample_loader": "sample_loader_28", + "part_filter": "part_filter_28", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.01, + }, + "nvlm/wdai/data/llava-onevision/math_combined_processed": { + "sample_loader": "sample_loader_29", + "part_filter": "part_filter_29", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.035, + }, + "nvlm/wdai/data/llava-onevision/robut_combined_processed": { + "sample_loader": "sample_loader_30", + "part_filter": "part_filter_30", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.01, + }, + "nvlm/wdai/data/llava-onevision/llavar_20k_processed": { + "sample_loader": "sample_loader_31", + "part_filter": "part_filter_31", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.007, + }, + "nvlm/wdai/data/llava-onevision/tallyqa_processed": { + "sample_loader": "sample_loader_32", + "part_filter": "part_filter_32", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.02, + }, + "nvlm/wdai/data/llava-onevision/ureader_ie_processed": { + "sample_loader": "sample_loader_33", + "part_filter": "part_filter_33", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.007, + }, + "nvlm/wdai/data/llava-onevision/visual7w_processed": { + "sample_loader": "sample_loader_34", + "part_filter": "part_filter_34", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.006, + }, + "nvlm/wdai/data/llava-onevision/mavis_math_rule_geo_processed": { + "sample_loader": "sample_loader_35", + "part_filter": "part_filter_35", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.01, + }, + "nvlm/wdai/data/llava-onevision/ureader_kg_processed": { + "sample_loader": "sample_loader_36", + "part_filter": "part_filter_36", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.005, + }, + "nvlm/wdai/data/llava-onevision/ureader_qa_processed": { + "sample_loader": "sample_loader_37", + "part_filter": "part_filter_37", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.02, + }, + "nvlm/wdai/data/ocr_multi_collection_cocotext_textocr_ReCTs": { + "sample_loader": "sample_loader_38", + "part_filter": "part_filter_38", + "data_class": "OCRWebdataset", + "data_weight": 0.01, + }, + "nvlm/wdai/data/pdfa-eng-wds/processed_word_len_500": { + "sample_loader": "sample_loader_39", + "part_filter": "part_filter_39", + "data_class": "OCRWebdataset", + "data_weight": 0.015, + }, + "nvlm/wdai/data/llava-onevision/super_clevr_processed": { + "sample_loader": "sample_loader_40", + "part_filter": "part_filter_40", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.007, + }, + "nvlm/wdai/data/llava-onevision/icon_qa_processed": { + "sample_loader": "sample_loader_41", + "part_filter": "part_filter_41", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.009, + }, + "nvlm/wdai/data/augmentations/chartqa_aug": { + "sample_loader": "sample_loader_42", + "part_filter": "part_filter_42", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.005, + }, + "nvlm/wdai/data/augmentations/gpt_chartqa": { + "sample_loader": "sample_loader_43", + "part_filter": "part_filter_43", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.006, + }, + "nvlm/wdai/data/augmentations/gpt_docvqa": { + "sample_loader": "sample_loader_44", + "part_filter": "part_filter_44", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.006, + }, + "nvlm/wdai/data/augmentations/docvqa_text": { + "sample_loader": "sample_loader_45", + "part_filter": "part_filter_45", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.006, + }, + "nvlm/wdai/data/augmentations/textvqa_text": { + "sample_loader": "sample_loader_46", + "part_filter": "part_filter_46", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.008, + }, + "nvlm/wdai/data/augmentations/i2s-musicsheet": { + "sample_loader": "sample_loader_47", + "part_filter": "part_filter_47", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.0005, + }, + "nvlm/wdai/data/augmentations/music": { + "sample_loader": "sample_loader_48", + "part_filter": "part_filter_48", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.007, + }, + "nvlm/wdai/data/augmentations/invoice": { + "sample_loader": "sample_loader_49", + "part_filter": "part_filter_49", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.002, + }, + "nvlm/wdai/data/augmentations/k12": { + "sample_loader": "sample_loader_50", + "part_filter": "part_filter_50", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.019, + }, + "nvlm/wdai/data/augmentations/MTVQA": { + "sample_loader": "sample_loader_51", + "part_filter": "part_filter_51", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.007, + }, + "nvlm/wdai/data/augmentations/VisualWebInstruct": { + "sample_loader": "sample_loader_52", + "part_filter": "part_filter_52", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.028, + }, + "nvlm/wdai/data/augmentations/financeqa": { + "sample_loader": "sample_loader_53", + "part_filter": "part_filter_53", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.005, + }, + "nvlm/wdai/data/augmentations/docreason": { + "sample_loader": "sample_loader_54", + "part_filter": "part_filter_54", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.004, + }, + "nvlm/wdai/data/augmentations/gpt_mtwi": { + "sample_loader": "sample_loader_55", + "part_filter": "part_filter_55", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.005, + }, + "nvlm/wdai/data/augmentations/geos_gpt": { + "sample_loader": "sample_loader_56", + "part_filter": "part_filter_56", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.0001, + }, + "nvlm/wdai/data/augmentations/cauldron_vistext": { + "sample_loader": "sample_loader_57", + "part_filter": "part_filter_57", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.005, + }, + "nvlm/wdai/data/augmentations/memes": { + "sample_loader": "sample_loader_58", + "part_filter": "part_filter_58", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.005, + }, + "nvlm/wdai/data/augmentations/gpt_roadtext": { + "sample_loader": "sample_loader_59", + "part_filter": "part_filter_59", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.0002, + }, + "nvlm/wdai/data/augmentations/indoor_qa": { + "sample_loader": "sample_loader_60", + "part_filter": "part_filter_60", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.001, + }, + "nvlm/wdai/data/augmentations/colpali": { + "sample_loader": "sample_loader_61", + "part_filter": "part_filter_61", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.007, + }, + "nvlm/wdai/data/augmentations/pmc_vqa": { + "sample_loader": "sample_loader_62", + "part_filter": "part_filter_62", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.01, + }, + "nvlm/wdai/data/augmentations/pathvqa": { + "sample_loader": "sample_loader_63", + "part_filter": "part_filter_63", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.004, + }, + "nvlm/wdai/data/augmentations/sciqa": { + "sample_loader": "sample_loader_64", + "part_filter": "part_filter_64", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.027, + }, + "nvlm/wdai/data/augmentations/chinese_meme": { + "sample_loader": "sample_loader_65", + "part_filter": "part_filter_65", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.001, + }, + "nvlm/wdai/data/augmentations/gpt_hiertext": { + "sample_loader": "sample_loader_66", + "part_filter": "part_filter_66", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.003, + }, + "nvlm/wdai/data/augmentations/cauldron_cocoqa": { + "sample_loader": "sample_loader_67", + "part_filter": "part_filter_67", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.007, + }, + "nvlm/wdai/data/cmm-math/processed": { + "sample_loader": "sample_loader_68", + "part_filter": "part_filter_68", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.001, + }, + "nvlm/wdai/data/mmtab/processed": { + "sample_loader": "sample_loader_69", + "part_filter": "part_filter_69", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.008, + }, + "nvlm/wdai/data/simchart9k/processed": { + "sample_loader": "sample_loader_70", + "part_filter": "part_filter_70", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.001, + }, + "nvlm/wdai/data/llava-onevision/mapqa_processed": { + "sample_loader": "sample_loader_71", + "part_filter": "part_filter_71", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.005, + }, + "nvlm/wdai/data/llava-onevision/vizwiz_processed": { + "sample_loader": "sample_loader_72", + "part_filter": "part_filter_72", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.002, + }, + "nvlm/wdai/data/augmentations/gpt_infovqa": { + "sample_loader": "sample_loader_73", + "part_filter": "part_filter_73", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.001, + }, + "nvlm/wdai/data/augmentations/viquae": { + "sample_loader": "sample_loader_74", + "part_filter": "part_filter_74", + "data_class": "SimilarityInterleavedWebdataset", + "data_weight": 0.0005, + }, + "captioning/ccs_recaptioned/webdataset": { + "sample_loader": "sample_loader_75", + "part_filter": "part_filter_75", + "data_class": "CaptioningWebdataset", + "data_weight": 0.2, + }, + "captioning/laion115m-clean": { + "sample_loader": "sample_loader_76", + "part_filter": "part_filter_76", + "data_class": "CaptioningWebdataset", + "data_weight": 0.579, + }, + "nvlm/wdai/data/dvqa_full/processed_pt": { + "sample_loader": "sample_loader_77", + "part_filter": "part_filter_77", + "data_class": "VQAWebdataset", + "data_weight": 0.02, + }, + "nvlm/wdai/data/docmatix/processed_pt": { + "sample_loader": "sample_loader_78", + "part_filter": "part_filter_78", + "data_class": "VQAWebdataset", + "data_weight": 0.02, + }, + "vqa/VQAv2/stage1": { + "sample_loader": "sample_loader_91", + "part_filter": "part_filter_91", + "data_class": "VQAWebdataset", + "data_weight": 1.0, + }, + "vqa/Visual_Genome": { + "sample_loader": "sample_loader_80", + "part_filter": "part_filter_80", + "data_class": "VQAWebdataset", + "data_weight": 0.01, + }, + "nvlm/wdai/data/pdfa-eng-wds/processed_word_len_300": { + "sample_loader": "sample_loader_81", + "part_filter": "part_filter_81", + "data_class": "OCRWebdataset", + "data_weight": 0.08, + }, + "nvlm/wdai/data/textocr/processed": { + "sample_loader": "sample_loader_82", + "part_filter": "part_filter_82", + "data_class": "OCRWebdataset", + "data_weight": 0.02, + }, + "nvlm/wdai/data/coco-text/processed": { + "sample_loader": "sample_loader_83", + "part_filter": "part_filter_83", + "data_class": "OCRWebdataset", + "data_weight": 0.002, + }, + "nvlm/wdai/data/ArT/processed": { + "sample_loader": "sample_loader_84", + "part_filter": "part_filter_84", + "data_class": "OCRWebdataset", + "data_weight": 0.001, + }, + "nvlm/wdai/data/ReCTs/processed": { + "sample_loader": "sample_loader_85", + "part_filter": "part_filter_85", + "data_class": "OCRWebdataset", + "data_weight": 0.001, + }, + "nvlm/wdai/data/lsvt/processed": { + "sample_loader": "sample_loader_86", + "part_filter": "part_filter_86", + "data_class": "OCRWebdataset", + "data_weight": 0.005, + }, + "nvlm/wdai/data/RCTW/processed": { + "sample_loader": "sample_loader_87", + "part_filter": "part_filter_87", + "data_class": "OCRWebdataset", + "data_weight": 0.001, + }, + "nvlm/wdai/data/coco-text/processed_multi": { + "sample_loader": "sample_loader_88", + "part_filter": "part_filter_88", + "data_class": "OCRWebdataset", + "data_weight": 0.0003, + }, + "nvlm/wdai/data/textocr/processed_multi": { + "sample_loader": "sample_loader_89", + "part_filter": "part_filter_89", + "data_class": "OCRWebdataset", + "data_weight": 0.0004, + }, + "nvlm/wdai/data/ReCTs/processed_multi": { + "sample_loader": "sample_loader_90", + "part_filter": "part_filter_90", + "data_class": "OCRWebdataset", + "data_weight": 0.0003, + }, +} + + +def get_sample_loader(path): + """Returns the correct sample_loader function for a dataset.""" + if path not in dataset_loader_mapping: + path = data_path_mapping(path) + assert path in dataset_loader_mapping, f"path {path} not in dataset_loader_mapping" + return globals().get(dataset_loader_mapping.get(path, {}).get("sample_loader")) + + +def get_part_filter(path): + """Returns the correct part_filter function for a dataset.""" + if path not in dataset_loader_mapping: + path = data_path_mapping(path) + assert path in dataset_loader_mapping, f"path {path} not in dataset_loader_mapping" + return globals().get(dataset_loader_mapping.get(path, {}).get("part_filter")) + + +def get_data_class(path): + """Returns the correct data_class for a dataset.""" + if path not in dataset_loader_mapping: + path = data_path_mapping(path) + + assert path in dataset_loader_mapping, f"path {path} not in dataset_loader_mapping" + return dataset_loader_mapping[path]["data_class"] diff --git a/cosmos_framework/data/vfm/augmentors/vlm/prompt_format.py b/cosmos_framework/data/vfm/augmentors/vlm/prompt_format.py new file mode 100644 index 0000000..58e41f2 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/prompt_format.py @@ -0,0 +1,121 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Visual-Text Transformations or Augmentations.""" + +import random +from typing import Dict, Literal + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor + +REASONING_SUFFIX = ( + "\nAnswer the question using the following format:\n\n" + "\nYour reasoning.\n\n\n" + "Write your final answer immediately after the tag." +) + + +class PromptFormat(Augmentor): + def __init__( + self, + input_keys: list = ["texts"], + text_chat_order: Literal["text_end", "text_start", "random"] = "text_end", + ) -> None: + """ + Args: + input_keys (list): List of input keys. + text_chat_order (Literal["text_end", "text_start", "random"]): Order of text items in user messages. + """ + self.input_keys = input_keys + self.text_chat_order = text_chat_order + + def __call__(self, data_dict: Dict) -> Dict: + conversation_key = self.input_keys[0] + + # retrive conversations from dict + try: + list_of_conversation = data_dict[conversation_key] + except KeyError: + url = data_dict["__url__"].root + "/" + data_dict["__url__"].path + print(f"KeyError: {conversation_key} not found in data_dict for url: {url}") + return None + + # check if this is list of list of dict or list of dict + + if isinstance(list_of_conversation[0], list): + selected_conversation = random.sample(list_of_conversation, 1)[0] + elif isinstance(list_of_conversation[0], dict): + + selected_conversation = list_of_conversation + else: + raise ValueError( + f"list_of_conversation is not a list of list of dict or list of dict: {list_of_conversation}" + ) + + # Now it should be list of dict + assert isinstance(selected_conversation, list) and isinstance(selected_conversation[0], dict), ( + f"selected_conversation is not a list of dict: {selected_conversation}" + ) + # Normalize all string content to list format + for message in selected_conversation: + if "content" in message and isinstance(message["content"], str): + message["content"] = [{"type": "text", "text": message["content"]}] + if "reasoning_content" in message and isinstance(message["reasoning_content"], str): + message["reasoning_content"] = [{"type": "text", "text": message["reasoning_content"]}] + + # Merge reasoning_content into assistant message content + for i, message in enumerate(selected_conversation): + if message.get("role") == "assistant" and message.get("reasoning_content"): + # Append reasoning instruction to the preceding user message + for j in range(i - 1, -1, -1): + if selected_conversation[j].get("role") == "user": + selected_conversation[j]["content"].append({"type": "text", "text": REASONING_SUFFIX}) + break + # Wrap reasoning items in ... tags + reasoning_items = message["reasoning_content"] + think_start = [{"type": "text", "text": "\n"}] + think_end = [{"type": "text", "text": "\n\n\n"}] + message["content"] = think_start + reasoning_items + think_end + message["content"] + del message["reasoning_content"] + + data_dict["conversation"] = selected_conversation + + del data_dict[conversation_key] + + + # # enforce chat order + # self._enforce_text_chat_order(selected_conversation) + + return data_dict + + def _enforce_text_chat_order(self, conversation: list) -> None: + """ + Reorder text content within user messages based on text_chat_order setting. + NOTE (maxzhaoshuol): this does NOT work for interleaved data!!!!!! + + Args: + conversation: List of message dictionaries + """ + for message in conversation: + if message.get("role") == "user" and "content" in message: + content = message["content"] + if isinstance(content, list): + # Separate text items from non-text items + text_items = [item for item in content if item.get("type") == "text"] + non_text_items = [item for item in content if item.get("type") != "text"] + + if text_items: + # Reorder based on text_chat_order + if self.text_chat_order == "text_start": + # Put text items at the beginning + message["content"] = text_items + non_text_items + elif self.text_chat_order == "text_end": + # Put text items at the end + message["content"] = non_text_items + text_items + elif self.text_chat_order == "random": + print("random") + # Randomly put text items at beginning or end + if random.random() < 0.5: + message["content"] = text_items + non_text_items + else: + message["content"] = non_text_items + text_items diff --git a/cosmos_framework/data/vfm/augmentors/vlm/shuffle_text_media_order.py b/cosmos_framework/data/vfm/augmentors/vlm/shuffle_text_media_order.py new file mode 100644 index 0000000..d56d315 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/shuffle_text_media_order.py @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +""" +Augmentations to randomly swap media/text order in user prompts if there is only one video/image and one text messeage. +Default swap probability is 1%. +""" + +import random +from typing import Dict, Optional + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log + + +class ShuffleTextMediaOrder(Augmentor): + def __init__( + self, + shuffle_ratio: float = 0.01, + ) -> None: + """ + Args: + input_keys (list): List of input keys. + """ + self.shuffle_ratio = shuffle_ratio + + def __call__(self, data_dict: Dict) -> Optional[Dict]: + url = data_dict["__url__"] + try: + # process conversation + conversation = data_dict["conversation"] + for item in conversation: + if item["role"] == "user": + if ( + len(item["content"]) == 2 + and item["content"][0]["type"] in ["video", "image"] + and item["content"][1]["type"] == "text" + ): + # random.shuffle(item["content"]) # randomly shuffle media and text + if random.random() < self.shuffle_ratio: + item["content"] = item["content"][::-1] + data_dict["conversation"] = conversation + return data_dict + except Exception as e: + log.warning( + f"Error replacing invalid characters in RFT: {e}. Skipping this sample {url.root} {data_dict['__key__']}." + ) + return None diff --git a/cosmos_framework/data/vfm/augmentors/vlm/timestamp.py b/cosmos_framework/data/vfm/augmentors/vlm/timestamp.py new file mode 100644 index 0000000..6ba5496 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/timestamp.py @@ -0,0 +1,518 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Augmentors for general video dense caption datasets. +Copied from projects/cosmos/reason1/datasets/augmentors/timestamp.py +Changes: + 1. Unify system prompt to 'You are a helpful assistant.' + 2. Move task requirements from system prompts to user prompts. + 3. Randomly change timestamp formats from ["seconds", "hh:mm:ss", "hh:mm:ss.sss", "mm:ss.sss"] + 4. Add json output format for event temporal localization. +""" + +import json +import random +from copy import deepcopy +from typing import Dict, List, Literal, Tuple + +from PIL import Image, ImageDraw, ImageFont + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log + + +def compute_timestamps(frame_index: int, fps: float, processor) -> float: + if processor is not None and "Qwen3" in processor.name: + frame_index_start = frame_index // processor.merge_size * processor.merge_size + frame_index_end = frame_index_start + processor.merge_size - 1 + timestamps_start = frame_index_start / fps + timestamps_end = frame_index_end / fps + timestamps = (timestamps_start + timestamps_end) / 2 + timestamps = float(f"{timestamps:.1f}") + return timestamps + else: + return frame_index / fps + + +def convert_timestamp(seconds: float | str, format: str = "hh:mm:ss") -> str: + if isinstance(seconds, str): + seconds = float(seconds) + # convert seconds to hh:mm:ss.sss format + minutes = int(seconds // 60) + hours = int(minutes // 60) + minutes = minutes % 60 + seconds = seconds % 60 + if format == "hh:mm:ss": + return f"{hours:02d}:{minutes:02d}:{int(seconds):02d}" + elif format == "hh:mm:ss.sss": + return f"{hours:02d}:{minutes:02d}:{seconds:06.3f}" + elif format == "mm:ss.sss": + return f"{minutes + 60 * hours:02d}:{seconds:06.3f}" + else: + raise ValueError(f"Invalid format: {format}") + + +def check_if_need_overlay_text(processor): + if processor is not None and ("Qwen3" in processor.name or "Nemotron" in processor.name): + return False + return True + + +timestamp_convertor = { + "seconds": lambda x: x, + "hh:mm:ss": lambda x: convert_timestamp(x, format="hh:mm:ss"), + "hh:mm:ss.sss": lambda x: convert_timestamp(x, format="hh:mm:ss.sss"), + "mm:ss.sss": lambda x: convert_timestamp(x, format="mm:ss.sss"), +} + + +def overlay_text( + images: List[Image.Image], + fps: float, + border_height: int = 28, # this is due to patch size of 28 + temporal_path_size: int = 2, # Number of positions to cycle through + font_size: int = 20, + font_color: str = "white", + processor=None, + debug=False, +) -> Tuple[List[Image.Image], List[float]]: + """ + Overlay text on a list of PIL images with black border. + The timestamp position cycles through available positions. + + Args: + images: List of PIL images to process + fps: Frames per second + border_height: Height of the black border in pixels (default: 28) + temporal_path_size: Number of positions to cycle through (default: 2) + font_size: Font size for the text (default: 20) + font_color: Color of the text (default: "white") + + Returns: + List of PIL images with text overlay + List of timestamps + """ + if not check_if_need_overlay_text(processor) and not debug: + # if debug is True, we still need to overlay text for visualization purpose + return images, [compute_timestamps(i, fps, processor) for i in range(len(images))] + + # Try to use DejaVu Sans Mono font for better readability + font = ImageFont.truetype("/usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf", font_size) + + # Process each image + processed_images = [] + + for i, image in enumerate(images): + # Get original dimensions + width, height = image.size + + # Create new image with black border at the bottom + new_height = height + border_height + if debug: # add border_height for visualization purpose + new_height = new_height + border_height + new_image = Image.new("RGB", (width, new_height), color="black") + + # Paste original image at the top + new_image.paste(image, (0, 0)) + + # Draw text on the black border + draw = ImageDraw.Draw(new_image) + + # Calculate timestamp for current frame + total_seconds = compute_timestamps(i, fps, processor) + text = f"{total_seconds:.2f}s" + + # Get text dimensions + try: + # Get text bounding box + bbox = draw.textbbox((0, 0), text, font=font) + text_width = bbox[2] - bbox[0] + text_height = bbox[3] - bbox[1] + except AttributeError: + # Fallback for older PIL versions + text_width, text_height = draw.textsize(text, font=font) + + # Define available positions (cycling through horizontal positions) + position_idx = i % temporal_path_size + section_width = width // temporal_path_size + + # Calculate x position based on cycling position + section_center_x = position_idx * section_width + section_width // 2 + text_x = section_center_x - text_width // 2 + + # Ensure text doesn't go outside bounds + text_x = max(0, min(text_x, width - text_width)) + + # Center vertically in the border + text_y = height + (border_height - text_height) // 2 + + # Draw the single timestamp + draw.text((text_x, text_y), text, fill=font_color, font=font) + + processed_images.append(new_image) + + return processed_images, [compute_timestamps(i, fps, processor) for i in range(len(images))] + + +def markdown_to_list(conversation_data: str | List[Dict]) -> List[Dict]: + if isinstance(conversation_data, list): + assert ( + isinstance(conversation_data[0], dict) + and conversation_data[0]["type"] == "text" + and len(conversation_data) == 1 + ) + conversation_data = conversation_data[0]["text"] + cleaned_data = conversation_data.strip() + if cleaned_data.startswith("```json"): + cleaned_data = cleaned_data[7:] # Remove '```json' + if cleaned_data.endswith("```"): + cleaned_data = cleaned_data[:-3] # Remove '```' + cleaned_data = cleaned_data.strip() + return json.loads(cleaned_data) + + +def json_to_markdown(conversation_data: List[Dict] | Dict) -> str: + json_string = json.dumps(conversation_data, indent=2) + return f"```json\n{json_string}\n```".strip() + + +def snap_timestamps_to_existing(assistant_message: List[Dict], existing_timestamps: List[float]) -> List[Dict]: + """ + Snap conversation start/end timestamps to the nearest existing timestamps. + + Args: + assistant_message: JSON string containing list of dictionaries with 'start', 'end', and 'caption' fields + existing_timestamps: List of existing timestamps (floats) to snap to + + Returns: + List of dictionaries with snapped timestamps + """ + snapped_message = [] + + for item in assistant_message: + if not isinstance(item, dict) or "start" not in item or "end" not in item: + raise ValueError("Each item must be a dictionary with 'start' and 'end' fields") + + snapped_item = item.copy() + + # Snap start and end timestamps to existing ones + snapped_item["start"] = min(existing_timestamps, key=lambda x: abs(x - item["start"])) + snapped_item["end"] = min(existing_timestamps, key=lambda x: abs(x - item["end"])) + + snapped_message.append(snapped_item) + + # Sort the merged events by start timestamp to ensure chronological order + # Merge captions that share identical start and end timestamps + merged_events: Dict[Tuple[float, float], Dict] = {} + for item in snapped_message: + item["start"] = round(item["start"], 2) + item["end"] = round(item["end"], 2) + + key = (item["start"], item["end"]) + if key in merged_events: + # Concatenate captions for the same time interval. + merged_events[key]["caption"] = merged_events[key]["caption"].rstrip() + " " + item["caption"].lstrip() + else: + merged_events[key] = item + + merged_events[key]["caption"] = merged_events[key]["caption"].strip() + + # Sort the merged events by start timestamp to ensure chronological order + new_assistant_message = sorted(merged_events.values(), key=lambda x: x["start"]) + if len(new_assistant_message) == 0: + raise ValueError("No valid assistant message found for data.") + + return new_assistant_message + + +def augment_assistant_message( + assistant_message: List[Dict], + output_format: Literal[ + "dense_video_caption_json", + "dense_video_caption_plain", + "dense_video_caption_json_with_types", + "dense_video_caption_plain_with_types", + "temporal_localization_plain", + "temporal_localization_json", + "temporal_caption", + ], + timestamp_format: str = "hh:mm:ss", +): + # change time stamp format to hh:mm:ss.sss + assistant_message = deepcopy(assistant_message) + for item in assistant_message: + item["start"] = timestamp_convertor[timestamp_format](item["start"]) + item["end"] = timestamp_convertor[timestamp_format](item["end"]) + if output_format == "dense_video_caption_json" or output_format == "dense_video_caption_json_with_types": + output_message = json_to_markdown(assistant_message) + return output_message + elif output_format == "dense_video_caption_plain" or output_format == "dense_video_caption_plain_with_types": + output_message = "" + for item in assistant_message: + if output_format == "dense_video_caption_plain": + output_message += f"{item['start']}, {item['end']}, {item['caption']}\n" + elif output_format == "dense_video_caption_plain_with_types": + output_message += f"{item['start']}, {item['end']}, {item['type']}, {item['caption']}\n" + return output_message + elif output_format == "temporal_localization_plain": + return f"{assistant_message[0]['start']}, {assistant_message[0]['end']}" + elif output_format == "temporal_localization_json": + output_message = { + "start": assistant_message[0]["start"], + "end": assistant_message[0]["end"], + } + output_message = json_to_markdown(output_message) + return output_message + elif output_format == "temporal_caption": + return assistant_message[0]["caption"] + else: + raise ValueError(f"Invalid output format: {output_format}") + + +def augment_user_prompt( + assistant_message: List[dict], + output_format: Literal[ + "dense_video_caption_json", + "dense_video_caption_plain", + "dense_video_caption_json_with_types", + "dense_video_caption_plain_with_types", + "temporal_localization_plain", + "temporal_localization_json", + "temporal_caption", + ], + timestamp_format: str = "hh:mm:ss", +): + if ( + output_format == "dense_video_caption_json" + or output_format == "dense_video_caption_plain" + or output_format == "dense_video_caption_json_with_types" + or output_format == "dense_video_caption_plain_with_types" + ): + if random.random() < 0.5: + user_prompt = random.choice( + [ + "Caption the notable events in the provided video.", + "Describe the notable events in the provided video.", + "Summarize the notable events in the provided video.", + "Localize a series of activity events in the video, output the start and end timestamp and description for each event.", + ] + ) + if random.random() < 0.5: + user_prompt = "Please " + user_prompt.lower() + else: + user_prompt = random.choice( + [ + "Can you caption the notable events in the provided video?", + "Can you describe the notable events in the provided video?", + "Can you summarize the notable events in the provided video?", + ] + ) + if output_format == "dense_video_caption_json": + # add format requirement. + if random.random() < 0.5: + user_prompt = user_prompt + ( + "\nPlease provide captions of all the events in the video with timestamps using the following format:\n" + "[\n" + " {\n" + f' "start": {timestamp_format},\n' + f' "end": {timestamp_format},\n' + ' "caption": \n' + " },\n" + " {\n" + f' "start": {timestamp_format},\n' + f' "end": {timestamp_format},\n' + ' "caption": \n' + " }\n" + "]" + ) + else: + user_prompt = ( + user_prompt + + f"\nProvide the result in json format with '{timestamp_format}' for time depiction for each event. Use keywords 'start', 'end' and 'caption' in the json output." + ) + elif output_format == "dense_video_caption_json_with_types": + # add format requirement. + if random.random() < 0.5: + user_prompt = user_prompt + ( + "\nPlease provide captions of all the events in the video with timestamps using the following format:\n" + "[\n" + " {\n" + f' "start": {timestamp_format},\n' + f' "end": {timestamp_format},\n' + ' "type": \n' + ' "caption": \n' + " },\n" + " {\n" + f' "start": {timestamp_format},\n' + f' "end": {timestamp_format},\n' + ' "type": \n' + ' "caption": \n' + " }\n" + "]" + ) + else: + user_prompt = ( + user_prompt + + f"\nProvide the result in json format with '{timestamp_format}' for time depiction for each event. Use keywords 'start', 'end', 'type' and 'caption' in the json output." + ) + elif output_format == "dense_video_caption_plain_with_types": + user_prompt = ( + user_prompt + + f"\nPlease provide captions of all the events in the video with start and end timestamps using the following format: \n{timestamp_format}, {timestamp_format}, , .\n{timestamp_format}, {timestamp_format}, , ." + ) + else: # plain format + user_prompt = ( + user_prompt + + f"\nPlease provide captions of all the events in the video with start and end timestamps using the following format: \n{timestamp_format}, {timestamp_format}, caption of event 1.\n{timestamp_format}, {timestamp_format}, caption of event 2." + ) + + elif output_format == "temporal_localization_plain" or output_format == "temporal_localization_json": + event = assistant_message[0] + event_caption = event["caption"] + if not event_caption[-1].isalpha(): + event_caption = event_caption[:-1] + user_prompt = random.choice( + [ + f"When does the following event happen? {event_caption}.", + f"When does the event '{event_caption.lower()}' happen?", + f"Can you find the event '{event_caption.lower()}'?", + ] + ) + if output_format == "temporal_localization_json": + user_prompt = ( + user_prompt + + f"\nPlease provide the result in json format with '{timestamp_format}' for time depiction for the event. Use keywords 'start', 'end' in the json output." + ) + else: + user_prompt = ( + user_prompt + + f"\nPlease provide the start and end timestamp of the event in the following format: {timestamp_format}, {timestamp_format}." + ) + + elif output_format == "temporal_caption": + event = assistant_message[0] + if random.random() < 0.333333: + + start = round(event["start"]) + end = round(event["end"]) + elif random.random() < 0.666666: + + start = round(event["start"] * 2) / 2 + end = round(event["end"] * 2) / 2 + else: + start = event["start"] + end = event["end"] + if start == end: # HACK (maxzhaoshuol): remove events with start == end + raise ValueError("Start and end time are the same for data.") + if timestamp_format == "seconds": + if random.random() < 0.5: + start = f"{start}s" + end = f"{end}s" + else: + start = f"{start} seconds" + end = f"{end} seconds" + else: + start = convert_timestamp(start, format=timestamp_format) + end = convert_timestamp(end, format=timestamp_format) + user_prompt = random.choice( + [ + f"Caption the event between {start} and {end}.", + f"Please describe the event between {start} and {end}.", + f"Please caption the event between the start time {start} and the end time {end}.", + f"Summarize the event between {start} and {end}.", + ] + ) + return user_prompt + + +class TimeStamp(Augmentor): + def __init__( + self, + input_key: list = "media", + output_format: Literal[ + "dense_video_caption", "temporal_localization", "temporal_caption", "caption", "random" + ] = "dense_video_caption", + urls_needs_timestamp: list = ["av_reasoning_localization_20250627", "tl_activitynet_20250630"], + processor=None, + ) -> None: + """ + Args: + input_keys (list): List of input keys. + """ + self.input_key = input_key + self.output_format = output_format + self.urls_needs_timestamp = urls_needs_timestamp + self.processor = processor + + def __call__(self, data_dict: Dict) -> Dict: + url = data_dict["__url__"] + if not any(url_pattern in url.root for url_pattern in self.urls_needs_timestamp): + return data_dict + + media_data = data_dict[self.input_key] + for k, v in media_data.items(): + if "video" in k: + video_frames_with_timestamp, timestamps = overlay_text(v["videos"], v["fps"], processor=self.processor) + media_data[k]["videos"] = video_frames_with_timestamp + + if self.output_format == "random": + output_format = random.choice(["dense_video_caption", "temporal_localization", "temporal_caption"]) + elif self.output_format == "caption": + output_format = random.choice(["dense_video_caption", "temporal_caption"]) + else: + output_format = self.output_format + + if output_format == "dense_video_caption": + output_format = random.choice( + [ + "dense_video_caption_json", + "dense_video_caption_plain", + "dense_video_caption_json_with_types", + "dense_video_caption_plain_with_types", + ] + ) + elif output_format == "temporal_localization": + output_format = random.choice(["temporal_localization_plain", "temporal_localization_json"]) + + timestamp_format = random.choice(list[str](timestamp_convertor.keys())) + + try: + # find the assistant message and parse into a list of dictionaries + for item in data_dict["conversation"]: + if item["role"] == "assistant": + if isinstance(item["content"], list): + assert len(item["content"]) == 1 + assert item["content"][0]["type"] == "text" + item["content"] = item["content"][0]["text"] + assistant_message = markdown_to_list(item["content"]) + assistant_message = snap_timestamps_to_existing(assistant_message, timestamps) + break + + if "types" in output_format: + for item in assistant_message: + if "type" not in item: # if type is not provided, use the default format + output_format = random.choice(["dense_video_caption_json", "dense_video_caption_plain"]) + break + + # if temporal localization or caption, sample one event + if output_format in ["temporal_localization_plain", "temporal_localization_json", "temporal_caption"]: + assistant_message = random.sample(assistant_message, 1) + + # process conversation + conversation = data_dict["conversation"] + for item in conversation: + if item["role"] == "system": + item["content"] = "You are a helpful assistant." + elif item["role"] == "user": + for content in item["content"]: + if content["type"] == "text": + content["text"] = augment_user_prompt(assistant_message, output_format, timestamp_format) + elif item["role"] == "assistant": + assistant_message = augment_assistant_message(assistant_message, output_format, timestamp_format) + item["content"] = assistant_message + data_dict["conversation"] = conversation + + return data_dict + except Exception as e: + log.warning(f"Error timestamping: {e}. Skipping this sample {url.root} {data_dict['__key__']}.") + return None diff --git a/cosmos_framework/data/vfm/augmentors/vlm/timestamp_with_subject_tracking.py b/cosmos_framework/data/vfm/augmentors/vlm/timestamp_with_subject_tracking.py new file mode 100644 index 0000000..bf9b446 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/timestamp_with_subject_tracking.py @@ -0,0 +1,430 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Augmentors for facebook/PLM-Video-Human dataset (video dense captions with SoM prompting). +Copied from projects/cosmos/reason1/datasets/augmentors/timestamp_with_subject_tracking.py +Changes: + 1. Unify system prompt to 'You are a helpful assistant.' + 2. Move task requirements from system prompts to user prompts. + 3. Randomly change timestamp formats from ["seconds", "hh:mm:ss", "hh:mm:ss.sss", "mm:ss.sss"] + 4. Add json output format for event temporal localization. +""" + +import random +from copy import deepcopy +from typing import Dict, List, Literal + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.augmentors.vlm.timestamp import ( + json_to_markdown, + markdown_to_list, + overlay_text, + timestamp_convertor, +) + + +# reorder dict entries +def reorder_dict_entries(conversation_data: Dict) -> Dict: + key_order = ["subject_id", "start", "end", "caption"] + output_dict = {} + for key in key_order: + if key in conversation_data: + output_dict[key] = conversation_data[key] + return output_dict + + +def snap_timestamps_to_existing(assistant_message: List[Dict], existing_timestamps: List[float]) -> List[Dict]: + """ + Snap conversation start/end timestamps to the nearest existing timestamps. + + Args: + assistant_message: JSON string containing list of dictionaries with 'start', 'end', and 'caption' fields + existing_timestamps: List of existing timestamps (floats) to snap to + + Returns: + List of dictionaries with snapped timestamps + """ + snapped_message = [] + + for item in assistant_message: + """ + { + "subject_id": "0", + "start": 0.0, + "end": 12.17, + "caption": "Woman is making garland from flowers beading by her hands." + } + """ + if not isinstance(item, dict) or "start" not in item or "end" not in item or "subject_id" not in item: + log.warning(f"Each item must be a dictionary with 'start', 'end', and 'subject_id' fields. getting {item}") + return None + + snapped_item = {"subject_id": item["subject_id"], "caption": item["caption"]} + + # Snap start and end timestamps to existing ones + snapped_item["start"] = min(existing_timestamps, key=lambda x: abs(x - item["start"])) + snapped_item["end"] = min(existing_timestamps, key=lambda x: abs(x - item["end"])) + + # round to 2 decimal places + snapped_item["start"] = round(snapped_item["start"], 2) + snapped_item["end"] = round(snapped_item["end"], 2) + + snapped_message.append(snapped_item) + + # Sort the merged events by start timestamp to ensure chronological order + new_assistant_message = sorted(snapped_message, key=lambda x: x["start"]) + if len(new_assistant_message) == 0: + log.warning("No valid assistant message found for data.") + return None + + return new_assistant_message + + +def augment_assistant_message( + assistant_message: List[Dict], + output_format: Literal[ + "dense_video_caption_json_per_subject", + "dense_video_caption_plain_per_subject", + "dense_video_caption_json_one_subject", + "dense_video_caption_plain_one_subject", + "temporal_location_subject_plain", + "temporal_location_subject_json", + "temporal_caption_subject", + ], + timestamp_format: str = "hh:mm:ss", +): + # change time stamp format to hh:mm:ss.sss + assistant_message = deepcopy(assistant_message) + for item in assistant_message: + item["start"] = timestamp_convertor[timestamp_format](item["start"]) + item["end"] = timestamp_convertor[timestamp_format](item["end"]) + + if output_format == "dense_video_caption_json_per_subject": + output_message = json_to_markdown(assistant_message) + return output_message + elif output_format == "dense_video_caption_plain_per_subject": + output_message = "" + for item in assistant_message: + output_message += f"Subject {item['subject_id']}, {item['start']}, {item['end']}, {item['caption']}\n" + return output_message + + elif output_format == "dense_video_caption_json_one_subject": + # remove subject_id + assistant_message = [ + {"start": item["start"], "end": item["end"], "caption": item["caption"]} for item in assistant_message + ] + output_message = json_to_markdown(assistant_message) + return output_message + elif output_format == "dense_video_caption_plain_one_subject": + output_message = "" + for item in assistant_message: + output_message += f"{item['start']}, {item['end']}, {item['caption']}\n" + return output_message + elif output_format == "temporal_location_subject_plain": + return f"{assistant_message[0]['start']}, {assistant_message[0]['end']}" + elif output_format == "temporal_location_subject_json": + output_message = { + "start": assistant_message[0]["start"], + "end": assistant_message[0]["end"], + } + return json_to_markdown(output_message) + elif output_format == "temporal_caption_subject": + return assistant_message[0]["caption"] + else: + raise ValueError(f"Invalid output format: {output_format}") + + +def augment_user_prompt( + assistant_message: List[dict], + output_format: Literal[ + "dense_video_caption_json_per_subject", + "dense_video_caption_plain_per_subject", + "dense_video_caption_json_one_subject", + "dense_video_caption_plain_one_subject", + "temporal_location_subject_plain", + "temporal_location_subject_json", + "temporal_caption_subject", + ], + timestamp_format: str = "hh:mm:ss", +): + if ( + output_format == "dense_video_caption_json_per_subject" + or output_format == "dense_video_caption_plain_per_subject" + ): + if random.random() < 0.5: + user_prompt = random.choice( + [ + "Caption the notable events in the provided video.", + "Describe the notable events in the provided video.", + "Summarize the notable events in the provided video.", + "Localize a series of activity events in the video, output the start and end timestamp, subject id and description for each event.", + ] + ) + if random.random() < 0.5: + user_prompt = "Please " + user_prompt.lower() + else: + user_prompt = random.choice( + [ + "Can you caption the notable events in the provided video?", + "Can you describe the notable events in the provided video?", + "Can you summarize the notable events in the provided video?", + ] + ) + if output_format == "dense_video_caption_json_per_subject": + if random.random() < 0.5: + user_prompt = user_prompt + ( + "\nList and describe all marked subjects in the video using the following format:\n" + "[\n" + " {\n" + f' "start": {timestamp_format},\n' + f' "end": {timestamp_format},\n' + ' "subject_id": ,\n' + ' "caption": ,\n' + " },\n" + " {\n" + f' "start": {timestamp_format},\n' + f' "end": {timestamp_format},\n' + ' "subject_id": ,\n' + ' "caption": ,\n' + " },\n" + "]" + ) + else: + user_prompt = ( + user_prompt + + f"\nProvide the result in json format with '{timestamp_format}' for time depiction for each event. Use keywords 'start', 'end', 'subject_id' and 'caption' in the json output." + ) + else: # plain format + user_prompt = ( + user_prompt + + f"\nList and describe all marked subjects in the video using the following format: \nSubject , {timestamp_format}, {timestamp_format}, caption of event 1.\nSubject , {timestamp_format}, {timestamp_format}, caption of event 2.\n" + ) + + elif output_format == "temporal_location_subject_plain" or output_format == "temporal_location_subject_json": + event = assistant_message[0] + user_prompt = random.choice( + [ + f"When does the following event happen to the tracked object with ID <{event['subject_id']}>? {event['caption']}", + f"When does the event '{event['caption'].lower()[:-1]}' happen to the tracked object with ID <{event['subject_id']}>?", + f"Can you find the event '{event['caption'].lower()[:-1]}' happen to the tracked object with ID <{event['subject_id']}>?", + ] + ) + if output_format == "temporal_location_subject_plain": + user_prompt = ( + user_prompt + + f"\nPlease provide the start and end timestamp in the following format: {timestamp_format}, {timestamp_format}." + ) + else: + user_prompt = ( + user_prompt + + f"\nPlease provide the result in json format with '{timestamp_format}' for time depiction for the event. Use keywords 'start', 'end' in the json output." + ) + + elif output_format == "temporal_caption_subject": + event = assistant_message[0] + if random.random() < 0.333333: + + start = round(event["start"]) + end = round(event["end"]) + elif random.random() < 0.666666: + + start = round(event["start"] * 2) / 2 + end = round(event["end"] * 2) / 2 + else: + start = event["start"] + end = event["end"] + if start == end: # HACK (maxzhaoshuol): remove events with start == end + log.warning(f"Start and end time are the same for data. {event}") + return None + + if timestamp_format == "seconds": + if random.random() < 0.5: + start = f"{start}s" + end = f"{end}s" + else: + start = f"{start} seconds" + end = f"{end} seconds" + else: + start = timestamp_convertor[timestamp_format](start) + end = timestamp_convertor[timestamp_format](end) + + user_prompt = random.choice( + [ + f"Caption the event between {start} and {end} of the tracked object with ID <{event['subject_id']}>.", + f"Please describe the event between {start} and {end} of the tracked object with ID <{event['subject_id']}>.", + f"Please caption the event between the start time {start} and the end time {end} of the tracked object with ID <{event['subject_id']}>.", + f"Summarize the event between {start} and {end} of the tracked object with ID <{event['subject_id']}>.", + ] + ) + elif ( + output_format == "dense_video_caption_json_one_subject" + or output_format == "dense_video_caption_plain_one_subject" + ): + event = assistant_message[0] + if random.random() < 0.5: + user_prompt = random.choice( + [ + f"Caption the notable events in the provided video for the tracked object with ID <{event['subject_id']}>.", + f"Describe the notable events in the provided video for the tracked object with ID <{event['subject_id']}>.", + f"Summarize the notable events in the provided video for the tracked object with ID <{event['subject_id']}>.", + f"Localize a series of activity events in the video for the the tracked object with ID <{event['subject_id']}>, output the start and end timestamp and description for each event.", + ] + ) + if random.random() < 0.5: + user_prompt = "Please " + user_prompt.lower() + else: + user_prompt = random.choice( + [ + f"Can you caption the notable events in the provided video for the tracked object with ID <{event['subject_id']}>?", + f"Can you describe the notable events in the provided video for the tracked object with ID <{event['subject_id']}>?", + f"Can you summarize the notable events in the provided video for the tracked object with ID <{event['subject_id']}>?", + ] + ) + if output_format == "dense_video_caption_json_one_subject": + if random.random() < 0.5: + user_prompt = user_prompt + ( + f"\nSummarize the notable events of the subject marked with ID <{event['subject_id']}> with timestamps in the video using the following format:\n" + "[\n" + " {\n" + f' "start": {timestamp_format},\n' + f' "end": {timestamp_format},\n' + ' "caption": ,\n' + " },\n" + " {\n" + f' "start": {timestamp_format},\n' + f' "end": {timestamp_format},\n' + ' "caption": ,\n' + " },\n" + "]" + ) + else: + user_prompt = ( + user_prompt + + f"\nProvide the result in json format with '{timestamp_format}' for time depiction for each event. Use keywords 'start', 'end' and 'caption' in the json output." + ) + else: # plain format + user_prompt = ( + user_prompt + + f"\nPlease provide captions of all the events of the tracked object with given ID in the video with start and end timestamps using the following format:\n{timestamp_format}, {timestamp_format}, caption of event 1.\n{timestamp_format}, {timestamp_format}, caption of event 2.\n" + ) + return user_prompt + + +class TimeStampWithSubjectTracking(Augmentor): + def __init__( + self, + input_key: str = "media", + output_format: Literal[ + "dense_video_caption_per_subject", + "dense_video_caption_one_subject", + "temporal_location_subject", + "temporal_caption_subject", + "random", + ] = "random", + urls_needs_timestamp: list = ["tl_plm_sav_20250714"], + processor=None, + ) -> None: + """ + Args: + input_keys (list): List of input keys. + """ + self.input_key = input_key + self.output_format = output_format + self.urls_needs_timestamp = urls_needs_timestamp + self.processor = processor + + def __call__(self, data_dict: Dict) -> Dict: + url = data_dict["__url__"] + if not any(url_pattern in url.root for url_pattern in self.urls_needs_timestamp): + return data_dict + + media_data = data_dict[self.input_key] + for k, v in media_data.items(): + if "video" in k: + video_frames_with_timestamp, timestamps = overlay_text(v["videos"], v["fps"], processor=self.processor) + media_data[k]["videos"] = video_frames_with_timestamp + + if self.output_format == "random": + output_format = random.choice( + [ + "dense_video_caption_per_subject", + "dense_video_caption_one_subject", + "temporal_location_subject", + "temporal_caption_subject", + ] + ) + else: + output_format = self.output_format + + if output_format == "dense_video_caption_per_subject": + output_format = random.choice( + ["dense_video_caption_json_per_subject", "dense_video_caption_plain_per_subject"] + ) + elif output_format == "dense_video_caption_one_subject": + output_format = random.choice( + ["dense_video_caption_json_one_subject", "dense_video_caption_plain_one_subject"] + ) + elif output_format == "temporal_location_subject": + output_format = random.choice(["temporal_location_subject_plain", "temporal_location_subject_json"]) + + # find the assistant message and parse into a list of dictionaries + for item in data_dict["conversation"]: + if item["role"] == "assistant": + """ + content dict: + ```json + [ + { + "subject_id": "0", + "start": 10.67, + "end": 11.17, + "caption": "A person enters the frame from the left riding a bike on the road towards the right frame wearing a yellow helmet." + } + ] + ``` + """ + assistant_message = markdown_to_list(item["content"]) + assistant_message = snap_timestamps_to_existing(assistant_message, timestamps) + if assistant_message is None: + return None # skip this sample + break + + # if temporal localization or caption, sample one event + if output_format in [ + "temporal_location_subject_plain", + "temporal_location_subject_json", + "temporal_caption_subject", + ]: + assistant_message = random.sample(assistant_message, 1) + elif output_format in ["dense_video_caption_json_one_subject", "dense_video_caption_plain_one_subject"]: + available_subject_ids = list( + set([assistant_message_i["subject_id"] for assistant_message_i in assistant_message]) + ) + # sample one subject id + subject_id = random.choice(available_subject_ids) + assistant_message = [ + assistant_message_i + for assistant_message_i in assistant_message + if assistant_message_i["subject_id"] == subject_id + ] + + timestamp_format = random.choice(list[str](timestamp_convertor.keys())) + + # process conversation + conversation = data_dict["conversation"] + for item in conversation: + if item["role"] == "system": + item["content"] = "You are a helpful assistant." + elif item["role"] == "user": + for content in item["content"]: + if content["type"] == "text": + content["text"] = augment_user_prompt(assistant_message, output_format, timestamp_format) + if content["text"] is None: # parse error + return None + elif item["role"] == "assistant": + assistant_message = augment_assistant_message(assistant_message, output_format, timestamp_format) + item["content"] = assistant_message + data_dict["conversation"] = conversation + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/vlm/timestamp_without_augment_message.py b/cosmos_framework/data/vfm/augmentors/vlm/timestamp_without_augment_message.py new file mode 100644 index 0000000..b0ef17e --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/timestamp_without_augment_message.py @@ -0,0 +1,214 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +""" +Copied from projects/cosmos/reason1/datasets/augmentors/timestamp_without_augment_message.py +Changes: + - overlay_text is now imported from cosmos3. +""" + +import json +import random +import re +from typing import Dict, List, Literal, Tuple + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.data.vfm.augmentors.vlm.timestamp import overlay_text + + +def list_to_markdown(conversation_data: List[Dict]) -> str: + json_string = json.dumps(conversation_data, indent=2) + return f"```json\n{json_string}\n```".strip() + + +def snap_timestamps_to_existing(assistant_message: List[Dict], existing_timestamps: List[float]) -> List[Dict]: + """ + Snap conversation start/end timestamps to the nearest existing timestamps. + + Args: + assistant_message: JSON string containing list of dictionaries with 'start', 'end', and 'caption' fields + existing_timestamps: List of existing timestamps (floats) to snap to + + Returns: + List of dictionaries with snapped timestamps + """ + snapped_message = [] + + for item in assistant_message: + if not isinstance(item, dict) or "start" not in item or "end" not in item: + raise ValueError("Each item must be a dictionary with 'start' and 'end' fields") + + snapped_item = item.copy() + + # Snap start and end timestamps to existing ones + snapped_item["start"] = min(existing_timestamps, key=lambda x: abs(x - item["start"])) + snapped_item["end"] = min(existing_timestamps, key=lambda x: abs(x - item["end"])) + + snapped_message.append(snapped_item) + + # Sort the merged events by start timestamp to ensure chronological order + # Merge captions that share identical start and end timestamps + merged_events: Dict[Tuple[float, float], Dict] = {} + for item in snapped_message: + item["start"] = round(item["start"], 2) + item["end"] = round(item["end"], 2) + + key = (item["start"], item["end"]) + if key in merged_events: + # Concatenate captions for the same time interval. + merged_events[key]["caption"] = merged_events[key]["caption"].rstrip() + " " + item["caption"].lstrip() + else: + merged_events[key] = item + + merged_events[key]["caption"] = merged_events[key]["caption"].strip() + + # Sort the merged events by start timestamp to ensure chronological order + new_assistant_message = sorted(merged_events.values(), key=lambda x: x["start"]) + if len(new_assistant_message) == 0: + raise ValueError("No valid assistant message found for data.") + + return new_assistant_message + + +def augment_assistant_message( + assistant_message: List[Dict], + output_format: Literal[ + "dense_video_caption_json", "dense_video_caption_plain", "temporal_localization", "temporal_caption" + ], +): + if output_format == "dense_video_caption_json": + output_message = list_to_markdown(assistant_message) + return output_message + elif output_format == "dense_video_caption_plain": + output_message = "" + for item in assistant_message: + output_message += f"<{item['start']}> <{item['end']}> {item['caption']}\n" + return output_message + elif output_format == "temporal_localization": + return f"<{assistant_message[0]['start']}> <{assistant_message[0]['end']}>" + elif output_format == "temporal_caption": + return assistant_message[0]["caption"] + else: + raise ValueError(f"Invalid output format: {output_format}") + + +def augment_system_prompt( + system_prompt: str, + output_format: Literal[ + "dense_video_caption_json", "dense_video_caption_plain", "temporal_localization", "temporal_caption" + ], + need_overlay_text=True, +): + if output_format == "dense_video_caption_json": + system_prompt = system_prompt + elif output_format == "dense_video_caption_plain": + system_prompt = re.sub(r"Please.*?\]", "", system_prompt, flags=re.DOTALL) # strip off existing format + system_prompt += """Please provide captions of all the events in the video with timestamps using the following format: + caption of event 1.\n caption of event 2.\n""" + elif output_format == "temporal_localization": + system_prompt = re.sub(r"Please.*?\]", "", system_prompt, flags=re.DOTALL) # strip off existing format + system_prompt += "Please locate the start and end time of a given event specified by the user using the following format: ." + elif output_format == "temporal_caption": + system_prompt = re.sub(r"Please.*?\]", "", system_prompt, flags=re.DOTALL) # strip off existing format + system_prompt += "Please provide a caption of the duration in the video based on the start and end time specified by the user." + else: + raise ValueError(f"Invalid output format: {output_format}") + + if need_overlay_text: + system_prompt = ( + system_prompt + + "\nAt each frame, the timestamp is embedded at the bottom of the video. You need to extract the timestamp and answer the user question." + ) + else: + system_prompt = system_prompt + "\nYou need to extract the timestamp and answer the user question." + + return system_prompt + + +def augment_user_prompt( + assistant_message: List[dict], + output_format: Literal[ + "dense_video_caption_json", "dense_video_caption_plain", "temporal_localization", "temporal_caption" + ], +): + if output_format == "dense_video_caption_json" or output_format == "dense_video_caption_plain": + if random.random() < 0.5: + user_prompt = random.choice( + [ + "Caption the notable events in the provided video.", + "Describe the notable events in the provided video.", + "Summarize the notable events in the provided video.", + ] + ) + if random.random() < 0.5: + user_prompt = "Please " + user_prompt.lower() + else: + user_prompt = random.choice( + [ + "Can you caption the notable events in the provided video?", + "Can you describe the notable events in the provided video?", + "Can you summarize the notable events in the provided video?", + ] + ) + elif output_format == "temporal_localization": + event = assistant_message[0] + user_prompt = random.choice( + [ + f"When does the following event happen? {event['caption']}", + f"When does the event '{event['caption'].lower()[:-1]}' happen?", + f"Can you find the event '{event['caption'].lower()[:-1]}'?", + ] + ) + elif output_format == "temporal_caption": + event = assistant_message[0] + if random.random() < 0.5: + + start = round(event["start"]) + end = round(event["end"]) + else: + + start = round(event["start"] * 2) / 2 + end = round(event["end"] * 2) / 2 + if start == end: # HACK (maxzhaoshuol): remove events with start == end + raise ValueError("Start and end time are the same for data.") + user_prompt = random.choice( + [ + f"Caption the event between {start}s and {end}s.", + f"Please describe the event between {start} and {end}.", + f"Please caption the event between the start time {start}s and the end time {end}s.", + f"Summarize the event between <{start}s, {end}s>.", + ] + ) + return user_prompt + + +class TimeStampWithoutAugmentMessage(Augmentor): + def __init__( + self, + input_key: str = "media", + output_format: Literal[ + "dense_video_caption", "temporal_localization", "temporal_caption", "random" + ] = "dense_video_caption", + urls_needs_timestamp: list = ["av_reasoning_localization_20250627", "tl_activitynet_20250630"], + processor=None, + ) -> None: + """ + Args: + input_keys (list): List of input keys. + """ + self.input_key = input_key + self.output_format = output_format + self.urls_needs_timestamp = urls_needs_timestamp + self.processor = processor + + def __call__(self, data_dict: Dict) -> Dict: + url = data_dict["__url__"] + if not any(url_pattern in url.root for url_pattern in self.urls_needs_timestamp): + return data_dict + + media_data = data_dict[self.input_key] + for k, v in media_data.items(): + if "video" in k: + video_frames_with_timestamp, timestamps = overlay_text(v["videos"], v["fps"], processor=self.processor) + media_data[k]["videos"] = video_frames_with_timestamp + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/vlm/timestamp_without_end_time.py b/cosmos_framework/data/vfm/augmentors/vlm/timestamp_without_end_time.py new file mode 100644 index 0000000..8df9dd1 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/timestamp_without_end_time.py @@ -0,0 +1,318 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Augmentors for video dense caption datasets without end time. +Copied from projects/cosmos/reason1/datasets/augmentors/timestamp_without_end_time.py +Changes: + 1. Unify system prompt to 'You are a helpful assistant.' + 2. Move task requirements from system prompts to user prompts. + 3. Randomly change timestamp formats from ["seconds", "hh:mm:ss", "hh:mm:ss.sss", "mm:ss.sss"] + 4. Add json output format for event temporal localization. +""" + +import random +from copy import deepcopy +from typing import Dict, List, Literal, Tuple + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.augmentors.vlm.timestamp import ( + json_to_markdown, + markdown_to_list, + overlay_text, + timestamp_convertor, +) + + +def snap_timestamps_to_existing(assistant_message: List[Dict], existing_timestamps: List[float]) -> List[Dict]: + """ + Snap conversation start/end timestamps to the nearest existing timestamps. + + Args: + assistant_message: List of dictionaries with 'start', 'end', and 'caption' fields + existing_timestamps: List of existing timestamps (floats) to snap to + + Returns: + List of dictionaries with snapped timestamps + """ + snapped_message = [] + for item in assistant_message: + if "caption" not in item and "event" in item: + # This is the nexar dataset + item["caption"] = item["event"] + del item["event"] + + if not isinstance(item, dict) or "start" not in item: + raise ValueError(f"Each item must be a dictionary with 'start' field. getting {item}") + + snapped_item = item.copy() + + # Snap start and end timestamps to existing ones + snapped_item["start"] = min(existing_timestamps, key=lambda x: abs(x - item["start"])) + + snapped_message.append(snapped_item) + + # Sort the merged events by start timestamp to ensure chronological order + # Merge captions that share identical start and end timestamps + merged_events: Dict[Tuple[float], Dict] = {} + for item in snapped_message: + item["start"] = round(item["start"], 2) + + key = (item["start"],) + if key in merged_events: + # Concatenate captions for the same time interval. + merged_events[key]["caption"] = merged_events[key]["caption"].rstrip() + " " + item["caption"].lstrip() + else: + merged_events[key] = item + + merged_events[key]["caption"] = merged_events[key]["caption"].strip() + + # Sort the merged events by start timestamp to ensure chronological order + new_assistant_message = sorted(merged_events.values(), key=lambda x: x["start"]) + if len(new_assistant_message) == 0: + raise ValueError("No valid assistant message found for data.") + + return new_assistant_message + + +def augment_assistant_message( + assistant_message: List[Dict], + output_format: Literal[ + "dense_video_caption_json", + "dense_video_caption_plain", + "single_event_localization", + "multiple_events_localization_json", + "temporal_caption", + ], + timestamp_format: str = "hh:mm:ss", +): + # change time stamp format to hh:mm:ss.sss + assistant_message = deepcopy(assistant_message) + for item in assistant_message: + item["start"] = timestamp_convertor[timestamp_format](item["start"]) + + if output_format == "dense_video_caption_json": + output_message = json_to_markdown(assistant_message) + return output_message + elif output_format == "dense_video_caption_plain": + output_message = "" + for item in assistant_message: + output_message += f"{item['start']}, {item['caption']}\n" + return output_message + elif output_format == "single_event_localization": + return f"{assistant_message[0]['start']}" + elif output_format == "multiple_events_localization_json": + for item in assistant_message: + if "start" in item: + item["time"] = item["start"] + del item["start"] + if "caption" in item: + item["event"] = item["caption"] + del item["caption"] + output_message = json_to_markdown(assistant_message) + return output_message + elif output_format == "temporal_caption": + return assistant_message[0]["caption"] + else: + raise ValueError(f"Invalid output format: {output_format}") + + +def augment_user_prompt( + assistant_message: List[dict], + output_format: Literal[ + "dense_video_caption_json", + "dense_video_caption_plain", + "single_event_localization", + "multiple_events_localization_json", + "temporal_caption", + ], + timestamp_format: str = "hh:mm:ss", +): + if output_format == "dense_video_caption_json" or output_format == "dense_video_caption_plain": + if random.random() < 0.5: + user_prompt = random.choice( + [ + "Caption the notable events in the provided video.", + "Describe the notable events in the provided video.", + "Summarize the notable events in the provided video.", + ] + ) + if random.random() < 0.5: + user_prompt = "Please " + user_prompt.lower() + else: + user_prompt = random.choice( + [ + "Can you caption the notable events in the provided video?", + "Can you describe the notable events in the provided video?", + "Can you summarize the notable events in the provided video?", + ] + ) + if output_format == "dense_video_caption_json": + if random.random() < 0.5: + user_prompt = user_prompt + ( + "\nPlease identify all the events in the following driving video with timestamps using the following format:\n" + "[\n" + " {\n" + f' "start": {timestamp_format},\n' + ' "event": ,\n' + " },\n" + " {\n" + f' "start": {timestamp_format},\n' + ' "event": ,\n' + " },\n" + "]\n" + "Each event corresponds to one and only one of the followinig five types: collision, near collision, hard brake, harsh acceleration, sharp cornering.\n" + ) + else: + user_prompt = ( + user_prompt + + f"\nPlease provide the result in json format with '{timestamp_format}' for time depiction for each event. Use keywords 'start', 'event' in the json output." + ) + else: + user_prompt = ( + user_prompt + + f"\nPlease provide short descriptions of all the events in the video with timestamps using the following format: \n{timestamp_format}, caption of event 1.\n{timestamp_format}, caption of event 2." + ) + elif output_format == "single_event_localization": + event = assistant_message[0] + event_caption = event["caption"] + if not event_caption[-1].isalpha(): + event_caption = event_caption[:-1] + user_prompt = random.choice( + [ + f"When does the following event happen? {event_caption}.", + f"When does the event '{event_caption.lower()}' happen?", + f"Can you find the event '{event_caption.lower()}'?", + ] + ) + user_prompt = ( + user_prompt + + f"\nPlease provide the start timestamp of the event in the following format: {timestamp_format}." + ) + + elif output_format == "multiple_events_localization_json": + user_prompt = random.choice( + [ + f"You should find the following {len(assistant_message)} events in the input video:", + f"Please find the following {len(assistant_message)} events based on descriptions:", + f"Please identify the following events in the input video:", + ] + ) + for i, event in enumerate(assistant_message): + user_prompt += f"\nEvent {i + 1}: {event['caption']}" + user_prompt += f"\nPlease provide the result in json format as a list of dictionaries. Use '{timestamp_format}' for time depiction for each event. Use keywords 'time', 'event' in each dictionary." + + elif output_format == "temporal_caption": + event = assistant_message[0] + if random.random() < 0.333333: + + start = round(event["start"]) + elif random.random() < 0.666666: + + start = round(event["start"] * 2) / 2 + else: + start = event["start"] + + if timestamp_format == "seconds": + if random.random() < 0.5: + start = f"{start}s" + else: + start = f"{start} seconds" + else: + start = timestamp_convertor[timestamp_format](start) + + user_prompt = random.choice( + [ + f"Caption the event starting at {start}.", + f"Please describe the event starting at {start}.", + f"Please caption the event starting at {start}.", + f"Summarize the event starting at {start}.", + ] + ) + return user_prompt + + +class TimeStampWithoutEndTime(Augmentor): + def __init__( + self, + input_key: str = "media", + output_format: Literal[ + "dense_video_caption", "temporal_localization", "temporal_caption", "random" + ] = "dense_video_caption", + urls_needs_timestamp: list = ["av_reasoning_localization_20250627", "tl_activitynet_20250630"], + processor=None, + ) -> None: + """ + Args: + input_keys (list): List of input keys. + """ + self.input_key = input_key + self.output_format = output_format + self.urls_needs_timestamp = urls_needs_timestamp + self.processor = processor + + def __call__(self, data_dict: Dict) -> Dict: + url = data_dict["__url__"] + if not any(url_pattern in url.root for url_pattern in self.urls_needs_timestamp): + return data_dict + + media_data = data_dict[self.input_key] + for k, v in media_data.items(): + if "video" in k: + video_frames_with_timestamp, timestamps = overlay_text(v["videos"], v["fps"], processor=self.processor) + media_data[k]["videos"] = video_frames_with_timestamp + + if self.output_format == "random": + output_format = random.choice(["dense_video_caption", "temporal_localization", "temporal_caption"]) + else: + output_format = self.output_format + + if output_format == "dense_video_caption": + output_format = random.choice(["dense_video_caption_json", "dense_video_caption_plain"]) + + try: + # find the assistant message and parse into a list of dictionaries + for item in data_dict["conversation"]: + if item["role"] == "assistant": + if isinstance(item["content"], list): + assert len(item["content"]) == 1 + assert item["content"][0]["type"] == "text" + item["content"] = item["content"][0]["text"] + assistant_message = markdown_to_list(item["content"]) + assistant_message = snap_timestamps_to_existing(assistant_message, timestamps) + break + + # remove end time if it exists + for item in assistant_message: + if "end" in item: + del item["end"] + + # if temporal localization or caption, sample one event + if output_format == "temporal_localization": + output_format = random.choice(["single_event_localization", "multiple_events_localization_json"]) + if output_format in ["single_event_localization", "temporal_caption"]: + assistant_message = random.sample(assistant_message, 1) + elif output_format == "multiple_events_localization_json": + assistant_message = random.sample(assistant_message, min(len(assistant_message), random.randint(1, 5))) + random.shuffle(assistant_message) + + timestamp_format = random.choice(list(timestamp_convertor.keys())) + # process conversation + conversation = data_dict["conversation"] + for item in conversation: + if item["role"] == "system": + item["content"] = "You are a helpful assistant." + elif item["role"] == "user": + for content in item["content"]: + if content["type"] == "text": + content["text"] = augment_user_prompt(assistant_message, output_format, timestamp_format) + elif item["role"] == "assistant": + assistant_message = augment_assistant_message(assistant_message, output_format, timestamp_format) + item["content"] = assistant_message + data_dict["conversation"] = conversation + + return data_dict + + except Exception as e: + log.warning(f"Error timestamping: {e}. Skipping this sample {url.root} {data_dict['__key__']}.") + return None diff --git a/cosmos_framework/data/vfm/augmentors/vlm/tokenize_data.py b/cosmos_framework/data/vfm/augmentors/vlm/tokenize_data.py new file mode 100644 index 0000000..a0ce29c --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/tokenize_data.py @@ -0,0 +1,363 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Visual-Text Transformations or Augmentations.""" + +import re +from typing import Dict, Optional + +import numpy as np +import torch +from PIL import Image + +from cosmos_framework.data.imaginaire.webdataset.augmentors.augmentor import Augmentor +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.vlm.video_decoder_qwen import token_to_pixels +from cosmos_framework.data.vfm.processors.qwen3vl_processor import Qwen3VLProcessor as Processor +from cosmos_framework.utils.vfm.vlm.constant import IGNORE_INDEX, PROCESSOR_KEYS_TO_ADD + + +def maybe_subsample_frames(model_name_or_path, list_of_pil_image, max_video_token_length, processor): + """ + Why do we need to subsample frames? For model like eagle_er, it does not support smart downsampling in the processor. + And all the frames are resized to the same size. There are 2 senerios the context length can easily exceed the limit. + 1: the video has >32 frames, it will create 256*32=8192 tokens which exceeds the limit. + 2: there are multiple images, by default, each image will be tiled into (at most) 13 tiles. Each tile is 256 tokens. + So if there are multiple images, or many frames in the video, we need to subsample the frames to shorten the context length. + """ + if "Qwen/Qwen2.5-VL" in model_name_or_path: + return list_of_pil_image + elif "eagle_er" in model_name_or_path or "InternVL3_5" in model_name_or_path: + tokens_per_tile = processor.tokens_per_tile + # 1 frames map to 256 tokens + estimated_num_frames = max_video_token_length // tokens_per_tile + if len(list_of_pil_image) > estimated_num_frames: + # Evenly sample frames + sample_idx = np.linspace(0, len(list_of_pil_image) - 1, estimated_num_frames).astype(int) + return [list_of_pil_image[i] for i in sample_idx] + else: + return list_of_pil_image + else: + return list_of_pil_image + + +def convert_all_images_to_rgb(conversation): + """ + Convert all images to RGB. Otherwise the tokenizer will raise error for image in LA mode. + """ + new_conversation = [] + for conversation_round in conversation: + if isinstance(conversation_round["content"], list): + new_content_list = [] + for content in conversation_round["content"]: + if "type" not in content: + log.critical( + f"content: {content} | conversation_round: {conversation_round} | full conversation: {conversation}" + ) + content = {"type": "text", "text": content} + content_type = content["type"] + if content_type in ["image", "video"]: + if isinstance(content[content_type], Image.Image): + content[content_type] = content[content_type].convert("RGB") + elif isinstance(content[content_type], list): + content_i = content[content_type] + new_content_i = [] + for img in content_i: + if isinstance(img, Image.Image): + img = img.convert("RGB") + new_content_i.append(img) + content[content_type] = new_content_i + new_content_list.append(content) + conversation_round["content"] = new_content_list + new_conversation.append(conversation_round) + + return new_conversation + + +def compress_repeated_tokens(dialog_str): + pattern = re.compile(r"((<\|[^|]+\|>|<｜[^<>]+｜>|\[[^\]]+\]))\1+") + + def replacer(match): + token = match.group(1) + count = len(match.group(0)) // len(token) + return f"{token}*{count}times" + + # Cap length to avoid regex hang on very long decoded sequences + max_len = 16 * 1024 + if len(dialog_str) > max_len: + dialog_str = dialog_str[:max_len] + "...[truncated]" + return pattern.sub(replacer, dialog_str) + + +class TokenizeData(Augmentor): + """ + Image-Text Transform for Supervised Fine-Tuning (SFT) data, for Vision-Language Model training. + """ + + def __init__( + self, + processor: Optional[Processor] = None, + max_video_token_length: int = 8192, + max_image_token_length: int = 8192, + add_system_prompt_if_missing: bool = False, + text_only: bool = False, + ) -> None: + """ + Args: + processor (Processor): Text/Image processor for tokenization. + max_video_token_length (int): Maximum number of video tokens to use. Defaults to 8192. + """ + # Create the tokenizer + self.text_only = text_only + self.processor = processor # Expecting a ImageTextTokenizer + self.max_video_token_length = max_video_token_length + self.max_image_token_length = max_image_token_length + self.add_system_prompt_if_missing = add_system_prompt_if_missing + + def __call__(self, data_dict: Dict) -> Dict: + r"""Tokenize a dialog and pad the sequence. + + "media" is a dict of + { + "video_1": {"video": [PIL.Image.Image, ...], "fps": int}, + "image_1": PIL.Image.Image, + } + + "conversation" is a list of dicts, each dict has the following fields: + { + "role": "user" or "assistant", + "content": [ + {"type": "video", "video": media_key_in_media_dict}, + {"type": "image", "image": media_key_in_media_dict}, + {"type": "text", "text": str}, + ], + } + or + { + "role": "user" or "assistant", + "content": str, + } + + Args: + data_dict (dict): Input data dict + + Returns: + data_dict (dict): Output dict + """ + conversation = data_dict["conversation"] + processor_kwargs = {} + total_images = 0 + total_videos = 0 + raw_images = [] + # Pre-compute the total_images and total_videos + for message in conversation: + if not isinstance(message, dict): + raise ValueError( + f"message is not a dict: {message} | conversation: {conversation} | data_dict: {data_dict} | __url__: {data_dict['__url__'].root}, {data_dict['__url__'].path}" + ) + if message["role"] == "user" and isinstance(message["content"], list): + total_images += len([content for content in message["content"] if content["type"] == "image"]) + total_videos += len([content for content in message["content"] if content["type"] == "video"]) + + assert total_videos == 1 or total_videos == 0, "Only one video is supported for now" + + # url + url = data_dict["__url__"].root + "/" + data_dict["__url__"].path + + # go through each message in the conversation + for message in conversation: + # for user message, we insert the media + + if message["role"] == "user" and isinstance( + message["content"], list + ): # Otherwise it's text and content is a string + images_content_idx_full = [ + content_idx for content_idx, content in enumerate(message["content"]) if content["type"] == "image" + ] + images_content_idx_subsampled = maybe_subsample_frames( + self.processor.name, images_content_idx_full, self.max_image_token_length, self.processor + ) + if ( + len(images_content_idx_subsampled) > 0 + ): # for eagle, we need to reduce the max_dynamic_tiles and not use thumbnail. These args only valid for eagle_er processor. + processor_kwargs["max_dynamic_tiles"] = 1 + processor_kwargs["use_thumbnail"] = False + + new_content_list = [] + for content_idx, content in enumerate(message["content"]): + if content["type"] == "image": + if content_idx not in images_content_idx_subsampled: + continue + # for image, we do NOT use the temporal patch size, this leads to a smaller max_pixels + # Later, each image will be repeated temporal_patch_size times + max_total_pixels = token_to_pixels( + self.max_image_token_length, + patch_size=self.processor.patch_size, + temporal_patch_size=1, # Because this is image, not video + ) + max_pixels_per_image = max_total_pixels // total_images + + if self.processor.use_smart_resize: + min_pixels_per_image = self.processor.processor.image_processor.size["shortest_edge"] + if max_pixels_per_image < min_pixels_per_image: + log.critical( + f"max_pixels_per_image: {max_pixels_per_image} < min_pixels_per_image: {min_pixels_per_image} | self.max_video_token_length = {self.max_video_token_length} is not enough for total_images: {total_images}, as the default min_pixels is {min_pixels_per_image} | Either increase max_video_token_length or include max_pixels in the content or reduce min_pixels" + ) + return None + + # Add each image to the content list + if "media" not in data_dict: + log.critical( + f"[TokenizerDataError]media not found in data_dict, available keys: {data_dict.keys()}. url: {url}, content: {message['content']}", + rank0_only=False, + ) + return None + + elif content["image"] not in data_dict["media"]: + log.critical( + f"[TokenizerDataError]image {content['image']} not found in media, available keys: {data_dict['media'].keys()}. url: {url}", + rank0_only=False, + ) + return None + image = data_dict["media"][content["image"]] + content["image"] = image + content["max_pixels"] = max_pixels_per_image + raw_images.append(image) + + elif content["type"] == "video": + + # as tokenization will NOT upsample the video, we can use a larger value here at the cost of multiple video having 1.5x token length + max_total_pixels = token_to_pixels(self.max_video_token_length * 1.5, temporal_patch_size=2) + media_key = content["video"] + # Add each video to the content list + if "media" not in data_dict: + log.critical( + f"[TokenizerDataError]media not found in data_dict, available keys: {data_dict.keys()}. url: {url}, content: {message['content']}", + rank0_only=False, + ) + return None + if media_key not in data_dict["media"]: + log.info( + f"[TokenizerDataError]video {media_key} not found in media, available keys: {data_dict['media'].keys()}. url: {url}" + ) + return None + if "videos" not in data_dict["media"][media_key]: + log.info( + f"[TokenizerDataError]videos not found in media[{media_key}], available keys: {data_dict['media'][media_key].keys()}. url: {url}" + ) + return None + videos = data_dict["media"][media_key]["videos"] # list of PIL images + fps = data_dict["media"][media_key]["fps"] + + # this is because videos are decoded to be around "max_video_token_length" tokens + + videos = maybe_subsample_frames( + self.processor.name, videos, self.max_video_token_length, self.processor + ) + content["video"] = videos + + max_pixels_per_image = max_total_pixels // total_videos // len(videos) + content["fps"] = fps + content["max_pixels"] = max_pixels_per_image + + data_dict["raw_video"] = torch.from_numpy(np.array(videos)).permute( + 3, 0, 1, 2 + ) # [3,T,H,W], range [0, 255] + new_content_list.append(content) + message["content"] = new_content_list + + if len(raw_images) > 1: + # resize the raw_image to the size of the first image + image_size = raw_images[0].size + raw_images = [image.resize(image_size) for image in raw_images] + + if len(raw_images) > 0: + data_dict["raw_image"] = torch.from_numpy(np.array(raw_images)).permute(3, 0, 1, 2) # [3,num_images,H,W] + + if conversation[0]["role"] != "system" and self.add_system_prompt_if_missing: + conversation.insert(0, {"role": "system", "content": "You are a helpful assistant."}) + + if self.text_only and (total_images > 0 or total_videos > 0): + log.critical( + f"Images or videos found in the conversation but expect only text, __url__: {url} | data_dict: {data_dict.keys()} | conversation={conversation}" + ) + return None + + if total_images > 1 or total_videos > 1: + add_vision_id = True + else: + add_vision_id = False + + try: + conversation = convert_all_images_to_rgb(conversation) + except Exception as e: + log.critical( + f"Error in convert_all_images_to_rgb: {e} | conversation: {conversation} | __url__: {url} | data_dict: {data_dict.keys()}" + ) + return None + + try: + tokenizer_output = self.processor.apply_chat_template( + conversation, + tokenize=True, + add_generation_prompt=False, + add_vision_id=add_vision_id, + **processor_kwargs, + ) + except Exception as e: + log.critical( + f"Error in tokenizer_output: {e} | conversation: {conversation} | __url__: {url} | data_dict: {data_dict.keys()}" + ) + return None + input_ids = tokenizer_output["input_ids"] + if "image_grid_thw" in tokenizer_output and "raw_image" in data_dict: + # image_grid_thw: (1, t, h, w) + t, h, w = tokenizer_output["image_grid_thw"][0] + # interpolate raw_image to the size of the image grid * 14 + data_dict["raw_image"] = torch.nn.functional.interpolate( + data_dict["raw_image"], size=(h * 14, w * 14), mode="bilinear", align_corners=False + ) # [3,num_images,h*14,w*14] + + try: + # token_mask: True for tokens to compute loss on; False for tokens to ignore + token_mask = self.processor.add_assistant_tokens_mask(input_ids) + except Exception as e: + log.critical( + f"Error in add_assistant_tokens_mask: {e} | conversation: {conversation} | __url__: {url} | data_dict: {data_dict.keys()}" + ) + return None + + input_ids = torch.LongTensor(input_ids) # [N_token] + token_mask = torch.BoolTensor(token_mask) # [N_token]; True = compute loss on this token + + data_dict.update( + { + "input_ids": input_ids, + "token_mask": token_mask, + } + ) + for key in PROCESSOR_KEYS_TO_ADD: + if key in tokenizer_output: + data_dict[key] = tokenizer_output[key] + labels = tokenizer_output["input_ids"].clone() # [N_token] + labels[~token_mask] = IGNORE_INDEX + data_dict["labels"] = labels + data_dict["pad_token_id"] = self.processor.pad_id + data_dict["ignore_index"] = IGNORE_INDEX + + # keep raw text for debugging/logging purpose. Add \n\n after each <|im_end|>. + dialog_str = self.processor.decode(input_ids) + data_dict["dialog_str"] = compress_repeated_tokens(dialog_str.replace("<|im_end|>", "<|im_end|>\n\n")) + + # For debugging purpose + msg = f"input_ids: {input_ids.shape[-1]} | __url__: {data_dict['__url__'].root}, {data_dict['__url__'].path} | __key__: {data_dict['__key__']}" + if "raw_video" in data_dict: + msg += f" | raw_video: {data_dict['raw_video'].shape} " + if "raw_image" in data_dict: + msg += f" | raw_image: {data_dict['raw_image'].shape} " + if "pixel_values" in data_dict: + msg += f" | pixel_values: {data_dict['pixel_values'].shape} " + + msg += f"original conversation: {data_dict['conversation']}" + + return data_dict diff --git a/cosmos_framework/data/vfm/augmentors/vlm/user_prompt_caption_general.json b/cosmos_framework/data/vfm/augmentors/vlm/user_prompt_caption_general.json new file mode 100644 index 0000000..8433bb7 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/user_prompt_caption_general.json @@ -0,0 +1,98 @@ +[ + "Describe the image.", + "What do you see in the image?", + "Provide a caption for this image.", + "Explain what is happening in the picture.", + "Summarize the content of the image.", + "Generate a description for the image.", + "Give a detailed description of this picture.", + "What is depicted in the image?", + "Interpret the scene in the image.", + "Analyze the contents of the image.", + "Write a caption for the image.", + "What information does the image convey?", + "Break down the elements in the picture.", + "Narrate what you see in the image.", + "Can you describe the image in detail?", + "Provide a textual representation of the image.", + "Create a description for the given picture.", + "List the main objects in the image.", + "What is the overall theme of the image?", + "Describe the colors and objects in the image.", + "Explain the emotions conveyed by the image.", + "Write a short summary of the image.", + "Generate an insightful caption for this picture.", + "Tell me what is present in the image.", + "Identify key elements in the picture.", + "What is the subject of this image?", + "Give a brief description of this image.", + "Describe the composition of the image.", + "Write an alt text for this image.", + "Create an image description in a sentence.", + "What’s the primary focus of the image?", + "Explain the scene depicted in the picture.", + "Summarize the visual content of the image.", + "Describe the setting of this picture.", + "What message does the image convey?", + "How would you describe this image to someone who can’t see it?", + "Describe the action occurring in the image.", + "Write a detailed breakdown of this picture.", + "What’s happening in this scene?", + "Provide an objective description of the image.", + "What does the image represent?", + "Describe the mood and atmosphere of the image.", + "What kind of scene is captured in this image?", + "Summarize this image in a few words.", + "What are the most prominent elements in this picture?", + "Tell me the meaning of this image.", + "Describe the perspective used in this image.", + "Identify the key features of the picture.", + "Write a natural language description of the image.", + "Give a creative description of this image.", + "Interpret the context of the image.", + "What do the objects in the image signify?", + "Tell me what story this image tells.", + "How would you caption this picture?", + "Describe this image like you would in an article.", + "What’s the background of this image?", + "Describe the lighting and shadows in this image.", + "How does this image make you feel?", + "Explain the composition and focal points of the image.", + "Write a summary of the visual elements in the image.", + "List and describe the objects in the image.", + "Break down the details of this picture.", + "Provide a short textual description of this image.", + "What’s the first thing you notice in this image?", + "Describe this image in a few sentences.", + "Write an accessible alt-text for this picture.", + "What’s visually interesting about this image?", + "Summarize the details of the picture.", + "What’s the story behind this image?", + "Give a brief analysis of this image.", + "Describe the symmetry or patterns in this image.", + "What key visual elements stand out in this picture?", + "Give an artistic interpretation of this image.", + "How would you explain this image to a child?", + "Provide a step-by-step breakdown of this image.", + "Explain the depth and perspective of the image.", + "Describe the relationships between elements in the image.", + "What’s the focal point of this image?", + "Write a caption for this image that explains it concisely.", + "Describe the textures visible in this image.", + "What visual style does this image have?", + "How does this image relate to its surroundings?", + "Describe the contrast and color palette of this image.", + "Tell me what action is happening in this scene.", + "Explain the significance of the objects in this image.", + "What symbols or metaphors are present in this image?", + "Describe the movement and flow in this picture.", + "What’s the central theme of this image?", + "Provide a written interpretation of this image.", + "Describe the balance and harmony in the picture.", + "Tell me what’s most eye-catching about this image.", + "How would you summarize this image in one line?", + "Write a caption that would fit well with this image.", + "Describe the emotions depicted in this image.", + "What’s unique about this image?", + "Explain the interaction between the subjects in this picture." + ] diff --git a/cosmos_framework/data/vfm/augmentors/vlm/user_prompt_ocr.json b/cosmos_framework/data/vfm/augmentors/vlm/user_prompt_ocr.json new file mode 100644 index 0000000..349b4c0 --- /dev/null +++ b/cosmos_framework/data/vfm/augmentors/vlm/user_prompt_ocr.json @@ -0,0 +1,100 @@ +[ + "Extract the text from the image.", + "Read the text in this image.", + "Perform OCR on this image.", + "Identify and extract the words from the image.", + "Recognize the text in this picture.", + "Convert the text in the image into readable text.", + "What does the text in the image say?", + "Retrieve the text content from the image.", + "Scan and extract the words from the image.", + "Detect and read the text in the picture.", + "Provide the text found in the image.", + "Extract and transcribe the words in the image.", + "Can you recognize the text in this image?", + "Extract text and provide a readable version.", + "List all the words present in this image.", + "Convert the image text into digital text.", + "Recognize and extract the characters from the image.", + "What is the written content in the image?", + "Decode the text embedded in the image.", + "Transcribe the words appearing in this image.", + "Identify the letters and numbers in the image.", + "Convert the visual text into a readable format.", + "Extract the words visible in the picture.", + "Recognize and digitize the text in the image.", + "Scan and retrieve the written information from the image.", + "Convert the printed text in the image to text format.", + "Analyze and extract text from the picture.", + "Provide a text version of the content in the image.", + "Detect characters and words in this image.", + "Generate a transcript of the text in this image.", + "What is written in this image?", + "Identify and extract any text found in the image.", + "Extract and display the words from this picture.", + "Scan the image for any textual content.", + "Retrieve and present the text detected in the image.", + "Recognize the letters and words in this picture.", + "Extract and provide the readable text from the image.", + "What can be read from this image?", + "Process the image to retrieve any visible text.", + "Convert this image-based text into editable text.", + "Can you scan this image for text?", + "Read and extract all text content from this image.", + "Convert handwritten or printed text from the image into digital text.", + "Identify any words or symbols in this image.", + "Recognize and output the text from the image.", + "Detect and convert any text in the image to readable text.", + "Extract all readable content from this picture.", + "Find and transcribe the text in this image.", + "Interpret the words appearing in the image.", + "Identify and read any textual elements in the image.", + "Scan for text and output the recognized words.", + "Recognize and retrieve any text in this photo.", + "Extract the characters from this picture.", + "What does the text in this image say?", + "Convert this image's text to machine-readable format.", + "Digitize the text in the image.", + "Retrieve and format the words from this image.", + "Can you extract the letters in this image?", + "Detect and extract words from this scanned document.", + "Analyze the image and provide the text found in it.", + "Retrieve the information written in this image.", + "Extract readable text from this document image.", + "Convert the letters in the image into digital form.", + "Analyze and output the text detected in the image.", + "Scan and extract text information from this picture.", + "Identify and copy any words in this image.", + "Transcribe the image's text into plain text.", + "Extract the visible letters and numbers from this image.", + "What words are present in this image?", + "Extract text as if scanning a document.", + "Digitally transcribe the text in this image.", + "What characters can be detected in this image?", + "Identify and extract any visible phrases from the image.", + "Detect and recognize any writing in this picture.", + "Find any readable words in the image and extract them.", + "Read and transcribe the contents of this image.", + "Recognize and convert the text in the image to digital text.", + "Retrieve and list any textual elements present.", + "Analyze the image for legible words.", + "Extract the written words from this image.", + "Provide the extracted words from this scanned photo.", + "Convert the textual content in the image into an editable format.", + "List the letters and words recognized from the image.", + "Generate a machine-readable version of the text in the image.", + "Retrieve any alphanumeric text from the image.", + "Recognize any textual symbols present in the image.", + "Identify words from this image and transcribe them.", + "Detect and retrieve the text printed in this picture.", + "Recognize and output any readable text from this image.", + "Convert the text in the image into selectable text.", + "Extract letters, words, and numbers from this image.", + "Scan for any handwritten or typed words in the image.", + "Identify all readable characters in this image.", + "Recognize the text and return it in digital form.", + "What readable information does this image contain?", + "Perform OCR processing on this image and extract text.", + "Detect, extract, and format the text from this image.", + "Retrieve the textual content visible in this picture." + ] diff --git a/cosmos_framework/data/vfm/cached_replay_dataloader.py b/cosmos_framework/data/vfm/cached_replay_dataloader.py new file mode 100644 index 0000000..7bfe8c1 --- /dev/null +++ b/cosmos_framework/data/vfm/cached_replay_dataloader.py @@ -0,0 +1,498 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import copy +import threading +import traceback +from typing import Callable, Dict, Iterator, List, Optional + +import numpy as np +import torch +from torch.utils.data import DataLoader + +from cosmos_framework.data.vfm.watchdog import OperationWatchdog + + +def generate_multiple_image_batches(data_batch, nh, nw, output_height, output_width): + """ + Generate (nh * nw) data batches, each with images randomly cropped from the original. + + Args: + data_batch: Input batch containing images and other elements + nh: Number of replications in height direction + nw: Number of replications in width direction + output_height: Target image height + output_width: Target image width + + Returns: + List of data batches with randomly cropped images + """ + # Access the original image tensor + original_images = data_batch["images"] # [B,C,H,W] + + # Get the original image dimensions + B, C, H, W = original_images.shape + + # Check if output dimensions are valid + if output_height > H or output_width > W: + raise ValueError("Output dimensions cannot be larger than original dimensions") + + # Calculate step sizes for uniform sampling with minimal overlap + h_step = max(1, (H - output_height) // max(1, nh - 1)) if nh > 1 else 0 + w_step = max(1, (W - output_width) // max(1, nw - 1)) if nw > 1 else 0 + + # Initialize list to store all created batches + all_batches = [] + + # Generate crops + for h_idx in range(nh): + for w_idx in range(nw): + # Create a deep copy of the original data batch + new_batch = copy.deepcopy(data_batch) + + # Calculate starting positions for this crop + # Add some randomness within each grid cell to avoid exact same positions + h_start = min(H - output_height, h_idx * h_step + torch.randint(0, max(1, h_step), (1,)).item()) + w_start = min(W - output_width, w_idx * w_step + torch.randint(0, max(1, w_step), (1,)).item()) + + # For the edge case where there's only one crop position, center the crop + if nh == 1: + h_start = (H - output_height) // 2 + if nw == 1: + w_start = (W - output_width) // 2 + + # Crop the image + new_batch["images"] = original_images[ + :, + :, + h_start : h_start + output_height, + w_start : w_start + output_width, + ] # [B,C,output_height,output_width] + + # Add to our list of batches + all_batches.append(new_batch) + + return all_batches + + +def generate_multiple_video_batches(data_batch, nt, nh, nw, output_length, output_height, output_width): + """ + Generate (nt * nh * nw) data batches, each with videos randomly cropped from the original. + + Args: + data_batch: Input batch containing videos and other elements + nt: Number of replications in temporal (length) direction + nh: Number of replications in height direction + nw: Number of replications in width direction + output_length: Target video length (temporal dimension) + output_height: Target video height + output_width: Target video width + + Returns: + List of data batches with randomly cropped videos + """ + # Access the original video tensor + original_video = data_batch["video"] # [B,C,T,H,W] + + # Get the original video dimensions + B, C, T, H, W = original_video.shape + + # Check if output dimensions are valid + if output_length > T or output_height > H or output_width > W: + raise ValueError("Output dimensions cannot be larger than original dimensions") + + # Calculate step sizes for uniform sampling with minimal overlap + t_step = max(1, (T - output_length) // max(1, nt - 1)) if nt > 1 else 0 + h_step = max(1, (H - output_height) // max(1, nh - 1)) if nh > 1 else 0 + w_step = max(1, (W - output_width) // max(1, nw - 1)) if nw > 1 else 0 + + # Initialize list to store all created batches + all_batches = [] + + # Generate crops + for t_idx in range(nt): + for h_idx in range(nh): + for w_idx in range(nw): + # Create a deep copy of the original data batch + new_batch = copy.deepcopy(data_batch) + + # Calculate starting positions for this crop + # Add some randomness within each grid cell to avoid exact same positions + t_start = min(T - output_length, t_idx * t_step + torch.randint(0, max(1, t_step), (1,)).item()) + h_start = min(H - output_height, h_idx * h_step + torch.randint(0, max(1, h_step), (1,)).item()) + w_start = min(W - output_width, w_idx * w_step + torch.randint(0, max(1, w_step), (1,)).item()) + + # For the edge case where there's only one crop position, center the crop + if nt == 1: + t_start = (T - output_length) // 2 + if nh == 1: + h_start = (H - output_height) // 2 + if nw == 1: + w_start = (W - output_width) // 2 + + # Crop the video + new_batch["video"] = original_video[ + :, + :, + t_start : t_start + output_length, + h_start : h_start + output_height, + w_start : w_start + output_width, + ] # [B,C,output_length,output_height,output_width] + + # Add to our list of batches + all_batches.append(new_batch) + + return all_batches + + +def duplicate_batches(data_batch, n: int) -> List[Dict]: + """ + Duplicate a data batch n times. + """ + return [copy.deepcopy(data_batch) for _ in range(n)] + + +_RNG = np.random.default_rng(123) + + +def duplicate_batches_random(data_batch, n: float) -> List[Dict]: + floor = int(np.floor(n)) + ceil = int(np.ceil(n)) + # generate a random number uniformly from [floor, ceil) + random_number = _RNG.uniform(floor, ceil) + if random_number > n: + return [copy.deepcopy(data_batch) for _ in range(floor)] + else: + return [copy.deepcopy(data_batch) for _ in range(ceil)] + + +def concatenate_batches(n: int, data_batches: List[Dict]) -> List[Dict]: + """ + Smartly concatenate n input data batches into m output data batches. + Each data batch is a dictionary with values that can be torch tensor, string, or list. + + Args: + n (int): Number of input batches to process per output batch + data_batches (list): List of dictionary data batches + + Returns: + list: List of concatenated data batches + """ + if n <= 0: + raise ValueError("n must be a positive integer") + + # Calculate m based on the input + total_batches = len(data_batches) + if total_batches % n != 0: + raise ValueError(f"Length of data_batches ({total_batches}) must be divisible by n ({n})") + + m = total_batches // n + + # Initialize output batches + output_batches = [] + + # Process in groups of n + for i in range(m): + # Get the corresponding batch from each group + batches_to_concat = [] + for j in range(n): + batch_idx = j * m + i + batches_to_concat.append(data_batches[batch_idx]) + + # Create a new merged dictionary + merged_batch = {} + + # Get all unique keys from the dictionaries + all_keys = set() + for batch in batches_to_concat: + all_keys.update(batch.keys()) + + # Process each key + for key in all_keys: + # Collect values for this key from all batches + values = [] + for batch in batches_to_concat: + if key in batch: + values.append(batch[key]) + + if not values: + continue + + # Determine the type of the first non-None value + first_value = next((v for v in values if v is not None), None) + if first_value is None: + merged_batch[key] = None + continue + + # Handle different types + if isinstance(first_value, torch.Tensor): + # Assuming this is a tensor-like object with cat method (e.g., torch.Tensor) + merged_batch[key] = torch.cat(values, dim=0) # [n*B,...] + elif isinstance(first_value, str): + merged_batch[key] = values[0] + elif isinstance(first_value, list): + # Extend lists + merged_list = [] + for v in values: + merged_list.extend(v) + merged_batch[key] = merged_list + else: + # For other types, just use the list of values + merged_batch[key] = values + + output_batches.append(merged_batch) + + return output_batches + + +class CachedReplayDataLoader: + """A DataLoader wrapper that asynchronously caches and replays data batches to + mitigate slow loading issues. Assumes the underlying DataLoader is infinite. + + This class delegates all augmentation logic to an external augmentation function, + which takes a batch from the data loader and returns multiple augmented versions. + The class handles caching these augmented batches and optionally concatenating + them when yielded. + + Attributes: + data_loader (DataLoader): The underlying infinite DataLoader. + cache_size (int): Maximum number of augmented batches to store in the cache. + cache_augmentation_fn (Callable): Function to create multiple augmented versions of each batch. + concat_size (int): Number of batches to concatenate when yielding from the iterator. + rng (numpy.random.Generator): Controlled random number generator for deterministic behavior. + """ + + def __init__( + self, + data_loader: DataLoader, + cache_size: int, + cache_augmentation_fn: Callable[[Dict], List[Dict]], + concat_size: int = 1, + name: str = "cached_replay_dataloader", + ) -> None: + """Initialize the CachedReplayDataLoader. + + Args: + data_loader (DataLoader): The infinite DataLoader to fetch data batches from. + cache_size (int): Maximum number of augmented data batches to store in the cache. + cache_augmentation_fn (Callable[[Dict], List[Dict]]): Function that takes a batch and returns + a list of augmented batches. + concat_size (int, optional): Number of batches to concatenate when yielding. Defaults to 1. + """ + self.data_loader = data_loader + self.cache_size = cache_size + self.cache_augmentation_fn = cache_augmentation_fn + self.concat_size = concat_size + + # Create controlled random number generator for deterministic behavior + self.rng = np.random.default_rng(123) + + # Create an iterator over the infinite DataLoader. + self._data_iter: Iterator = iter(self.data_loader) + # Internal cache to store augmented batches. + self._cache: List[Dict] = [] + # Condition variable to manage cache access. + self._cache_cond = threading.Condition() + # Event to signal the background thread to stop. + self._stop_event = threading.Event() + # Store exceptions from the background thread + self._prefetch_exception = None + + self._watchdog = OperationWatchdog(warning_threshold=100, verbose_interval=600, name=name) + self._prefetch_thread = threading.Thread( + target=self._prefetch_loop, daemon=True, name=f"{name}_prefetch_thread" + ) + self._prefetch_thread.start() + + def _prefetch_loop(self) -> None: + """Continuously fetch batches from the DataLoader, augment them, and store in the cache. + + If the cache is full (reaches `cache_size`), this loop waits until space is available. + Catches exceptions and stores them for later propagation to the main thread. + """ + try: + while not self._stop_event.is_set(): + try: + with self._watchdog.watch("fetch raw batch", verbose_first_n=5): + batch = next(self._data_iter) + except Exception as e: + # Capture DataLoader errors + self._set_exception(e, "Error fetching batch from DataLoader") + break + + try: + # Apply augmentation function to generate multiple augmented batches + with self._watchdog.watch("augmentation", verbose_first_n=5): + augmented_batches = self.cache_augmentation_fn(batch) + except Exception as e: + # Capture augmentation function errors + self._set_exception(e, "Error in augmentation function") + break + + try: + # Use controlled random generator for shuffling + permutation = self.rng.permutation(len(augmented_batches)) + augmented_batches = [augmented_batches[i] for i in permutation] + + for aug_batch in augmented_batches: + with self._cache_cond: + while len(self._cache) >= self.cache_size and not self._stop_event.is_set(): + self._cache_cond.wait(timeout=1.0) + if self._stop_event.is_set(): + break + self._cache.append(aug_batch) + self._cache_cond.notify_all() + except Exception as e: + # Capture other errors during caching + self._set_exception(e, "Error adding batch to cache") + break + except Exception as e: + # Catch any other unforeseen errors + self._set_exception(e, "Unexpected error in prefetch thread") + + def _set_exception(self, exception: Exception, context: str = "") -> None: + """Store an exception from the background thread with context information. + + Args: + exception (Exception): The exception that was raised + context (str, optional): Additional context about where the error occurred + """ + error_info = f"{context}: {str(exception)}\n{traceback.format_exc()}" + with self._cache_cond: + self._prefetch_exception = RuntimeError(error_info) + self._cache_cond.notify_all() # Wake up any waiting threads + + def _check_for_errors(self) -> None: + """Check if the background thread has encountered an error and raise it if so.""" + if self._prefetch_exception is not None: + raise self._prefetch_exception + + def __iter__(self) -> Iterator[Dict]: + """Yield augmented data batches from the cache, optionally concatenated based on concat_size. + + This method starts the background prefetch thread if it hasn't been started yet. + If concat_size > 1, it collects multiple batches and concatenates them. + + Raises: + RuntimeError: If the background thread encountered an error + """ + while not self._stop_event.is_set(): + if self.concat_size <= 1: + # Simple case: yield single batches + with self._watchdog.watch("main thread fetch single batch", verbose_first_n=5): + with self._cache_cond: + while not self._cache and not self._stop_event.is_set() and self._prefetch_exception is None: + self._cache_cond.wait(timeout=1.0) # Add timeout to periodically check for errors + + # Check for errors before proceeding + self._check_for_errors() + + if self._stop_event.is_set(): + break + + if not self._cache: # If cache is still empty after timeout + continue + + # Use controlled random generator to select batch index + idx = self.rng.integers(0, len(self._cache)) + batch = self._cache.pop(idx) + self._cache_cond.notify_all() + yield batch + else: + # Collect concat_size batches and concatenate them + with self._watchdog.watch("main thread fetch smaples", verbose_first_n=5): + collected_batches = [] + for _ in range(self.concat_size): + with self._cache_cond: + while ( + not self._cache and not self._stop_event.is_set() and self._prefetch_exception is None + ): + self._cache_cond.wait(timeout=1.0) # Add timeout to periodically check for errors + + # Check for errors before proceeding + self._check_for_errors() + + if self._stop_event.is_set(): + break + + if not self._cache: # If cache is still empty after timeout + continue + + # Use controlled random generator to select batch index + idx = self.rng.integers(0, len(self._cache)) + batch = self._cache.pop(idx) + self._cache_cond.notify_all() + collected_batches.append(batch) + + if self._stop_event.is_set(): + break + + if not collected_batches: + continue + + if len(collected_batches) < self.concat_size: + # Not enough batches collected, just concatenate the ones we have + concat_batches = concatenate_batches(len(collected_batches), collected_batches) + yield concat_batches[0] + else: + # Concatenate the collected batches + try: + concat_batches = concatenate_batches(self.concat_size, collected_batches) + yield concat_batches[0] + except Exception as e: + # Handle errors in batch concatenation + raise RuntimeError(f"Error concatenating batches: {str(e)}") from e + + def __len__(self) -> int: + """Return the length of the underlying DataLoader.""" + return len(self.data_loader) + + def close(self) -> None: + """Stop the prefetch thread and clear the cache. + Also checks for any errors in the background thread and raises them. + """ + self._stop_event.set() + with self._cache_cond: + self._cache_cond.notify_all() + if self._prefetch_thread is not None: + self._prefetch_thread.join(timeout=5.0) # Add timeout to avoid hanging on thread join + with self._cache_cond: + self._cache.clear() + + # Check and propagate any errors from the background thread + self._check_for_errors() + + +def get_cached_replay_dataloader( + use_cache: bool = False, + cache_size: int = 32, + concat_size: int = 1, + cache_augment_fn: Optional[Callable] = None, + cache_replay_name: str = "cached_replay_dataloader", + webdataset: bool = True, + **kwargs, +): + if webdataset: + from imaginaire.datasets.webdataset.dataloader import DataLoader as _DataLoader + else: + from torch.utils.data import DataLoader as _DataLoader + + if not use_cache: + return _DataLoader(**kwargs) + + expected_batch_size = kwargs["batch_size"] + assert expected_batch_size % concat_size == 0, ( + f"Batch size {expected_batch_size} must be divisible by concat_size {concat_size}" + ) + kwargs["batch_size"] = expected_batch_size // concat_size + + dataloader = _DataLoader(**kwargs) + + # wrapper it with cached replay dataloader + return CachedReplayDataLoader( + data_loader=dataloader, + cache_size=cache_size, + concat_size=concat_size, + cache_augmentation_fn=cache_augment_fn, + name=cache_replay_name, + ) diff --git a/cosmos_framework/data/vfm/data_packer.py b/cosmos_framework/data/vfm/data_packer.py new file mode 100644 index 0000000..17dfc08 --- /dev/null +++ b/cosmos_framework/data/vfm/data_packer.py @@ -0,0 +1,107 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +""" +Storage-agnostic sample-level transform protocol for OSS-compatible training. + +Interface is name-compatible with cosmos-rl's ``BaseDataPacker`` SFT methods: + + sft_process_sample ↔ sft_process_sample (identical) + sft_collate_fn ↔ sft_collate_fn (identical) + compute_num_tokens ← NEW single-sample token cost for packing budget + +After adding a one-line ``compute_num_tokens`` default to cosmos-rl's +``BaseDataPacker``, existing cosmos-rl packers (``HFVLMDataPacker``, +``Qwen3_VL_DataPacker``, etc.) become directly usable here with no other changes. + +Usage +----- +Subclass ``DataPacker`` and implement three methods, then plug into +``DataPackerDataLoader``:: + + class MyPacker(DataPacker): + def sft_process_sample(self, item): + return {"input_ids": tokenizer(item["text"]).input_ids} + + def compute_num_tokens(self, sample): + return len(sample["input_ids"]) + + def sft_collate_fn(self, samples, max_len, ignore_label_id=-100): + # pad and stack + ... + + loader = DataPackerDataLoader( + data_source=my_dataset, + data_packer=MyPacker(), + max_tokens=16000, + ) +""" + +from __future__ import annotations + +from abc import ABC, abstractmethod +from typing import Any + + +class DataPacker(ABC): + """Storage-agnostic protocol for transforming dataset items into training batches. + + OSS users subclass this to support any model or dataset format. + Three abstract methods are required; the rest of the training infrastructure + (packing, worker management, Hydra config) is inherited automatically. + """ + + @abstractmethod + def sft_process_sample(self, item: Any) -> dict: + """Convert one raw dataset item into a training-ready sample dict. + + Parameters + ---------- + item: + Whatever the user's ``data_source`` iterable yields — + a HuggingFace record, a ``{"image": PIL.Image, "text": str}`` dict, + or any other format. + + Returns + ------- + dict + Must contain at minimum the keys expected by ``sft_collate_fn`` + and must have a token-countable representation for + ``compute_num_tokens``. + """ + + @abstractmethod + def compute_num_tokens(self, sample: dict) -> int: + """Return the token cost of one sample for the packing budget. + + For VLM/text models this is typically ``len(sample["input_ids"])``. + For VFM models override with the VAE spatial/temporal formula. + + This method corresponds to the *per-sample* granularity needed by + ``PackingIterableDataset._best_fit_batch``. It differs from + cosmos-rl's ``sft_compute_max_len`` (batch-level) intentionally. + """ + + @abstractmethod + def sft_collate_fn( + self, + samples: list[dict], + max_len: int, + ignore_label_id: int = -100, + ) -> dict: + """Collate a list of packed samples into one training batch. + + Parameters + ---------- + samples: + List of dicts returned by ``sft_process_sample``. + max_len: + Maximum token length in this batch (for padding). + ignore_label_id: + Label value for masked/padding positions (default ``-100``). + + Returns + ------- + dict + Batch ready for ``model.forward()``. + """ diff --git a/cosmos_framework/data/vfm/data_packer_dataloader.py b/cosmos_framework/data/vfm/data_packer_dataloader.py new file mode 100644 index 0000000..5b4777e --- /dev/null +++ b/cosmos_framework/data/vfm/data_packer_dataloader.py @@ -0,0 +1,623 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +""" +OSS-facing dataloader that wires any Python iterable + DataPacker into the +shared PackingIterableDataset engine. + +Follows the same two-layer pattern as the internal path: + private _DataPackerIterableDataset ↔ private _JointIterableDataset + public DataPackerDataLoader ↔ public JointDatasetDynamicBatchingWebLoader + +Data-parallel sharding +---------------------- +When ``torch.distributed`` is initialized, ``DataPackerDataLoader`` automatically +shards ``data_source`` across ranks **and** DataLoader workers using round-robin +filtering — the same pattern as ``SFTDataset`` in +``projects/cosmos3/vfm/datasets/local_datasets/sft_dataset.py``. + +Each ``(dp_rank, worker_id)`` pair sees every +``dp_world_size × num_workers``-th item, giving disjoint coverage. + +Usage +----- +Pass a pre-built iterable directly:: + + loader = DataPackerDataLoader( + data_source=my_dataset, # any Python iterable + data_packer=MyDataPacker(...), + max_tokens=16000, + num_workers=4, + ) + +Or load a HuggingFace / local dataset via ``load_data_source`` — compatible +with Hydra ``LazyCall`` so CLI overrides work without editing Python files:: + + from cosmos_framework.utils.lazy_config import LazyCall as L + from cosmos_framework.data.vfm.data_packer_dataloader import ( + DataPackerDataLoader, + load_data_source, + ) + + dataloader_train = L(DataPackerDataLoader)( + data_source=L(load_data_source)( + name="liuhaotian/LLaVA-Instruct-150K", + split=["train"], + ), + data_packer=L(MyDataPacker)(...), + max_tokens=16000, + ) + + # CLI override (no Python file edit needed): + # dataloader_train.data_source.name=my-org/my-dataset + # dataloader_train.data_source.split=[train,validation] + + # FSDP + TP/PP (pass parallel_dims for correct DP rank): + loader = DataPackerDataLoader( + data_source=..., + data_packer=..., + max_tokens=16000, + parallel_dims=parallel_dims, # uses parallel_dims.dp_coord + ) +""" + +from __future__ import annotations + +import os +from typing import Any + +import numpy as np +import torch +import torch.utils.data + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.data_packer import DataPacker +from cosmos_framework.data.vfm.packing_iterable_dataset import PackingIterableDataset + + +def load_data_source( + name: str, + split: str | list[str] = "train", + subset: str | None = None, + revision: str | None = None, +) -> Any: + """Load a HuggingFace or local dataset for use as ``data_source``. + + Designed to be used as a ``LazyCall`` in Hydra experiment configs so that + dataset name and split can be overridden from the CLI without editing Python + files (see module docstring for an example). + + Parameters + ---------- + name: + HuggingFace dataset name (e.g. ``"liuhaotian/LLaVA-Instruct-150K"``) or + a local directory path to a dataset saved with ``dataset.save_to_disk()``. + Local paths are detected via ``os.path.isdir`` and loaded with + ``load_from_disk``; all other values go through ``load_dataset``. + split: + Split name or list of split names to load. When a list is given the + splits are concatenated into a single dataset. + subset: + HuggingFace dataset subset / config name (optional). + revision: + Git revision / commit hash of the dataset (optional). + + Returns + ------- + datasets.Dataset + A concatenated ``datasets.Dataset`` ready to be passed to + ``DataPackerDataLoader`` as ``data_source``. + + Raises + ------ + ImportError + If the ``datasets`` package is not installed. + """ + try: + from datasets import Dataset, concatenate_datasets, load_dataset, load_from_disk + except ImportError as exc: + raise ImportError( + "The 'datasets' package is required by load_data_source. Install it with: pip install datasets" + ) from exc + + import os + + if os.path.isdir(name): + # Dataset saved with dataset.save_to_disk() — use load_from_disk. + raw = load_from_disk(name) + else: + # HuggingFace Hub name or other format supported by load_dataset. + raw = load_dataset(name, subset, revision=revision) + + if isinstance(raw, Dataset): + # load_from_disk on a single Dataset (not DatasetDict) — return as-is. + return raw + + # DatasetDict: select and concatenate requested splits. + splits = [split] if isinstance(split, str) else split + return concatenate_datasets([raw[s] for s in splits]) + + +class _IterableWrapper(torch.utils.data.IterableDataset): + """Wraps any Python iterable as a ``torch.utils.data.IterableDataset`` + with built-in data-parallel + multi-worker sharding. + + Sharding follows the same ``(dp_rank × num_workers)`` formula as + ``SFTDataset`` — each ``(dp_rank, worker_id)`` pair receives every + ``dp_world_size × num_workers``-th item starting at + ``dp_rank * num_workers + worker_id``. + + .. warning:: + For ``num_workers=0``, worker-level sharding is skipped automatically. + """ + + def __init__(self, iterable: Any, dp_rank: int = 0, dp_world_size: int = 1): + super().__init__() + self._iterable = iterable + self._dp_rank = dp_rank + self._dp_world_size = dp_world_size + + def __iter__(self): + worker_info = torch.utils.data.get_worker_info() + if worker_info is not None: + num_workers = worker_info.num_workers + worker_id = worker_info.id + else: + num_workers, worker_id = 1, 0 + + # Total independent streams = dp_world_size × num_workers. + # Each (rank, worker) pair owns stream = rank * num_workers + worker_id. + total_streams = self._dp_world_size * num_workers + my_stream = self._dp_rank * num_workers + worker_id + + for i, item in enumerate(self._iterable): + if i % total_streams == my_stream: + yield item + + +class _ShuffledMapIterableDataset(torch.utils.data.IterableDataset): + """Stateful, sharded wrapper for map-style ``torch.utils.data.Dataset``. + + Used for ALL map-style ``data_source`` inputs, regardless of ``shuffle``. + Handles DP × worker sharding and stateful checkpoint/resume. + + - Shuffle (``shuffle=True``): per-epoch ``torch.randperm(n)`` seeded with + ``base_seed + epoch``, giving a different but reproducible ordering every epoch. + - No shuffle (``shuffle=False``): sequential iteration ``[0, 1, ..., n-1]`` + each epoch — deterministic and resumable at the exact position. + - Sharding: ``stream_id = dp_rank * num_workers + worker_id``; each stream + yields ``perm[stream_id :: total_streams]`` — disjoint, full coverage. + - Resume: reads ``DP_STATE_WORKER_{worker_id}_EPOCH`` / + ``DP_STATE_WORKER_{worker_id}_INDEX`` env vars set by + ``DataLoaderStateCallback.load_state_dict`` before workers start. + When a dataset ``name`` is provided (non-empty), env vars are namespaced + as ``DP_STATE_{name}_WORKER_{worker_id}_EPOCH`` to avoid conflicts when + multiple ``DataPackerDataLoader`` instances share the same process (e.g. + inside ``JointDataPackerDataLoader``). + + The generator body is lazy: ``worker_info`` (and env vars) are read on the + first ``next()`` call inside the worker process, not at construction time. + Requires ``persistent_workers=True`` and ``fork`` start method (Linux/CUDA + default) — both enforced / documented by ``DataPackerDataLoader``. + """ + + def __init__( + self, + dataset: torch.utils.data.Dataset, + seed: int, + dp_rank: int, + dp_world_size: int, + shuffle: bool = True, + name: str = "", + ) -> None: + super().__init__() + self._dataset = dataset + self._seed = seed + self._dp_rank = dp_rank + self._dp_world_size = dp_world_size + self._shuffle = shuffle + self._name = name + + def __len__(self) -> int: + return len(self._dataset) # type: ignore[arg-type] + + def __iter__(self): + worker_info = torch.utils.data.get_worker_info() + num_workers = worker_info.num_workers if worker_info is not None else 1 + worker_id = worker_info.id if worker_info is not None else 0 + + stream_id = self._dp_rank * num_workers + worker_id + total_streams = self._dp_world_size * num_workers + n = len(self._dataset) # type: ignore[arg-type] + + # os.environ.pop: consume once so a hypothetical second __iter__ call + # in the same worker process defaults to a fresh-start sentinel instead of + # re-fast-forwarding. -1 means "no items seen yet" → start = 0. + # DataLoaderStateCallback always saves index ≥ 0, so -1 is unambiguous. + # When self._name is non-empty, env vars are namespaced to avoid conflicts + # between multiple DataPackerDataLoader instances in the same process. + _pfx = f"DP_STATE_{self._name}_" if self._name else "DP_STATE_" + resume_epoch = int(os.environ.pop(f"{_pfx}WORKER_{worker_id}_EPOCH", 0)) + resume_pos = int(os.environ.pop(f"{_pfx}WORKER_{worker_id}_INDEX", -1)) + + epoch = resume_epoch + while True: + if self._shuffle: + g = torch.Generator().manual_seed(self._seed + epoch) + perm = torch.randperm(n, generator=g).tolist() + else: + perm = list(range(n)) + stream_slice = perm[stream_id::total_streams] + + # resume_pos is the last index successfully included in a training + # batch, so start one past it. On new epochs start from 0. + start = (resume_pos + 1) if epoch == resume_epoch else 0 + for pos in range(start, len(stream_slice)): + item = self._dataset[stream_slice[pos]] + # Attach position metadata; _DataPackerIterableDataset strips + # these before sft_process_sample and re-attaches after so they + # survive through the pool to collate_batch. + yield {"_dp_epoch": epoch, "_dp_stream_pos": pos, **item} + + epoch += 1 + + +class _DataPackerIterableDataset(PackingIterableDataset): + """Private: injects a DataPacker into the shared packing engine. + + Not registered in Hydra directly. Use ``DataPackerDataLoader`` instead. + """ + + def __init__( + self, + data_source: Any, + data_packer: DataPacker, + max_tokens: int, + pool_size: int, + max_batch_size: int, + long_threshold: int, + batching_strategy: str, + dp_rank: int = 0, + dp_world_size: int = 1, + shuffle: bool = False, + seed: int = 0, + name: str = "", + apply_long_sample_halving: bool = True, + ): + is_map = isinstance(data_source, torch.utils.data.Dataset) and not isinstance( + data_source, torch.utils.data.IterableDataset + ) + is_iterable = isinstance(data_source, torch.utils.data.IterableDataset) + if not is_map and not is_iterable: + raise TypeError( + f"data_source must be a torch.utils.data.Dataset or " + f"torch.utils.data.IterableDataset, got {type(data_source).__name__}" + ) + + if is_map: + # All map-style datasets go through _ShuffledMapIterableDataset, + # which handles sharding and position metadata regardless of shuffle. + # This enables stateful checkpoint/resume even when shuffle=False. + data_source = _ShuffledMapIterableDataset( + dataset=data_source, + seed=seed, + dp_rank=dp_rank, + dp_world_size=dp_world_size, + shuffle=shuffle, + name=name, + ) + self._has_dp_meta = True + else: + # Iterable-style: wrap with _IterableWrapper for sharding only. + # Stateful resume is not supported for IterableDataset sources. + data_source = _IterableWrapper(data_source, dp_rank=dp_rank, dp_world_size=dp_world_size) + self._has_dp_meta = False + + datasets_cfg = {"default": {"dataset": data_source, "ratio": 1.0}} + super().__init__( + datasets_cfg=datasets_cfg, + max_tokens=max_tokens, + pool_size=pool_size, + max_batch_size=max_batch_size, + long_threshold=long_threshold, + batching_strategy=batching_strategy, + apply_long_sample_halving=apply_long_sample_halving, + ) + self._data_packer = data_packer + + def _get_next_sample(self) -> dict: + raw_item = super()._get_next_sample() + if self._has_dp_meta: + # Strip _dp_* keys before sft_process_sample so the user's packer + # receives a clean item, then re-attach so metadata survives the pool. + dp_meta = {k: raw_item.pop(k) for k in list(raw_item) if k.startswith("_dp_")} + processed = self._data_packer.sft_process_sample(raw_item) + processed.update(dp_meta) + return processed + return self._data_packer.sft_process_sample(raw_item) + + def compute_sample_tokens(self, sample: dict) -> int: + return self._data_packer.compute_num_tokens(sample) + + def collate_batch(self, samples: list) -> dict: + max_len = max(self.compute_sample_tokens(s) for s in samples) + + if self._has_dp_meta and "_dp_epoch" in samples[0]: + max_epoch = max(s["_dp_epoch"] for s in samples) + max_pos = max(s["_dp_stream_pos"] for s in samples) + clean = [{k: v for k, v in s.items() if not k.startswith("_dp_")} for s in samples] + batch = self._data_packer.sft_collate_fn(clean, max_len) + worker_info = torch.utils.data.get_worker_info() + worker_id = worker_info.id if worker_info is not None else 0 + batch["sample_worker_id"] = torch.tensor([worker_id] * len(samples)) + batch["sample_epoch"] = torch.tensor([max_epoch] * len(samples)) + batch["sample_index"] = torch.tensor([max_pos] * len(samples)) + else: + batch = self._data_packer.sft_collate_fn(samples, max_len) + + return batch + + +class DataPackerDataLoader(torch.utils.data.DataLoader): + """Public OSS entry point for bringing any dataset into i4 training. + + Wraps ``_DataPackerIterableDataset`` in a standard + ``torch.utils.data.DataLoader`` — no WebDataset dependency required. + OSS users' data can be HuggingFace datasets, local files, generators, + or any Python iterable. + + Data-parallel sharding is automatic when ``torch.distributed`` is + initialized. Each ``(dp_rank, worker_id)`` pair receives a disjoint + subset of ``data_source``. + + Parameters + ---------- + data_source: + ``torch.utils.data.Dataset`` (map-style) or + ``torch.utils.data.IterableDataset`` — HuggingFace datasets, custom + datasets, or generators wrapped in an ``IterableDataset``. Plain + lists / generators are not accepted; wrap them in an ``IterableDataset`` + first. + data_packer: + A ``DataPacker`` subclass instance. Provides sample-level transform + (``sft_process_sample``), token counting (``compute_num_tokens``), and + batch collation (``sft_collate_fn``). + max_tokens: + Token budget per batch. + pool_size: + Samples to buffer before bin-packing. + max_batch_size: + Hard cap on items per batch. + long_threshold: + Samples with token count >= this are emitted as singleton batches. + batching_strategy: + ``"prefer_closest"`` (default) or ``"prefer_first"``. + shuffle: + If ``True`` and ``data_source`` is a map-style ``Dataset``, shuffle + samples with a per-epoch ``torch.randperm`` seeded by ``seed + epoch``. + Enables stateful checkpoint/resume via ``DataLoaderStateCallback`` + (``distributor_type="data_packer"``). Has no effect for + ``IterableDataset`` inputs — a warning is logged in that case. + seed: + Base seed for the per-epoch shuffle permutation. Epoch ``e`` uses + ``seed + e`` as the generator seed. Ignored when ``shuffle=False``. + num_workers, prefetch_factor, persistent_workers, pin_memory: + Forwarded to ``torch.utils.data.DataLoader``. When ``shuffle=True`` + and ``num_workers > 0``, ``persistent_workers`` is automatically + promoted to ``True`` (required for correct resume behaviour). + parallel_dims: + Optional ``ParallelDims`` instance (from cosmos-rl). When provided, + ``parallel_dims.dp_coord`` supplies the data-parallel rank and world + size, which is correct for FSDP+TP/PP where the DP degree differs from + the global world size. When ``None`` (default), rank info is read from + ``torch.distributed`` if initialized, else defaults to ``(0, 1)``. + name: + Optional identifier used to namespace resume env vars when multiple + ``DataPackerDataLoader`` instances share the same process (e.g. inside + ``JointDataPackerDataLoader``). When non-empty, env vars become + ``DP_STATE_{name}_WORKER_{id}_EPOCH/INDEX`` instead of the default + ``DP_STATE_WORKER_{id}_EPOCH/INDEX``. Must match the ``name`` passed + to the corresponding ``DataLoaderStateCallback`` or + ``JointDataLoaderStateCallback``. Leave empty (default) for + single-loader configurations. + apply_long_sample_halving: + When ``True`` (default), the inner ``PackingIterableDataset._max_tokens`` + halves the budget for any batch whose largest sample has >= 1000 tokens + — a memory-safety heuristic. Set ``False`` to use the literal + ``max_tokens`` budget unconditionally; only do this when memory + headroom at the un-halved budget has been validated for the recipe + (large MoT + LoRA recipes can OOM at the literal budget — see + ``packing_iterable_dataset.py::_max_tokens``). + """ + + def __init__( + self, + data_source: Any, + data_packer: DataPacker, + max_tokens: int, + pool_size: int = 16, + max_batch_size: int = 1, + long_threshold: int = 6400, + batching_strategy: str = "prefer_closest", + shuffle: bool = False, + seed: int = 0, + num_workers: int = 0, + prefetch_factor: int | None = None, + persistent_workers: bool = False, + pin_memory: bool = False, + parallel_dims=None, + name: str = "", + apply_long_sample_halving: bool = True, + ): + is_map = isinstance(data_source, torch.utils.data.Dataset) and not isinstance( + data_source, torch.utils.data.IterableDataset + ) + is_iterable = isinstance(data_source, torch.utils.data.IterableDataset) + if shuffle and is_iterable: + log.warning( + "DataPackerDataLoader: shuffle=True has no effect for IterableDataset " + "data_source. Shuffle the dataset before passing it in.", + rank0_only=True, + ) + + # Correctness requirement: map-style datasets use _ShuffledMapIterableDataset + # which reads resume env vars on the first __iter__ call inside each worker. + # With persistent_workers=False, workers re-spawn each iteration and + # re-inherit the env vars, causing incorrect fast-forward on every epoch + # boundary. Enforce persistent_workers=True for all map-style datasets. + if is_map and num_workers > 0 and not persistent_workers: + log.info( + "DataPackerDataLoader: map-style data_source requires persistent_workers=True " + "for correct stateful resume behaviour. Overriding persistent_workers to True.", + rank0_only=True, + ) + persistent_workers = True + + # Resolve data-parallel rank and world-size. + # Priority: explicit parallel_dims > torch.distributed > single-GPU default. + if parallel_dims is not None: + dp_rank, dp_world_size = parallel_dims.dp_coord + elif torch.distributed.is_initialized(): + dp_rank = torch.distributed.get_rank() + dp_world_size = torch.distributed.get_world_size() + + # rank/world_size differ from the data-parallel rank/world_size. + # Pass `parallel_dims` to use the correct DP coordinates; otherwise + # data sharding will be incorrect (each logical DP group sees the + # same shard as another group). + if dp_world_size > 1: + log.info( + "DataPackerDataLoader: using global rank for DP sharding. " + "For FSDP+TP/PP setups pass parallel_dims= to use the correct " + "DP rank/world_size.", + rank0_only=True, + ) + else: + dp_rank, dp_world_size = 0, 1 + + dataset = _DataPackerIterableDataset( + data_source=data_source, + data_packer=data_packer, + max_tokens=max_tokens, + pool_size=pool_size, + max_batch_size=max_batch_size, + long_threshold=long_threshold, + batching_strategy=batching_strategy, + dp_rank=dp_rank, + dp_world_size=dp_world_size, + shuffle=shuffle, + seed=seed, + name=name, + apply_long_sample_halving=apply_long_sample_halving, + ) + loader_kwargs: dict = dict( + num_workers=num_workers, + persistent_workers=persistent_workers and num_workers > 0, + pin_memory=pin_memory, + ) + if num_workers > 0 and prefetch_factor is not None: + loader_kwargs["prefetch_factor"] = prefetch_factor + # batch_size=None disables PyTorch's automatic batching/collation. + # _DataPackerIterableDataset.__iter__ already yields fully-collated batch dicts; + # letting the DataLoader re-collate them adds spurious batch dimensions. + super().__init__(dataset, batch_size=None, **loader_kwargs) + + +class JointDataPackerDataLoader: + """Wraps multiple ``DataPackerDataLoader`` instances with ratio-based seeded selection. + + Mirrors the design of ``IterativeJointDataLoader``: one output batch = one + inner loader, selected deterministically by ratio at each step. Adds a + ``"dataset_name"`` key to every yielded batch so downstream callbacks can + route state updates to the correct inner loader. + + Parameters + ---------- + dataloaders: + ``{name: {"dataloader": DataPackerDataLoader, "ratio": int}}`` mapping. + Entries with ``ratio <= 0`` are silently skipped. + seed: + Base seed for the per-step dataset selection. Step ``i`` uses + ``np.random.RandomState(seed + i)`` to pick the inner loader index, + giving the same sequence on every rank (assuming synchronized + ``set_start_iteration`` calls) and fully reproducible resume. + + Stateful checkpoint/resume + -------------------------- + Pair with ``JointDataLoaderStateCallback`` (from + ``cosmos_framework.callbacks.dataloader_state``). That callback saves the outer + ``global_id`` and each inner loader's per-worker ``(epoch, index)`` state + in a single DCP checkpoint entry. On resume: + + 1. ``JointDataLoaderStateCallback.load_state_dict`` calls + ``set_start_iteration(global_id)`` to restore the selection sequence. + 2. Each inner ``DataLoaderStateCallback.load_state_dict`` sets namespaced + env vars so inner-loader workers fast-forward to the saved position. + + Each ``DataPackerDataLoader`` must be constructed with a unique ``name`` + that matches the key used in this ``dataloaders`` dict so env vars are + namespaced correctly (see ``DataPackerDataLoader`` ``name`` parameter). + """ + + def __init__( + self, + dataloaders: dict[str, dict], + seed: int = 42, + ) -> None: + entries = [ + (name, cfg["dataloader"], cfg["ratio"]) + for name, cfg in dataloaders.items() + if cfg.get("ratio", 0) > 0 + ] + if not entries: + raise ValueError("JointDataPackerDataLoader: no dataloaders with ratio > 0") + + self._names: list[str] = [e[0] for e in entries] + if "global_id" in self._names: + raise ValueError( + "JointDataPackerDataLoader: dataset name 'global_id' is reserved " + "by the checkpoint state format; use a different name." + ) + self._loaders: list[DataPackerDataLoader] = [e[1] for e in entries] + ratios = np.array([e[2] for e in entries], dtype=float) + self._probs: np.ndarray = ratios / ratios.sum() + self._seed = seed + self._global_id = 0 + # Iterators are created lazily on the first __iter__ call so that + # DataLoaderStateCallback.load_state_dict can install resume env vars + # before workers are spawned (for num_workers > 0, iter(DataLoader) + # forks workers immediately; env vars must be set in the parent first). + self._iterators: list | None = None + + total = ratios.sum() + lines = [f"JointDataPackerDataLoader: {len(self._names)} streams"] + for name, ratio in zip(self._names, ratios): + lines.append(f" {name}: ratio={ratio:.4g} ({ratio / total:.1%})") + log.info("\n".join(lines)) + + def set_start_iteration(self, iteration: int) -> None: + """Restore deterministic selection sequence after checkpoint resume. + + Called by ``JointDataLoaderStateCallback.load_state_dict`` and by the + trainer (if present) via ``hasattr`` guard. + """ + self._global_id = iteration + + def __iter__(self): + # Lazy init: create iterators here (not in __init__) so that + # load_state_dict can set resume env vars before workers fork. + if self._iterators is None: + self._iterators = [iter(loader) for loader in self._loaders] + while True: + rng = np.random.RandomState(self._seed + self._global_id) + idx = int(rng.choice(len(self._loaders), p=self._probs)) + try: + batch = next(self._iterators[idx]) + except StopIteration: + # Inner DataPackerDataLoaders are infinite; this guard handles + # the unlikely case of a finite IterableDataset inner source. + self._iterators[idx] = iter(self._loaders[idx]) + batch = next(self._iterators[idx]) + batch["dataset_name"] = self._names[idx] + self._global_id += 1 + yield batch diff --git a/cosmos_framework/data/vfm/dataset_provider.py b/cosmos_framework/data/vfm/dataset_provider.py new file mode 100644 index 0000000..8f14603 --- /dev/null +++ b/cosmos_framework/data/vfm/dataset_provider.py @@ -0,0 +1,253 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Callable, Optional + +import omegaconf +from webdataset.handlers import warn_and_continue + +import imaginaire.datasets.webdataset.decoders.image as image_decoders +import imaginaire.datasets.webdataset.decoders.pickle as pickle_decoders +import imaginaire.datasets.webdataset.distributors as distributors +import cosmos_framework.data.vfm.decoders.video_decoder as video_decoder +import cosmos_framework.data.vfm.webdataset as webdataset +from imaginaire.datasets.webdataset.config.schema import DatasetConfig +from cosmos_framework.utils.lazy_config import LazyCall as L +from cosmos_framework.utils.lazy_config import LazyDict +from cosmos_framework.data.vfm.augmentor_provider import AUGMENTOR_OPTIONS +from cosmos_framework.data.vfm.augmentors import sequence_plan as _sequence_plan +from cosmos_framework.data.vfm.data_sources.data_registration import DATASET_OPTIONS +from cosmos_framework.data.vfm.utils import IMAGE_RES_SIZE_INFO, VIDEO_RES_SIZE_INFO + + +def get_video_dataset( + dataset_name: str, + video_decoder_name: str, + resolution: str, + is_train: bool = True, + num_video_frames: int = 121, + chunk_size: int = 0, + min_fps_thres: int = 10, + max_fps_thres: int = 60, + dataset_resolution_type: str = "all", + augmentor_name: str = "video_basic_augmentor_v1", + object_store: Optional[str] = "s3", + caption_type: str = "t2w_qwen2p5_7b", + embedding_type: str = "t5_xxl", + detshuffle: bool = False, + long_caption_ratio: int = 7, + medium_caption_ratio: int = 2, + short_caption_ratio: int = 1, + user_caption_ratio: int = 90, + dataset_info_fn: Optional[Callable] = None, + use_native_fps: bool = True, + use_original_fps: bool = False, + tokenizer_config: Optional[LazyDict] = None, + cfg_dropout_rate: float = 0.0, + caption_config: dict | None = None, + append_duration_fps_timestamps: bool = True, + append_resolution_info: bool = True, + use_dynamic_fps: bool = False, + low_fps_bias: float = 0.5, + min_frames: int | None = None, + max_frames: int | None = None, + resize_on_read: bool = False, + conditioning_config: Optional[dict[int, float]] = None, + uniform_conditioning: bool = False, + temporal_compression_factor: int = 4, + sound_generation_mode: str = "t2vs", + audio_sample_rate: int = 48000, + key_renames: dict[str, str] | None = None, +) -> omegaconf.dictconfig.DictConfig: + assert resolution in VIDEO_RES_SIZE_INFO.keys(), "The provided resolution cannot be found in VIDEO_RES_SIZE_INFO." + assert object_store in [ + "s3", + "swiftstack", + "gcp", + "pdx_cosmos_gen", + "team_sil_gws_data", + False, + ], "We support s3, swiftstack, gcp, pdx_cosmos_gen, team_sil_gws_data or False for local loading." + basic_augmentor_names = [ + "video_basic_augmentor_v2", + "video_basic_augmentor_v2_with_control", + "video_basic_augmentor_v2_with_tokenization", + "noframedrop_nocameramove_video_augmentor_v1", + "video_basic_augmentor_v3", + "video_basic_augmentor_v3_with_audio", + ] + if video_decoder_name == "video_naive_bytes": + assert augmentor_name in basic_augmentor_names, ( + "We can only use video_basic_augmentor_v2 with video_naive_bytes decoder." + ) + if augmentor_name in basic_augmentor_names: + assert video_decoder_name == "video_naive_bytes", ( + "We can only use video_naive_bytes decoder with video_basic_augmentor_v2." + ) + + assert dataset_resolution_type in [ + "all", + "gt480p", + "gt720p", + "gt1080p", + ], f"The provided dataset resolution type {dataset_resolution_type} is not supported." + # dataset_resolution_type + # -- all - uses all dataset resolutions + # -- gt720p - Uses only resolutions >= 720p + # -- gt1080p - Uses only resolutions >= 1080p + if not object_store: + assert dataset_info_fn is not None, "dataset_info_fn is required for local loading." + dataset_info = dataset_info_fn() + else: + dataset_info_fn = DATASET_OPTIONS[dataset_name] + dataset_info = dataset_info_fn( + object_store, caption_type, embedding_type, dataset_resolution_type, min_frames, max_frames + ) + + augmentator_kwargs = {} + if augmentor_name in ("video_basic_augmentor_v3", "video_basic_augmentor_v3_with_audio"): + augmentator_kwargs["resize_on_read"] = resize_on_read + if augmentor_name == "video_basic_augmentor_v3_with_audio": + augmentator_kwargs["audio_sample_rate"] = audio_sample_rate + augmentator_kwargs["sound_generation_mode"] = sound_generation_mode + if key_renames: + augmentator_kwargs["key_renames"] = key_renames + + # v3 augmentors handle conditioning natively via **kwargs; all others get post-factory injection. + _V3_AUGMENTORS = ("video_basic_augmentor_v3", "video_basic_augmentor_v3_with_audio") + if (conditioning_config is not None or uniform_conditioning) and augmentor_name in _V3_AUGMENTORS: + augmentator_kwargs["conditioning_config"] = conditioning_config + augmentator_kwargs["uniform_conditioning"] = uniform_conditioning + augmentator_kwargs["temporal_compression_factor"] = temporal_compression_factor + + augmentor = AUGMENTOR_OPTIONS[augmentor_name]( + resolution=resolution, + caption_type=caption_type, + embedding_type=embedding_type, + min_fps=min_fps_thres, + max_fps=max_fps_thres, + long_caption_ratio=long_caption_ratio, + medium_caption_ratio=medium_caption_ratio, + short_caption_ratio=short_caption_ratio, + user_caption_ratio=user_caption_ratio, + num_video_frames=num_video_frames, + use_native_fps=use_native_fps, + use_original_fps=use_original_fps, + tokenizer_config=tokenizer_config, + cfg_dropout_rate=cfg_dropout_rate, + caption_config=caption_config, + append_duration_fps_timestamps=append_duration_fps_timestamps, + append_resolution_info=append_resolution_info, + use_dynamic_fps=use_dynamic_fps, + low_fps_bias=low_fps_bias, + dataset_resolution_type=dataset_resolution_type, + **augmentator_kwargs, + ) + if (conditioning_config is not None or uniform_conditioning) and augmentor_name not in _V3_AUGMENTORS: + augmentor["sequence_plan"] = L(_sequence_plan.SequencePlanAugmentor)( + input_keys=["video"], + args={ + "conditioning_config": conditioning_config, + "uniform_conditioning": uniform_conditioning, + "temporal_compression_factor": temporal_compression_factor, + }, + ) + + distributor = distributors.ShardlistMultiAspectRatio( + shuffle=True, + split_by_node=True, + split_by_worker=True, + resume_flag=True, + verbose=False, + is_infinite_loader=is_train, + ) + + video_data_config = DatasetConfig( + keys=[], # use the per_dataset_keys in DatasetInfo instead + buffer_size=25, + streaming_download=True, + dataset_info=dataset_info, + distributor=distributor, + decoders=[ + video_decoder.construct_video_decoder( + video_decoder_name=video_decoder_name, + sequence_length=num_video_frames, + chunk_size=chunk_size, + min_fps_thres=min_fps_thres, + max_fps_thres=max_fps_thres, + ), + pickle_decoders.pkl_decoder, + ], + augmentation=augmentor, + remove_extension_from_keys=True, + sample_keys_full_list_path=None, + ) + + return webdataset.Dataset(config=video_data_config, decoder_handler=warn_and_continue, detshuffle=detshuffle) + + +def get_image_dataset( + dataset_name: str, + resolution: str, + dataset_resolution_type: str = "all", + is_train: bool = True, + augmentor_name: str = "image_basic_augmentor", + object_store: str = "s3", + detshuffle: bool = False, + caption_type: str = "ai_v3p1", + embedding_type: str = "t5_xxl", + train_on_captions: list[str] = [], + tokenizer_config: Optional[LazyDict] = None, + cfg_dropout_rate: float = 0.0, + append_resolution_info: bool = True, +) -> omegaconf.dictconfig.DictConfig: + assert resolution in IMAGE_RES_SIZE_INFO.keys(), "The provided resolution cannot be found in IMAGE_RES_SIZE_INFO." + assert object_store in ["s3", "swiftstack", "gcp"], "We support s3, gcp and swiftstack only." + assert dataset_resolution_type in [ + "all", + "gt480p", + "gt720p", + "gt1080p", + ], f"The provided dataset resolution type {dataset_resolution_type} is not supported." + # dataset_resolution_type + # -- all - uses all dataset resolutions + # -- gt480p - Uses only resolutions >= 480p + # -- gt720p - Uses only resolutions >= 720p + # -- gt1080p - Uses only resolutions >= 1080p + dataset_info_fn = DATASET_OPTIONS[dataset_name] + dataset_info = dataset_info_fn(object_store, caption_type, embedding_type, dataset_resolution_type) + augmentation = AUGMENTOR_OPTIONS[augmentor_name]( + resolution=resolution, + caption_type=caption_type, + embedding_type=embedding_type, + train_on_captions=train_on_captions, + tokenizer_config=tokenizer_config, + cfg_dropout_rate=cfg_dropout_rate, + dataset_resolution_type=dataset_resolution_type, + append_resolution_info=append_resolution_info, + ) + + distributor = distributors.ShardlistMultiAspectRatio( + shuffle=True, + split_by_node=True, + split_by_worker=True, + resume_flag=True, + verbose=False, + is_infinite_loader=is_train, + ) + + image_data_config = DatasetConfig( + keys=[], + + buffer_size=25, + streaming_download=True, + dataset_info=dataset_info, + distributor=distributor, + decoders=[ + image_decoders.pil_loader, + pickle_decoders.pkl_decoder, + ], + augmentation=augmentation, + ) + + return webdataset.Dataset(config=image_data_config, detshuffle=detshuffle) diff --git a/cosmos_framework/data/vfm/joint_dataloader.py b/cosmos_framework/data/vfm/joint_dataloader.py new file mode 100644 index 0000000..56b13e7 --- /dev/null +++ b/cosmos_framework/data/vfm/joint_dataloader.py @@ -0,0 +1,992 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from collections import deque +from dataclasses import dataclass +from typing import Any, ClassVar, Dict, Union + +import numpy as np +import torch +import webdataset +from torch.utils.data.dataloader import default_collate + +from cosmos_framework.utils.lazy_config import instantiate +from cosmos_framework.utils import log + +_TIMING_KEYS = {"_sample_time", "_aug_time", "_pre_aug_time", "_aug_step_times"} +_BATCH_TIMING_KEYS = { + "_worker_batch_time", + "_worker_aug_time", + "_worker_io_time", + "_worker_aug_step_times", + "_worker_id", +} + + +def custom_collate_fn(batch): + """ + Collate function that works like default_collate for all keys other than "text_token_ids", "images", and "video". + For "text_token_ids", "images", and "video" it simply returns them in a list, instead of stacking them as a tensor. + """ + list_collate_keys = { + "text_token_ids", + "images", + "video", + "action", + "domain_id", + "sequence_plan", + "sound", + "raw_action_dim", + "image_size", + } + + # Data keys where a per-sample value of ``None`` is a meaningful signal + # (e.g. audio extraction failed for that sample → ``sound=None`` paired + # with ``plan.has_sound=False``). These keys must be kept as a list with + # ``None`` placeholders so the model can align per-sample data 1:1 with + # per-sample plans. Dropping the entire key on any None would leave the + # remaining sound tensors mis-aligned with the plans whose ``has_sound`` + # flag was set BEFORE collation, causing ``sequence_packing`` to index + # past the end of ``x0_tokens_sound``. + sparse_data_keys = {"sound"} + + # Handle the case where the batch is already a dictionary (e.g. column-wise batching) + if isinstance(batch, dict): + return {key: (value if key in list_collate_keys else default_collate(value)) for key, value in batch.items()} + + # Handle standard list of samples + elem = batch[0] + if isinstance(elem, dict): + + # Some Action datasets add optional metadata keys (for example + # ``additional_view_description`` for concat-view captions) only for a + # subset of samples. PyTorch can batch such samples together when + # DataLoader batch_size > 1; collating only elem's keys and indexing + # every sample by that key turns the optional field into a fatal + # KeyError. Use the union of keys and skip optional keys that are not + # present in every sample. Required training keys still fail loudly via + # downstream assertions if actually missing. + result = {} + keys = set().union(*(d.keys() for d in batch)) + for key in keys: + if key in _TIMING_KEYS: + continue + values = [d.get(key) for d in batch] + if any(value is None for value in values): + # Sparse data keys keep their None placeholders to preserve + # 1:1 alignment with sequence_plan. Other (optional metadata) + # keys not present in every sample are dropped. + if key in sparse_data_keys: + result[key] = values + continue + if key in list_collate_keys: + result[key] = values + else: + result[key] = default_collate(values) + result.update(_aggregate_worker_timing(batch)) + return result + else: + return default_collate(batch) + + +def _aggregate_worker_timing(samples: list[dict]) -> dict: + """Extract per-sample timing keys, aggregate into per-batch scalars.""" + info: dict[str, float | int] = {} + if "_sample_time" in samples[0]: + info["_worker_batch_time"] = sum(s.get("_sample_time", 0.0) for s in samples) + if "_aug_time" in samples[0]: + aug_total = sum(s.get("_aug_time", 0.0) for s in samples) + info["_worker_aug_time"] = aug_total + if "_worker_batch_time" in info: + info["_worker_io_time"] = info["_worker_batch_time"] - aug_total + if "_aug_step_times" in samples[0]: + agg: dict[str, float] = {} + for s in samples: + for step_name, t in s.get("_aug_step_times", {}).items(): + agg[step_name] = agg.get(step_name, 0.0) + t + info["_worker_aug_step_times"] = agg + worker_info = torch.utils.data.get_worker_info() + info["_worker_id"] = worker_info.id if worker_info is not None else 0 + return info + + +@dataclass +class _PackingMetrics: + """Per-batch packing statistics collected during the packing loop. + + Also serves as the single source of truth for packing-related metric names + via ``STATS_SPEC``, which the dataloading monitor callback consumes to + drive accumulation and logging. + """ + + current_sequence_length: int = 0 + num_samples: int = 0 + dropped_count: int = 0 + from_buffer: int = 0 + from_workers: int = 0 + + STATS_SPEC: ClassVar[list[tuple[str, str, str]]] = [ + # (batch_key, wandb_suffix, aggregation_type) + ("_num_tokens", "token_fraction", "scalar"), + ("_num_samples", "samples_per_batch", "list"), + ("_from_buffer", "from_buffer", "list"), + ("_from_workers", "from_workers", "list"), + ("_buffer_size", "buffer_size", "list"), + ("_dropped_count", "dropped", "scalar"), + ] + + def attach_to(self, output_batch: dict, buffer_size: int) -> None: + """Write packing statistics into the output batch dict.""" + output_batch["_num_tokens"] = self.current_sequence_length + output_batch["_num_samples"] = self.num_samples + output_batch["_from_buffer"] = self.from_buffer + output_batch["_from_workers"] = self.from_workers + output_batch["_buffer_size"] = buffer_size + output_batch["_dropped_count"] = self.dropped_count + + +class JointDataLoader(webdataset.WebLoader): + r""" + A joint dataloader that supports loading both images and videos. + """ + + _DEFAULT_LOOKAHEAD_LIMIT: ClassVar[int] = 10 + + def __init__( + self, + dataloaders: Dict[str, Dict[str, Union[torch.utils.data.DataLoader, webdataset.WebLoader, int]]], + tokenizer_spatial_compression_factor: int, + tokenizer_temporal_compression_factor: int, + patch_spatial: int, + max_sequence_length: int | None, + max_samples_per_batch: int | None, + sound_latent_fps: float = 0, + audio_sample_rate: int = 48000, + prewarm: bool = True, + default_lookahead_limit: int = _DEFAULT_LOOKAHEAD_LIMIT, + lookahead_limits: Dict[str, int] | None = None, + ): + """ + Initialize the JointDataLoader with multiple datasets. + + The effective mini-batch size can be controlled with either max_sequence_length or + max_samples_per_batch. To use max_sequence_length, max_samples_per_batch needs to be None. + Vice versa, to use max_samples_per_batch, max_sequence_length needs to be None. + max_sequence_length and max_samples_per_batch cannot both be None simultaneously. + + Args: + dataloaders: key - dataset_name; value - {"dataloader": dataloader, "ratio": data_ratio} + tokenizer_spatial_compression_factor: The spatial compression factor of the tokenizer. + tokenizer_temporal_compression_factor: The temporal compression factor of the tokenizer. + patch_spatial: Spatial pathification factor. + max_samples_per_batch: Max number of samples per packed batch (alternative to max_sequence_length). + sound_latent_fps: Sound tokenizer latent rate in Hz (e.g. 25). If 0, sound tokens are not counted. + audio_sample_rate: Audio sample rate in Hz (e.g. 48000). Used with sound_latent_fps to estimate + sound token count. + default_lookahead_limit: Packing-loop look-ahead fallback for dataloaders not in + ``lookahead_limits``. + lookahead_limits: Optional ``{dataset_name: int}`` per-dataloader override. + + Example: + joint_loader = IterativeJointDataLoader( + dataloaders{ + "image_data": { + "dataloader": webdataset.WebLoader(...), + "ratio": 4, + }, + "video_data": { + "dataloader": torch.utils.data.DataLoader(...), + "ratio": 1, + }, + } + ) + """ + self.dataloader_list, self.dataset_name_list, self.data_ratios = [], [], [] + self.lookahead_limits: list[int] = [] + self.tokenizer_spatial_compression_factor = tokenizer_spatial_compression_factor + self.tokenizer_temporal_compression_factor = tokenizer_temporal_compression_factor + self.patch_spatial = patch_spatial + self.max_sequence_length = max_sequence_length + self.max_samples_per_batch = max_samples_per_batch + self.sound_latent_fps = sound_latent_fps + self.audio_sample_rate = audio_sample_rate + self.default_lookahead_limit = int(default_lookahead_limit) + + assert (self.max_sequence_length is None) != (self.max_samples_per_batch is None), ( + "Exactly one of max_sequence_length or max_samples_per_batch must be None, but not both." + ) + + _lookahead_overrides: Dict[str, int] = dict(lookahead_limits) if lookahead_limits else {} + unknown = set(_lookahead_overrides) - set(dataloaders) + assert not unknown, f"lookahead_limits references unknown dataloaders {unknown}; valid: {sorted(dataloaders)}" + + for dataset_name, dataloader_data in dataloaders.items(): + assert set(dataloader_data.keys()) == {"dataloader", "ratio"}, f"Invalid config: {dataloader_data}" + if dataloader_data["ratio"] <= 0: + continue + self.dataset_name_list.append(dataset_name) + self.dataloader_list.append(instantiate(dataloader_data["dataloader"], collate_fn=custom_collate_fn)) + self.data_ratios.append(dataloader_data["ratio"]) + self.lookahead_limits.append(int(_lookahead_overrides.get(dataset_name, self.default_lookahead_limit))) + + self.global_id = 0 + self.ratio_sum = sum(self.data_ratios) + + total = self.ratio_sum if self.ratio_sum > 0 else 1.0 + lines = [f"JointDataLoader: {len(self.dataset_name_list)} streams"] + for name, ratio in zip(self.dataset_name_list, self.data_ratios): + lines.append(f" {name}: ratio={ratio:.4g} ({ratio / total:.1%})") + log.info("\n".join(lines)) + + self.data_len = 0 + self.dataloaders = [iter(dataloader) for dataloader in self.dataloader_list] + self.buffers = [deque() for _ in range(len(self.dataloader_list))] + for data in self.dataloader_list: + self.data_len += len(data) + + # Pre-warm all dataloaders: force worker process spawning and first + # batch loading so that slow dataset initialisation (e.g. action + # datasets with spawn workers) happens here rather than mid-training + # where it would cause NCCL collective timeouts. + if prewarm: + self._prewarm_dataloaders() + else: + log.info( + "JointDataLoader: prewarm DISABLED (debug mode); first iteration may incur per-stream cold-load cost" + ) + + def _prewarm_dataloaders(self) -> None: + """Force all dataloader iterators to spawn workers and produce one batch. + + The first ``next()`` call on an ``InfiniteDataLoader`` iterator triggers + ``DataLoader.__iter__()`` which spawns worker processes. For action + dataloaders using ``multiprocessing_context='spawn'``, each worker must + fully initialise heavy datasets (BridgeOrigLeRobotDataset, EMBODIMENT_A, etc.) + from scratch. If this happens lazily during training, the resulting + delay (potentially minutes) causes NCCL collective timeouts when faster + ranks enter the forward pass while slower ranks are still loading data. + + By pulling one batch from every dataloader here — before any training + iteration — we ensure all workers are alive and warmed up. The fetched + samples are pushed into the per-dataloader buffer so they are consumed + normally by the first iteration that selects that dataloader. + + A ``dist.barrier()`` at the end synchronises all ranks so that training + only begins once every rank has finished pre-warming. + """ + import time + + for i, (name, dl_iter) in enumerate(zip(self.dataset_name_list, self.dataloaders)): + t0 = time.monotonic() + try: + batch = next(dl_iter) + except StopIteration: + log.warning(f"Pre-warm: dataloader {name!r} is empty, skipping") + continue + elapsed = time.monotonic() - t0 + + # Split the collated batch into individual samples and push them + # into the buffer — identical to the splitting logic in + # _get_next_sample — so the samples are not wasted. + is_image_batch = "images" in batch + input_images_or_videos = batch["images" if is_image_batch else "video"] + batch_size = len(input_images_or_videos) + + for j in range(batch_size): + sample = {} + for k, v in batch.items(): + if k in _BATCH_TIMING_KEYS: + sample[k] = v + elif isinstance(v, list) and k in self._MULTI_ITEM_KEYS: + elem = v[j] + if isinstance(elem, list): + sample[k] = elem + else: + sample[k] = v[j : j + 1] + elif isinstance(v, list): + sample[k] = v[j] + elif isinstance(v, torch.Tensor) and v.dim() > 0: + sample[k] = v[j : j + 1] + else: + sample[k] = v[j : j + 1] + self.buffers[i].append(sample) + + log.info( + f"Pre-warm: dataloader {name!r} ready — {batch_size} samples buffered in {elapsed:.1f}s", + rank0_only=False, + ) + + # Synchronise so training only starts once every rank is warmed up. + if torch.distributed.is_initialized(): + log.info("Pre-warm: waiting at barrier for all ranks …") + torch.distributed.barrier() + log.info("Pre-warm: all ranks ready") + + def _compute_num_tokens_per_sample(self, data_batch: dict) -> int: + """ + This function computes the number of tokens per sample in the data batch. + This includes text + vision generation tokens + action tokens. + + Args: + data_batch (dict): The data batch containing the text tokens. + + Returns: + int: The number of tokens per sample. + """ + + # The token sequence we have is + # [] + # The spatial dimension of image tokens is compressed by + # vae spatial downsampling factor + pathification + # The temporal dimension of image tokens is compressed by + # vae temporal downsampling factor + # Action tokens have 1 token per time step (no spatial dimension) + + text_token_ids = data_batch["text_token_ids"] + if isinstance(text_token_ids, list): + num_text_tokens = text_token_ids[0].shape[0] + else: + num_text_tokens = text_token_ids.shape[1] + + num_tokens = num_text_tokens + 1 + + # Vision part + is_image_batch = "images" in data_batch + input_images_or_videos = data_batch["images" if is_image_batch else "video"] + + # iterate over all the media in the batch + for media in input_images_or_videos if isinstance(input_images_or_videos, list) else [input_images_or_videos]: + if is_image_batch: + _, H, W = media.shape + T = 1 + else: + _, T, H, W = media.shape + + vae_spatial_downsample = self.tokenizer_spatial_compression_factor * self.patch_spatial + vae_temporal_downsample = self.tokenizer_temporal_compression_factor + + latent_h_shape = H // vae_spatial_downsample + latent_w_shape = W // vae_spatial_downsample + latent_t_shape = 1 + (T - 1) // vae_temporal_downsample + + num_vision_tokens = latent_h_shape * latent_w_shape * latent_t_shape + 2 + num_tokens += num_vision_tokens + + # Action part: each action time step is 1 token. + # Action tensor shape is (T_action, D) per sample; stored as a single-element list. + if "action" in data_batch: + list_of_actions = data_batch["action"] + for action in list_of_actions: + # skip None actions + if action is None: + continue + num_action_tokens = action.shape[0] + num_tokens += num_action_tokens + + # Sound part — estimate sound tokens from audio waveform length + if self.sound_latent_fps > 0 and "sound" in data_batch: + sound_data = data_batch["sound"] + if isinstance(sound_data, list) and len(sound_data) > 0: + first_sound = sound_data[0] + # Unwrap nested list if needed + if isinstance(first_sound, list): + first_sound = first_sound[0] + if first_sound is not None and isinstance(first_sound, torch.Tensor): + num_audio_samples = first_sound.shape[-1] + audio_duration = num_audio_samples / self.audio_sample_rate + num_sound_tokens = int(audio_duration * self.sound_latent_fps) + num_tokens += num_sound_tokens + + return num_tokens + + # Keys whose value per sample is a list of tensors to be flattened into one list in the batch + _FLATTEN_LIST_KEYS = {"image_size"} + + def _update_output_batch(self, output_batch: dict, output: dict): + for key, value in output.items(): + if key in _BATCH_TIMING_KEYS: + if key not in output_batch: + output_batch[key] = value + elif key in self._FLATTEN_LIST_KEYS and isinstance(value, list): + if key not in output_batch: + output_batch[key] = value + else: + output_batch[key].extend(value) + elif key not in output_batch: + output_batch[key] = [value] + else: + output_batch[key].append(value) + + def __len__(self) -> int: + return self.data_len + + # Keys where each sample may hold multiple tensors (e.g. multiple video + # clips in a packed sequence). Kept as single-element lists per sample + # via v[i:i+1] so that _update_output_batch yields list[list[Tensor]]. + _MULTI_ITEM_KEYS = {"text_token_ids", "images", "video", "action", "sound"} + + def _get_next_sample(self, index_id: int) -> dict: + """Pop the next single-sample dict from the buffer for the given dataloader. + + If the buffer is empty, fetches the next collated batch from the inner + dataloader and splits it into individual samples. + + Splitting rules: + - Multi-item list values (keys in ``_MULTI_ITEM_KEYS``): sliced + via ``v[i:i+1]`` to yield a single-element list ``[tensor]``. + A packed sequence can contain multiple items per key. + - Per-sequence metadata list values (all other list keys, e.g. + ``sequence_plan``, ``domain_id``): direct-indexed via ``v[i]`` + to yield the bare element. + - Tensor values ``(B, ...)``: sliced to ``(1, ...)`` via + ``v[i : i + 1]`` to preserve the batch dimension. + + After ``_update_output_batch`` accumulates samples, the packed output + batch has the following shapes: + - Multi-item keys (``text_token_ids``, ``video``, ``images``, + ``action``): ``list[list[Tensor]]`` — each inner list has one + element from one sub-sample. + - Per-sequence metadata keys (``sequence_plan``, ``domain_id``, + ``dataset_name``, etc.): ``list[element]`` — flat list. + - Tensor-origin keys: ``list[Tensor(1, ...)]``. + + Args: + index_id: Index of the dataloader to fetch from. + + Returns: + A single-sample dictionary. + """ + buffer = self.buffers[index_id] + if not buffer: + try: + batch = next(self.dataloaders[index_id]) + except StopIteration: + raise + + is_image_batch = "images" in batch + input_images_or_videos = batch["images" if is_image_batch else "video"] + batch_size = len(input_images_or_videos) + + for i in range(batch_size): + sample = {} + for k, v in batch.items(): + if k in _BATCH_TIMING_KEYS: + sample[k] = v + elif isinstance(v, list) and k in self._MULTI_ITEM_KEYS: + # For multi-item keys (images, video, etc.), the collated + # value is a list with one element per sample. If the element + # is itself a list (e.g. image editing: [src, tgt]), use v[i] + # directly to avoid wrapping it in a redundant single-element + # list. Otherwise keep the v[i:i+1] slice so that + # _update_output_batch produces list[list[Tensor]]. + elem = v[i] + if isinstance(elem, list): + sample[k] = elem + else: + sample[k] = v[i : i + 1] + elif isinstance(v, list): + sample[k] = v[i] + else: + sample[k] = v[i : i + 1] + buffer.append(sample) + + return buffer.popleft() + + def set_start_iteration(self, iteration: int): + self.global_id = iteration + + def __iter__(self): + raise NotImplementedError("__iter__ function is not implemented yet") + + +class IterativeJointDataLoader(JointDataLoader): + r""" + An iterative joint dataloader that supports loading multiple modalities. + + The behavior depends on the ``seed`` parameter: + + - **seed is not None** (Default): + The modality is randomly selected at each iteration based on the probability distribution + derived from the ratios. The random state is seeded with ``seed + global_id``, ensuring + that all ranks select the same modality at the same iteration (assuming synchronized global_id). + This prevents load imbalance due to mixed resolutions across ranks. + + - **seed is None**: + The modality selection follows a deterministic round-robin pattern based on the ratios. + For example, with 2 modalities (image and video) and ratio 2:1: + - Iterations 0, 1: all ranks process images + - Iteration 2: all ranks process videos + - ... and so on. + This also ensures all ranks process the same modality at the same iteration. + """ + + def __init__( + self, + dataloaders: Dict[str, Dict[str, Union[torch.utils.data.DataLoader, webdataset.WebLoader, int]]], + tokenizer_spatial_compression_factor: int, + tokenizer_temporal_compression_factor: int, + patch_spatial: int, + max_sequence_length: int | None = None, + max_samples_per_batch: int | None = None, + sound_latent_fps: float = 0, + audio_sample_rate: int = 48000, + seed: int | None = 42, + prewarm: bool = True, + default_lookahead_limit: int = JointDataLoader._DEFAULT_LOOKAHEAD_LIMIT, + lookahead_limits: Dict[str, int] | None = None, + ): + super().__init__( + dataloaders, + tokenizer_spatial_compression_factor, + tokenizer_temporal_compression_factor, + patch_spatial, + max_sequence_length, + max_samples_per_batch, + sound_latent_fps=sound_latent_fps, + audio_sample_rate=audio_sample_rate, + prewarm=prewarm, + default_lookahead_limit=default_lookahead_limit, + lookahead_limits=lookahead_limits, + ) + self.seed = seed + # Calculate probabilities for random sampling + total_ratio = sum(self.data_ratios) + self.data_probs = np.array([ratio / total_ratio for ratio in self.data_ratios]) + + def __iter__(self): + while True: + if self.seed is not None: + rng = np.random.RandomState(self.seed + self.global_id) + index_id = rng.choice(len(self.dataloader_list), p=self.data_probs) + else: + data_id = self.global_id % self.ratio_sum + index_id = self._get_dataloader_index(data_id) + + metrics = _PackingMetrics() + output_batch = dict() + skipped_samples = deque() + lookahead_limit = self.lookahead_limits[index_id] + lookahead_count = 0 + + while True: + # Check max samples limit first + if self.max_samples_per_batch is not None and metrics.num_samples >= self.max_samples_per_batch: + break + + # If we have started packing and tried lookahead_limit times to find a fitting sample but failed, stop. + if len(output_batch) > 0 and lookahead_count >= lookahead_limit: + break + + had_buffer = len(self.buffers[index_id]) > 0 + try: + output = self._get_next_sample(index_id) + except StopIteration: + break # No more data in this dataloader + + if had_buffer: + metrics.from_buffer += 1 + else: + metrics.from_workers += 1 + + num_tokens_in_current_sample = self._compute_num_tokens_per_sample(output) + + if ( + self.max_sequence_length is not None + and metrics.current_sequence_length + num_tokens_in_current_sample >= self.max_sequence_length + ): + if len(output_batch) == 0: + # This case happens when current_sequence_length = 0 and num_tokens_in_current_sample > self.max_sequence_length + # In this case, we should simply discard the current sample and get the next sample. + log.info( + f"Discarding oversized sample with {num_tokens_in_current_sample} tokens. Max sequence length: {self.max_sequence_length}", + rank0_only=False, + ) + metrics.dropped_count += 1 + continue + + # current_sequence_length > 0 and selected sample is too large to fit in the remaining space. + # Instead of stopping immediately (creating large padding), we buffer this large sample + # and try to find a smaller one that fits in the remaining space. + skipped_samples.append(output) + lookahead_count += 1 + continue + + metrics.current_sequence_length += num_tokens_in_current_sample + metrics.num_samples += 1 + output["dataset_name"] = self.dataset_name_list[index_id] + self._update_output_batch(output_batch, output) + + # Add back skipped samples to the buffer for the next batch. + # appendleft puts item at HEAD. So we insert S3, then S2, then S1. + for sample in reversed(skipped_samples): + self.buffers[index_id].appendleft(sample) + + if len(output_batch) == 0: + return + + metrics.attach_to(output_batch, buffer_size=len(self.buffers[index_id])) + self.global_id += 1 + yield output_batch + + def _get_dataloader_index(self, data_id): + """Maps global id to the corresponding dataloader index based on ratio.""" + for i, r in enumerate(self.data_ratios): + if data_id < r: + return i + data_id -= r + raise ValueError("Invalid data_id") + + +class RankPartitionedDataLoader: + """Assigns each rank to exactly one dataset based on ratios. + + For N GPUs with datasets having ratios r_1:r_2:...:r_k, the first + N * r_1 / sum(r) ranks are assigned dataset 1, the next N * r_2 / sum(r) + ranks are assigned dataset 2, etc. Each rank instantiates a single + PyTorch DataLoader for its assigned dataset. + + The sharding information (``shard_world_size`` and ``shard_rank``) is set + on each dataset so that it shards data only across ranks that share the + same dataset, rather than across the full world. + + Example: + With 128 GPUs and datasets ``{"video": {"dataset": ..., "ratio": 3}, + "image": {"dataset": ..., "ratio": 1}}``: + + - Ranks 0-95 -> video (shard_world_size=96, shard_rank=0..95) + - Ranks 96-127 -> image (shard_world_size=32, shard_rank=0..31) + """ + + def __init__( + self, + datasets: dict[str, dict[str, Any]], + **dataloader_kwargs: Any, + ): + """ + Args: + datasets: Mapping of dataset name to config dict with keys: + + - ``"dataset"`` (required): a lazy config or dataset instance. + - ``"ratio"`` (required): positive int weight. + - ``"dataloader_kwargs"`` (optional): dict of keyword arguments + that override the top-level ``**dataloader_kwargs`` for this + dataset only (e.g. different ``num_workers`` or ``batch_size``). + + **dataloader_kwargs: Default kwargs forwarded to + ``torch.utils.data.DataLoader``. ``collate_fn`` defaults to + ``custom_collate_fn`` if not given. + """ + world_size = torch.distributed.get_world_size() + rank = torch.distributed.get_rank() + log.info(f"RankPartitionedDataLoader: world_size: {world_size} and rank: {rank}", rank0_only=False) + + _VALID_KEYS = {"dataset", "ratio", "dataloader_kwargs"} + names: list[str] = [] + dataset_configs: list[Any] = [] + ratios: list[int] = [] + per_dataset_kwargs: list[dict[str, Any]] = [] + for name, cfg in datasets.items(): + extra = set(cfg.keys()) - _VALID_KEYS + assert not extra, f"Dataset {name!r}: unexpected keys {extra}. Allowed: {_VALID_KEYS}" + if cfg["ratio"] <= 0: + log.warning( + f"RankPartitionedDataLoader: Skipping dataset {name} with ratio {cfg['ratio']}", rank0_only=False + ) + continue + names.append(name) + dataset_configs.append(cfg["dataset"]) + ratios.append(cfg["ratio"]) + per_dataset_kwargs.append(cfg.get("dataloader_kwargs", {})) + + assert len(names) > 0, "No datasets with positive ratios provided." + assert world_size >= len(names), ( + f"world_size ({world_size}) must be >= number of datasets ({len(names)}) " + f"so each dataset gets at least one rank." + ) + + total_ratio = sum(ratios) + ideal = [r / total_ratio * world_size for r in ratios] + allocations = [max(1, int(q)) for q in ideal] + remaining = world_size - sum(allocations) + if remaining > 0: + remainders = sorted(range(len(ratios)), key=lambda i: ideal[i] - allocations[i], reverse=True) + for j in range(remaining): + allocations[remainders[j]] += 1 + elif remaining < 0: + deficit = -remaining + while deficit > 0: + best = max( + (i for i in range(len(allocations)) if allocations[i] > 1), + key=lambda i: (allocations[i] - ideal[i], allocations[i]), + ) + allocations[best] -= 1 + deficit -= 1 + + expected_ratios = [r / total_ratio for r in ratios] + actual_ratios = [a / world_size for a in allocations] + lines = [f"RankPartitionedDataLoader allocation ({world_size} GPUs):"] + start = 0 + for i, (name, alloc) in enumerate(zip(names, allocations)): + end = start + alloc - 1 + lines.append( + f" {name} (ratio {ratios[i]}): ranks {start}-{end} ({alloc} GPUs) " + f"| expected {expected_ratios[i]:.2%}, actual {actual_ratios[i]:.2%}" + ) + start += alloc + log.info("\n".join(lines), rank0_only=False) + + cumulative = 0 + my_dataset_idx = -1 + for i, alloc in enumerate(allocations): + if rank < cumulative + alloc: + my_dataset_idx = i + break + cumulative += alloc + assert my_dataset_idx >= 0 + + shard_rank = rank - cumulative + shard_world_size = allocations[my_dataset_idx] + + dataset: Any = instantiate(dataset_configs[my_dataset_idx]) + dataset.shard_world_size = shard_world_size + dataset.shard_rank = shard_rank + dataset.shard_id = my_dataset_idx + + merged_kwargs = {**dataloader_kwargs, **per_dataset_kwargs[my_dataset_idx]} + merged_kwargs.setdefault("collate_fn", custom_collate_fn) + self.dataloader = torch.utils.data.DataLoader(dataset, **merged_kwargs) + self.dataset_name = names[my_dataset_idx] + self.dataset = dataset + + def __iter__(self): + return iter(self.dataloader) + + def __len__(self) -> int: + return len(self.dataloader) + + +class PackingDataLoader(JointDataLoader): + """Packs multiple samples from a single dataloader into token-budget-constrained batches. + + Unlike the other ``JointDataLoader`` subclasses which manage multiple + dataloaders with configurable ratios, this class wraps a single dataloader + and greedily packs consecutive samples until the token budget + (``max_sequence_length``) or sample count limit (``max_samples_per_batch``) + is reached. + """ + + def __init__( + self, + dataloader: torch.utils.data.DataLoader | webdataset.WebLoader, + tokenizer_spatial_compression_factor: int, + tokenizer_temporal_compression_factor: int, + patch_spatial: int, + max_sequence_length: int | None = None, + max_samples_per_batch: int | None = None, + sound_latent_fps: float = 0, + audio_sample_rate: int = 48000, + dataset_name: str = "default", + lookahead_limit: int = JointDataLoader._DEFAULT_LOOKAHEAD_LIMIT, + ): + """ + Args: + dataloader: A single dataloader (or lazy config) to draw samples from. + tokenizer_spatial_compression_factor: Spatial compression factor of the tokenizer. + tokenizer_temporal_compression_factor: Temporal compression factor of the tokenizer. + patch_spatial: Spatial patchification factor. + max_sequence_length: Max total tokens per packed batch. Mutually exclusive with + ``max_samples_per_batch``. + max_samples_per_batch: Max number of samples per packed batch. Mutually exclusive + with ``max_sequence_length``. + sound_latent_fps: Sound tokenizer latent rate in Hz. If 0, sound tokens are not counted. + audio_sample_rate: Audio sample rate in Hz. + dataset_name: Name tag attached to every sample in the output batch. + lookahead_limit: Packing-loop look-ahead for the wrapped dataloader. + """ + wrapped = {dataset_name: {"dataloader": dataloader, "ratio": 1}} + super().__init__( + dataloaders=wrapped, + tokenizer_spatial_compression_factor=tokenizer_spatial_compression_factor, + tokenizer_temporal_compression_factor=tokenizer_temporal_compression_factor, + patch_spatial=patch_spatial, + max_sequence_length=max_sequence_length, + max_samples_per_batch=max_samples_per_batch, + sound_latent_fps=sound_latent_fps, + audio_sample_rate=audio_sample_rate, + lookahead_limits={dataset_name: int(lookahead_limit)}, + ) + + def __iter__(self): + inner = self.dataloader_list[0] + ds_name = getattr(inner, "dataset_name", self.dataset_name_list[0]) + + while True: + current_sequence_length = 0 + num_samples = 0 + output_batch: dict = {} + + skipped_samples: deque = deque() + # PackingDataLoader wraps a single dataloader, so lookahead_limits has one entry. + lookahead_limit = self.lookahead_limits[0] + lookahead_count = 0 + + while True: + if self.max_samples_per_batch is not None and num_samples >= self.max_samples_per_batch: + break + + if len(output_batch) > 0 and lookahead_count >= lookahead_limit: + break + + try: + output = self._get_next_sample(0) + except StopIteration: + break + + num_tokens_in_current_sample = self._compute_num_tokens_per_sample(output) + + if ( + self.max_sequence_length is not None + and current_sequence_length + num_tokens_in_current_sample >= self.max_sequence_length + ): + if len(output_batch) == 0: + # This case happens when current_sequence_length = 0 and num_tokens_in_current_sample > self.max_sequence_length + # In this case, we should simply discard the current sample and get the next sample. + log.error( + f"PackingDataLoader: Discarding oversized sample with {num_tokens_in_current_sample} tokens. Max sequence length: {self.max_sequence_length}", + rank0_only=False, + ) + continue + + skipped_samples.append(output) + lookahead_count += 1 + continue + + current_sequence_length += num_tokens_in_current_sample + num_samples += 1 + output["dataset_name"] = ds_name + self._update_output_batch(output_batch, output) + + for sample in reversed(skipped_samples): + self.buffers[0].appendleft(sample) + + if len(output_batch) == 0: + return + + self.global_id += 1 + yield output_batch + + +class RandomJointDataLoader(JointDataLoader): + r""" + A random joint dataloader that supports loading multiple modalities with stochastic sampling. + + In this dataloader, the modality is randomly selected at each iteration based on the + probability distribution derived from the ratios. Each rank independently samples a + modality, so different ranks may process different modalities at the same iteration. + + For example, with 2 modalities (image and video) and ratio 2:1: + - Each iteration has 66.7% probability of selecting images + - Each iteration has 33.3% probability of selecting videos + - The selection is independent across iterations and ranks + + Note: Unlike IterativeJointDataLoader, this does not guarantee synchronized modality + selection across ranks. + """ + + def __init__( + self, + dataloaders: Dict[str, Dict[str, Union[torch.utils.data.DataLoader, webdataset.WebLoader, int]]], + tokenizer_spatial_compression_factor: int, + tokenizer_temporal_compression_factor: int, + patch_spatial: int, + max_sequence_length: int | None = None, + max_samples_per_batch: int | None = None, + sound_latent_fps: float = 0, + audio_sample_rate: int = 48000, + default_lookahead_limit: int = JointDataLoader._DEFAULT_LOOKAHEAD_LIMIT, + lookahead_limits: Dict[str, int] | None = None, + ): + super().__init__( + dataloaders, + tokenizer_spatial_compression_factor, + tokenizer_temporal_compression_factor, + patch_spatial, + max_sequence_length, + max_samples_per_batch, + sound_latent_fps=sound_latent_fps, + audio_sample_rate=audio_sample_rate, + default_lookahead_limit=default_lookahead_limit, + lookahead_limits=lookahead_limits, + ) + + # Convert data ratios to probabilities + self.data_ratios = np.array([ratio / sum(self.data_ratios) for ratio in self.data_ratios]) + + def __iter__(self): + while True: + index_id = np.random.choice(len(self.dataloader_list), p=self.data_ratios) + + metrics = _PackingMetrics() + output_batch = dict() + skipped_samples = deque() + lookahead_limit = self.lookahead_limits[index_id] + lookahead_count = 0 + + while True: + # Check max samples limit first + if self.max_samples_per_batch is not None and metrics.num_samples >= self.max_samples_per_batch: + break + + # If we have started packing and tried lookahead_limit times to find a fitting sample but failed, stop. + if len(output_batch) > 0 and lookahead_count >= lookahead_limit: + break + + had_buffer = len(self.buffers[index_id]) > 0 + try: + output = self._get_next_sample(index_id) + except StopIteration: + break # No more data in this dataloader + + if had_buffer: + metrics.from_buffer += 1 + else: + metrics.from_workers += 1 + + num_tokens_in_current_sample = self._compute_num_tokens_per_sample(output) + + if ( + self.max_sequence_length is not None + and metrics.current_sequence_length + num_tokens_in_current_sample >= self.max_sequence_length + ): + if len(output_batch) == 0: + # This case happens when current_sequence_length = 0 and num_tokens_in_current_sample > self.max_sequence_length + # In this case, we should simply discard the current sample and get the next sample. + log.info( + f"Discarding oversized sample with {num_tokens_in_current_sample} tokens. Max sequence length: {self.max_sequence_length}", + rank0_only=False, + ) + metrics.dropped_count += 1 + continue + + # current_sequence_length > 0 and selected sample is too large to fit in the remaining space. + # Instead of stopping immediately (creating large padding), we buffer this large sample + # and try to find a smaller one that fits in the remaining space. + skipped_samples.append(output) + lookahead_count += 1 + continue + + metrics.current_sequence_length += num_tokens_in_current_sample + metrics.num_samples += 1 + output["dataset_name"] = self.dataset_name_list[index_id] + self._update_output_batch(output_batch, output) + + # Add back skipped samples to the buffer for the next batch. + # appendleft puts item at HEAD. So we insert S3, then S2, then S1. + for sample in reversed(skipped_samples): + self.buffers[index_id].appendleft(sample) + + if len(output_batch) == 0: + return + + metrics.attach_to(output_batch, buffer_size=len(self.buffers[index_id])) + yield output_batch diff --git a/cosmos_framework/data/vfm/local_datasets/__init__.py b/cosmos_framework/data/vfm/local_datasets/__init__.py new file mode 100644 index 0000000..503ec1b --- /dev/null +++ b/cosmos_framework/data/vfm/local_datasets/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + diff --git a/cosmos_framework/data/vfm/local_datasets/helper.py b/cosmos_framework/data/vfm/local_datasets/helper.py new file mode 100644 index 0000000..22fa25b --- /dev/null +++ b/cosmos_framework/data/vfm/local_datasets/helper.py @@ -0,0 +1,253 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Shared helpers for local datasets (S3, video decoding, aspect ratio).""" + +import io +import json +import subprocess +import time +from collections.abc import Generator +from pathlib import Path +from typing import Any + +import numpy as np +from boto3.s3.transfer import TransferConfig +from botocore.config import Config + +from cosmos_framework.utils import log + +client_config = Config( + response_checksum_validation="when_required", + request_checksum_calculation="when_required", + connect_timeout=10, + read_timeout=5, +) +transfer_config = TransferConfig(use_threads=True, max_concurrency=8, multipart_chunksize=8 * 1024 * 1024) + + +def parse_s3_url(s3_url: str) -> tuple[str, str]: + s3_url = s3_url.removeprefix("s3://") + bucket, key = s3_url.split("/", 1) + return bucket, key + + +def download_from_s3(s3_client: Any, s3_url: str, max_tries: int = 20) -> bytes | None: + """Download a file from S3.""" + if not s3_url.startswith("s3://"): + return Path(s3_url).read_bytes() + tries = 0 + while True: + tries += 1 + try: + bucket, key = parse_s3_url(s3_url) + buffer = io.BytesIO() + s3_client.download_fileobj(Bucket=bucket, Key=key, Fileobj=buffer, Config=transfer_config) + data = buffer.getvalue() + return data + except Exception as e: + log.error(f"Error downloading from S3 (try {tries}): {e}\n{s3_url}") + if tries >= max_tries: + return None + time.sleep(1) + + +def get_video_metadata(video_path: str) -> dict: + """ + Get video metadata using ffprobe. + + Args: + video_path: Path to the video file + + Returns: + Dictionary containing width, height, fps, and total_frames + """ + cmd = [ + "ffprobe", + "-v", + "quiet", + "-print_format", + "json", + "-show_streams", + "-select_streams", + "v:0", + video_path, + ] + result = subprocess.run(cmd, stdin=subprocess.DEVNULL, capture_output=True, check=True, text=True) + probe_data = json.loads(result.stdout) + + # Decode output + stream = probe_data["streams"][0] + width = int(stream["width"]) + height = int(stream["height"]) + fps_parts = stream["r_frame_rate"].split("/") + video_fps = float(fps_parts[0]) / float(fps_parts[1]) + if "nb_frames" in stream: + total_frames = int(stream["nb_frames"]) + else: + duration = float(stream.get("duration") or 0) + total_frames = int(duration * video_fps) + + return dict(width=width, height=height, fps=video_fps, total_frames=total_frames) + + +def ffmpeg_decode_video( + video_path: str, scale_hw: tuple[int, int] | None = None, num_threads: int = 1 +) -> Generator[np.ndarray, None, None]: + """ + Decode video frames using ffmpeg and yield HWC uint8 RGB frames. + + Args: + video_path: Path to the video file + scale_hw: Tuple of width and height to scale the video to (default: None) + + Yields: + np.ndarray: HWC uint8 RGB frames + """ + if scale_hw is None: + metadata = get_video_metadata(video_path) + out_width = metadata["width"] + out_height = metadata["height"] + else: + out_height, out_width = scale_hw + + # Calculate frame size in bytes + frame_size = out_width * out_height * 3 # 3 channels (RGB) + + # Build ffmpeg command to decode and output raw RGB frames + ffmpeg_cmd = [ + "ffmpeg", + "-loglevel", + "quiet", + "-threads", + str(num_threads), + "-filter_threads", + str(num_threads), + "-filter_complex_threads", + str(num_threads), + "-i", + video_path, + "-threads", + str(num_threads), + "-filter_threads", + str(num_threads), + "-filter_complex_threads", + str(num_threads), + "-pix_fmt", + "rgb24", + "-sws_flags", + "bicubic+accurate_rnd", # lanczos too much ringing on graphics + *(["-vf", f"scale={scale_hw[1]}:{scale_hw[0]}"] if scale_hw else []), # WH + "-f", + "rawvideo", + "-vsync", + "0", + "-", + ] + + process = subprocess.Popen( + ffmpeg_cmd, + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, # Set to None to print errors + bufsize=-1, + ) + + try: + while True: + raw_frame = process.stdout.read(frame_size) + + if len(raw_frame) != frame_size: + assert len(raw_frame) == 0, f"Incomplete frame: {len(raw_frame)} bytes" + break + + frame = np.frombuffer(raw_frame, dtype=np.uint8) + frame = frame.reshape((out_height, out_width, 3)) + + yield frame + finally: + process.stdout.close() + process.wait() + + +def get_aspect_ratio(width: int, height: int) -> str: + """Compute aspect ratio bucket from width and height.""" + ratio = width / height + + if ratio < 0.65: + return "9,16" # 0.5625 + elif ratio < 0.88: + return "3,4" # 0.75 + elif ratio < 1.16: + return "1,1" # 1.0 + elif ratio < 1.55: + return "4,3" # 1.3333 + else: + return "16,9" # 1.7778 + + +def save_video_frames_to_mp4( + frames: np.ndarray | Any, + output_path: str, + fps: float = 24.0, + overlay_frame_id: bool = False, + fps_to_show: float | None = None, +) -> None: + """Encode video frames to MP4 using FFmpeg. + + Args: + frames: Video frames as numpy (T, H, W, 3) or torch tensor (C, T, H, W), uint8. + output_path: Path for the output .mp4 file. + fps: Output video frame rate. + overlay_frame_id: If True, draw frame index (0, 1, ...) on each frame via FFmpeg drawtext. + fps_to_show: If provided, draw the FPS value on the video instead of the actual FPS. + """ + cpu_fn = getattr(frames, "cpu", None) + if callable(cpu_fn): + frames = cpu_fn().numpy() # type: ignore[union-attr] + frames = np.asarray(frames, dtype=np.uint8) + if frames.ndim == 4 and frames.shape[0] == 3: + # CTHW -> THWC + frames = np.transpose(frames, (1, 2, 3, 0)) + if frames.ndim != 4 or frames.shape[-1] != 3: + raise ValueError("frames must be (T, H, W, 3) or (C, T, H, W) uint8") + t, h, w, _ = frames.shape + cmd = [ + "ffmpeg", + "-y", + "-f", + "rawvideo", + "-pix_fmt", + "rgb24", + "-s", + f"{w}x{h}", + "-r", + str(fps), + "-i", + "pipe:0", + ] + if overlay_frame_id: + # %{n} = frame index (0-based); add fps and resolution as literal text + drawtext_frame = "drawtext=text='%{n}':x=10:y=10:fontsize=24:fontcolor=white:box=1:boxcolor=black@0.6" + drawtext_fps = ( + f"drawtext=text='fps: {fps_to_show or fps}':x=10:y=40:fontsize=24:fontcolor=white:box=1:boxcolor=black@0.6" + ) + drawtext_res = f"drawtext=text='{w}x{h}':x=10:y=70:fontsize=24:fontcolor=white:box=1:boxcolor=black@0.6" + cmd += ["-vf", ",".join([drawtext_frame, drawtext_fps, drawtext_res])] + cmd += [ + "-c:v", + "libx264", + "-pix_fmt", + "yuv420p", + output_path, + ] + process = subprocess.Popen( + cmd, + stdin=subprocess.PIPE, + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + ) + _, stderr = process.communicate(input=frames.tobytes()) + if process.returncode != 0: + log.error(f"FFmpeg failed: {stderr.decode()}") + raise RuntimeError(f"FFmpeg exited with {process.returncode}") diff --git a/cosmos_framework/data/vfm/local_datasets/sft_dataset.py b/cosmos_framework/data/vfm/local_datasets/sft_dataset.py new file mode 100644 index 0000000..c06e511 --- /dev/null +++ b/cosmos_framework/data/vfm/local_datasets/sft_dataset.py @@ -0,0 +1,680 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +# SFT dataset loader — reads video metadata + captions from a JSONL file on S3. +import gzip +import hashlib +import io +import json +import os +import random +import tempfile +from pathlib import Path +from typing import Any, Optional + +import boto3 +import numpy as np +import torch + +from cosmos_framework.utils.flags import INTERNAL +from cosmos_framework.utils.lazy_config import instantiate as lazy_instantiate +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.local_datasets.helper import ( + client_config, + download_from_s3, + ffmpeg_decode_video, + get_aspect_ratio, + get_video_metadata, + parse_s3_url, +) +from cosmos_framework.data.vfm.sequence_packing import SequencePlan, add_special_tokens +from cosmos_framework.data.vfm.utils import VIDEO_RES_SIZE_INFO +from cosmos_framework.model.vfm.vlm.qwen3_vl.utils import tokenize_caption + +_MAX_NUM_TOKENS = 1024 +_DURATION_TEMPLATE = "The video is {duration:.1f} seconds long and is of {fps:.0f} FPS." +_RESOLUTION_TEMPLATE = "This video is of {height}x{width} resolution." + +# Caption types available in the SFT JSONL. +# Format: {model}_{style} +# model: qwen3_235b | qwen3_32b | qwen3p5_397b +# style: short | temporal | descriptive | dense +CAPTION_TYPES_AND_WEIGHTS: dict[str, float] = { + # short: 10% total + "qwen3_235b_short": 0.1, + "qwen3_32b_short": 0.1, + "qwen3p5_397b_short": 0.1, + # descriptive: 20% total + "qwen3_235b_descriptive": 0.2, + "qwen3_32b_descriptive": 0.2, + "qwen3p5_397b_descriptive": 0.2, + # dense: 70% total + "qwen3_235b_dense": 0.7, + "qwen3_32b_dense": 0.7, + "qwen3p5_397b_dense": 0.7, + # temporal: 0% total + "qwen3_235b_temporal": 0.0, + "qwen3_32b_temporal": 0.0, + "qwen3p5_397b_temporal": 0.0, +} +CAPTION_TYPES = list(CAPTION_TYPES_AND_WEIGHTS.keys()) +CAPTION_WEIGHTS = list(CAPTION_TYPES_AND_WEIGHTS.values()) + + +class SFTDataset(torch.utils.data.IterableDataset): + """Dataset for loading SFT video clips with captions from JSONL metadata on S3.""" + + def __init__( + self, + metadata: list[dict], + num_video_frames: int, + resolution: str, + s3_credentials: dict, + temporal_interval_mode: str = "entire_chunk", + frame_selection_mode: str = "center", + tokenizer_config: Optional[Any] = None, + cfg_dropout_rate: float = 0.0, + use_system_prompt: bool = False, + append_duration_fps_timestamps: bool = True, + append_resolution_info: bool = True, + cfg_dropout_keep_metadata: bool = False, + caption_suffix: str = "", + conditioning_fps: float = 24, + conditioning_fps_noise_std: float = 0.0, + conditioning_config: dict[int, float] | None = None, + temporal_compression_factor: int = 4, + ): + assert temporal_interval_mode in ("force_one", "max_30fps", "entire_chunk"), ( + f"Unknown temporal_interval_mode={temporal_interval_mode!r}" + ) + assert frame_selection_mode in ("center", "first", "random"), ( + f"Unknown frame_selection_mode={frame_selection_mode!r}" + ) + assert temporal_compression_factor >= 1, "temporal_compression_factor must be >= 1" + self.metadata = metadata + self.num_video_frames = num_video_frames + self.resolution = resolution + self.s3_credentials = s3_credentials + self.temporal_interval_mode = temporal_interval_mode + self.frame_selection_mode = frame_selection_mode + self.tokenizer_config = tokenizer_config + self.cfg_dropout_rate = cfg_dropout_rate + self.use_system_prompt = use_system_prompt + self.append_duration_fps_timestamps = append_duration_fps_timestamps + self.append_resolution_info = append_resolution_info + self.cfg_dropout_keep_metadata = cfg_dropout_keep_metadata + self.caption_suffix = caption_suffix.strip() + self.conditioning_fps = conditioning_fps + self.conditioning_fps_noise_std = conditioning_fps_noise_std + + self.temporal_compression_factor = temporal_compression_factor + self.conditioning_config: dict[int, float] | None = None + if conditioning_config is not None: + total_prob = sum(conditioning_config.values()) + assert total_prob > 0, "conditioning_config probabilities must sum to a positive number" + self.conditioning_config = {k: v / total_prob for k, v in conditioning_config.items()} + log.info(f"Conditioning config: {self.conditioning_config}") + # They will be set by the RankPartitionedDataLoader + self.shard_world_size = None + self.shard_rank = None + self.shard_id = 0 + self.is_initialized = False + self.output_sizes = VIDEO_RES_SIZE_INFO[resolution] + + _vlm_proc = lazy_instantiate(self.tokenizer_config) + self.vlm_tokenizer = _vlm_proc.tokenizer + self.vlm_tokenizer, _ = add_special_tokens(self.vlm_tokenizer) + + def __len__(self): + return len(self.metadata) + + def _tokenize_caption(self, caption: str) -> tuple[list[int], str]: + text_ids = tokenize_caption( + caption, + self.vlm_tokenizer, + is_video=True, + use_system_prompt=self.use_system_prompt, + ) + if len(text_ids) > _MAX_NUM_TOKENS: + log.warning(f"Text ids are too long, truncating: {len(text_ids)} > {_MAX_NUM_TOKENS}") + text_ids = text_ids[:_MAX_NUM_TOKENS] + return text_ids, caption + + def process_one_sample(self, metadata: dict) -> dict | None: + """Process a single SFT sample: download, decode, and prepare for training. + + A random t2w_window is picked from the video's list of windows each time. + """ + windows = metadata["t2w_windows"] + win_idx = random.randrange(len(windows)) + t2w_window = windows[win_idx] + window_start = t2w_window["start_frame"] + window_end = t2w_window["end_frame"] + + # Compute output resolution + input_w, input_h = metadata["width"], metadata["height"] + target_w, target_h = self.output_sizes[metadata["aspect_ratio"]] + resize_ratio = max(target_w / input_w, target_h / input_h) + resize_h, resize_w = (round(input_h * resize_ratio), round(input_w * resize_ratio)) + crop_y, crop_x = (round((resize_h - target_h) / 2), round((resize_w - target_w) / 2)) + + video_bytes = download_from_s3(self.s3_client, metadata["vision_path"]) + if video_bytes is None: + log.warning(f"Failed to download video from S3: {metadata['vision_path']}") + return None + + # Decode all frames to (T, H, W, 3) + with tempfile.NamedTemporaryFile(suffix=".mp4", delete=True) as tmp_input: + tmp_input.write(video_bytes) + tmp_input.flush() + input_video_path = tmp_input.name + video_info = get_video_metadata(input_video_path) + original_fps = video_info["fps"] + total_frames = video_info["total_frames"] + + # Constrain to the t2w window + actual_end = min(window_end, total_frames - 1) + frames_in_window = actual_end - window_start + 1 + + if self.num_video_frames == -1: + # Native chunk mode: use start/end/interval directly from the window + temporal_interval = t2w_window["temporal_interval"] + start_frame = window_start + end_frame = actual_end + else: + if frames_in_window < self.num_video_frames: + log.warning( + f"Not enough frames in window: {metadata['uuid']}, " + f"frames_in_window: {frames_in_window}, required: {self.num_video_frames}" + ) + return None + + # Compute temporal interval + if self.temporal_interval_mode == "force_one": + temporal_interval = 1 + elif self.temporal_interval_mode == "max_30fps": + temporal_interval = max(1, int(original_fps / 30.0)) + elif self.temporal_interval_mode == "entire_chunk": + temporal_interval = frames_in_window // self.num_video_frames + temporal_interval = max(1, temporal_interval) + else: + raise ValueError(f"Unknown temporal_interval_mode: {self.temporal_interval_mode}") + + num_frames_before_downsample = (self.num_video_frames - 1) * temporal_interval + 1 + if self.frame_selection_mode == "first": + start_frame = window_start + elif self.frame_selection_mode == "center": + start_frame = window_start + (frames_in_window - num_frames_before_downsample) // 2 + elif self.frame_selection_mode == "random": + max_offset = frames_in_window - num_frames_before_downsample + start_frame = window_start + random.randint(0, max(0, max_offset)) + else: + raise ValueError(f"Unknown frame_selection_mode: {self.frame_selection_mode}") + end_frame = start_frame + num_frames_before_downsample - 1 + + fps = original_fps / temporal_interval + + video_chunk = [] + for idx, frame in enumerate( + ffmpeg_decode_video(input_video_path, scale_hw=(resize_h, resize_w), num_threads=2) + ): + if idx < start_frame: + continue + elif idx <= end_frame: + if (idx - start_frame) % temporal_interval == 0: + video_chunk.append(frame) + else: + break + + if not video_chunk: + log.warning( + f"No frames decoded for sample: {metadata['uuid']} " + f"(start={start_frame}, end={end_frame}, path={metadata['vision_path']})" + ) + return None + + video_chunk = np.stack(video_chunk, axis=0) # [T,H,W,3] + + # Truncate temporally to temporal_compression_factor * N + 1 + target_t = (video_chunk.shape[0] - 1) // self.temporal_compression_factor * self.temporal_compression_factor + 1 + + # Apply spatial center crop and temporal truncation + video_chunk = video_chunk[:target_t, crop_y : crop_y + target_h, crop_x : crop_x + target_w] # [T,H,W,3] + + # THWC -> CTHW + video_chunk = np.transpose(video_chunk, (3, 0, 1, 2)) # [3,T,H,W] + video = torch.from_numpy(np.ascontiguousarray(video_chunk)).to(torch.uint8) # [3,T,H,W] + padding_mask = torch.zeros((1, target_h, target_w), dtype=torch.float32) + # image_size: [target_h, target_w, orig_h, orig_w] in pixel space, for the model to crop the video + image_size = torch.tensor([target_h, target_w, target_h, target_w], dtype=torch.float32) + + available_types = [ct for ct in CAPTION_TYPES if ct in t2w_window] + if "qwen3_32b_rewrite-dense" in t2w_window: + caption_key = "qwen3_32b_rewrite-dense" + elif "caption" in t2w_window: + caption_key = "caption" + elif available_types: + available_weights = [CAPTION_TYPES_AND_WEIGHTS[ct] for ct in available_types] + caption_key = random.choices(available_types, weights=available_weights, k=1)[0] + else: + log.warning( + f"No known caption key found in t2w_window for sample {metadata['uuid']}. " + f"Keys: {list(t2w_window)}. Skipping sample." + ) + return None + caption = t2w_window[caption_key] + caption = caption.strip().rstrip(".") + "." + + num_decoded_frames = video.shape[1] + cond_fps = fps if self.conditioning_fps < 0 else self.conditioning_fps + if self.conditioning_fps_noise_std > 0: + noise_factor = np.exp(np.random.randn() * self.conditioning_fps_noise_std) + cond_fps = cond_fps * noise_factor + + if self.caption_suffix: + caption = (caption + " " + self.caption_suffix).strip() + + # CFG dropout: when cfg_dropout_keep_metadata is True, dropout fires + # before appending resolution/duration/FPS so that metadata text is + # preserved even under unconditional guidance. + if self.cfg_dropout_keep_metadata and self.cfg_dropout_rate > 0: + if random.random() < self.cfg_dropout_rate: + caption = "" + + if self.append_duration_fps_timestamps: + duration = num_decoded_frames / cond_fps + suffix = _DURATION_TEMPLATE.format(duration=duration, fps=cond_fps) + caption = caption + " " + suffix + if self.append_resolution_info: + suffix = _RESOLUTION_TEMPLATE.format(height=target_h, width=target_w) + caption = caption + " " + suffix + caption = caption.strip() + + if not self.cfg_dropout_keep_metadata and self.cfg_dropout_rate > 0: + if random.random() < self.cfg_dropout_rate: + caption = "" + text_ids, caption = self._tokenize_caption(caption) + + ret = dict( + __key__=f"{metadata['uuid']}_w{win_idx}", + __url__=metadata["vision_path"], + fps=original_fps, + n_orig_video_frames=total_frames, + chunk_index=win_idx, + frame_start=start_frame, + frame_end=end_frame, + num_frames=video.shape[1], + video=video, + num_multiplier=temporal_interval, + conditioning_fps=cond_fps, + padding_mask=padding_mask, + image_size=image_size, + ai_caption=caption, + sampled_caption_style=caption_key, + text_token_ids=torch.tensor(text_ids), + ) + + if self.conditioning_config is not None: + num_frames_pixel = video.shape[1] + t_latent = 1 + (num_frames_pixel - 1) // self.temporal_compression_factor + frames_options = list(self.conditioning_config.keys()) + weights = list(self.conditioning_config.values()) + num_cond = random.choices(frames_options, weights=weights, k=1)[0] + num_cond = min(num_cond, t_latent - 1) + ret["sequence_plan"] = SequencePlan( + has_text=True, + has_vision=True, + condition_frame_indexes_vision=list(range(num_cond)), + ) + + return ret + + def __iter__(self): + assert not self.is_initialized, "Dataset can only be initialized once." + assert len(self.metadata) > 0, "Did not find any data." + + self.s3_client = boto3.client( + "s3", + **self.s3_credentials, + config=client_config, + ) + # Ranks of the same pp/tp/cp group will have the same dp rank and thus share the same group id. + # zhao: Cosmos3 does not support TP/SP/CP + if self.shard_world_size is not None: + train_world_size = self.shard_world_size + train_rank = self.shard_rank + log.info(f"Using shard_world_size: {train_world_size} and shard_rank: {train_rank}", rank0_only=False) + else: + train_world_size = torch.distributed.get_world_size() + train_rank = torch.distributed.get_rank() + train_dp_rank = train_rank + train_num_dp_groups = train_world_size + train_dp_group_size = 1 + + # Get data worker rank. Each trainer have multiple dataloaders + worker_info = torch.utils.data.get_worker_info() + if worker_info is not None: + worker_rank = worker_info.id + total_data_ranks = worker_info.num_workers * train_num_dp_groups + data_rank = worker_rank + train_dp_rank * worker_info.num_workers + seed = worker_info.seed + else: + log.warning("No data worker info found. Using default worker rank and number of workers.", rank0_only=False) + total_data_ranks = train_num_dp_groups + data_rank = train_dp_rank + seed = 42 + + log.info( + f"train_world_size: {train_world_size}; " + f"train_rank: {train_rank}; " + f"train_dp_rank: {train_dp_rank}; " + f"train_num_dp_groups: {train_num_dp_groups}; " + f"train_dp_group_size: {train_dp_group_size}; " + f"worker_info: {worker_info}; " + f"total_data_ranks: {total_data_ranks}; " + f"data_rank: {data_rank}; " + f"seed: {seed}" + f"shard_id: {self.shard_id}; " + f"shard_world_size: {self.shard_world_size}; " + f"shard_rank: {self.shard_rank}", + rank0_only=False, + ) + + # Make sure len(self.metadata) is divisible by self.num_groups + multiplier = max(1, total_data_ranks * 50 // len(self.metadata)) + log.info(f"Dataset multiplier: {multiplier}", rank0_only=False) + self.metadata = self.metadata * multiplier # reduce bias caused by sharding + num_pad = total_data_ranks - len(self.metadata) % total_data_ranks + self.metadata = self.metadata + self.metadata[:num_pad] + # Deterministic shuffle based on the sha256 hash of uuid + # Note that the repeated samples are grouped together. + # Split list to keep only the data for this rank + if True: # This gives more diversity + random.Random(self.shard_id).shuffle(self.metadata) + log.info(f"Shuffled metadata for shard {self.shard_id}", rank0_only=False) + self.metadata = self.metadata[data_rank::total_data_ranks] + else: + # Keep the repeated samples together to aid cache hits. + self.metadata.sort(key=lambda x: hashlib.sha256(x["vision_path"].encode("utf-8")).hexdigest()) + # Equally chunk the list (guaranteed to be divisible by total_data_ranks) + chunk_size = len(self.metadata) // total_data_ranks + start = data_rank * chunk_size + end = (data_rank + 1) * chunk_size + log.info( + f"DRank {data_rank} has got a chunk {start}-{end} from {len(self.metadata)} data.", rank0_only=False + ) + self.metadata = self.metadata[start:end] + num_unique_vision_paths = len(set(metadata["vision_path"] for metadata in self.metadata)) + log.info( + f"DRank {data_rank} has {len(self.metadata)} data with {num_unique_vision_paths} unique vision_paths.", + rank0_only=False, + ) + + self.is_initialized = True + + # Make sure the data within a DRank is identical + rng = random.Random(data_rank + self.shard_id * 12345) + while True: + rng.shuffle(self.metadata) + for metadata in self.metadata: + sample = self.process_one_sample(metadata) + if sample is None: + log.warning(f"Failed to process sample {metadata['uuid']}, skipping...") + continue + yield sample + + +def _flatten_metadata_by_window(metadata_list: list[dict]) -> list[dict]: + """Expand metadata so each entry maps to exactly one t2w_window. + + Each output dict is a shallow copy of the original whose ``t2w_windows`` + list contains a single window. The ``uuid`` is suffixed with ``_w{idx}`` + so every entry has a unique identifier. + """ + flat: list[dict] = [] + for entry in metadata_list: + for win_idx, window in enumerate(entry["t2w_windows"]): + flat.append( + { + **entry, + "uuid": f"{entry['uuid']}_w{win_idx}", + "t2w_windows": [window], + } + ) + return flat + + +def _load_sft_metadata_from_s3( + s3_client, + jsonl_url: str, + min_frames: int, + uuid_prefix: str = "", + min_short_edge: int = 0, +) -> list[dict]: + """Load SFT metadata from a single JSONL file on S3. + + Returns one entry per video. Each entry keeps only the windows whose frame + span is at least *min_frames*; videos with no qualifying windows are dropped. + + Args: + s3_client: Boto3 S3 client + jsonl_url: S3 URL to the JSONL metadata file + min_frames: Minimum number of frames required per window + uuid_prefix: Prefix prepended to each uuid for disambiguation when + multiple JSONL files are loaded + min_short_edge: Drop videos whose shortest spatial edge (min of width, + height) is below this value. 0 disables the filter. + """ + log.info(f"Downloading SFT metadata from {jsonl_url}", rank0_only=False) + metadata_list: list[dict] = [] + num_raw_records = 0 + num_raw_windows = 0 + num_filtered_duration = 0 + num_filtered_windows = 0 + num_filtered_short_edge = 0 + + with io.BytesIO() as buffer: + if jsonl_url.startswith("s3://"): + bucket, key = parse_s3_url(jsonl_url) + s3_client.download_fileobj(Bucket=bucket, Key=key, Fileobj=buffer) + else: + path = Path(jsonl_url).absolute() + jsonl_url = str(path) + buffer.write(path.read_bytes()) + buffer.seek(0) + log.info("Finished downloading. Decoding...", rank0_only=False) + + line_iter = gzip.open(buffer, "rb") if jsonl_url.endswith(".gz") else buffer + for line in line_iter: + num_raw_records += 1 + record = json.loads(line.decode("utf-8")) + uuid = f"{uuid_prefix}{record['uuid']}" if uuid_prefix else record["uuid"] + if record["duration"] > 61.0: + print(f"Skipping video with too long duration: {uuid}, {record['duration']} > 61.0") + num_filtered_duration += 1 + continue + if min_short_edge > 0 and min(record["width"], record["height"]) < min_short_edge: + num_filtered_short_edge += 1 + continue + + windows = record.get("t2w_windows") + if not windows: + continue + + kept_windows = [] + for window in windows: + num_raw_windows += 1 + frames_in_window = window["end_frame"] - window["start_frame"] + 1 + if frames_in_window < min_frames: + num_filtered_windows += 1 + else: + kept_windows.append(window) + + if not kept_windows: + continue + + vision_path = record["vision_path"] + if "://" not in vision_path and not vision_path.startswith("/"): + # Relative path to the JSONL file + vision_path = f"{os.path.dirname(jsonl_url)}/{vision_path}" + + aspect_ratio = get_aspect_ratio(record["width"], record["height"]) + metadata_list.append( + { + "uuid": uuid, + "vision_path": vision_path, + "width": record["width"], + "height": record["height"], + "nb_frames": record.get("nb_frames"), + "framerate": record.get("framerate"), + "aspect_ratio": aspect_ratio, + "t2w_windows": kept_windows, + } + ) + + log.info( + f"Finished decoding SFT metadata from {jsonl_url}. " + f"Records: {num_raw_records}, " + f"Duration > 61s: {num_filtered_duration}, " + f"Short edge < {min_short_edge}: {num_filtered_short_edge}, " + f"Windows: {num_raw_windows}, Windows < {min_frames}f: {num_filtered_windows}, " + f"Videos kept: {len(metadata_list)}" + ) + return metadata_list + + +def get_sft_dataset( + jsonl_paths: str | list[str] = "s3://nv-00-10206-vfm/cosmos3_video_sft/human_1k/captions_full.jsonl", + resolution: str = "720", + num_video_frames: int = 93, + temporal_interval_mode: str = "entire_chunk", + frame_selection_mode: str = "center", + tokenizer_config: Optional[Any] = None, + cfg_dropout_rate: float = 0.1, + use_system_prompt: bool = False, + append_duration_fps_timestamps: bool = True, + append_resolution_info: bool = True, + cfg_dropout_keep_metadata: bool = False, + sample_by_window: bool = False, + min_short_edge: int = 0, + caption_suffix: str = "", + conditioning_fps: float = 24, + conditioning_fps_noise_std: float = 0.0, + conditioning_config: dict[int, float] | None = None, + temporal_compression_factor: int = 4, + **kwargs, +) -> SFTDataset: + """Create SFT video dataset from one or more JSONL files on S3. + + Args: + jsonl_paths: S3 path(s) to JSONL metadata file(s). A single string or + a list of strings. When multiple files are given, their samples are + concatenated and each file's uuids are prefixed with ``/`` + to avoid collisions. + resolution: Output resolution (e.g., "720", "480") + num_video_frames: Number of frames to extract from each video. + Videos with fewer frames are skipped at decode time. + Use -1 to take native chunks from the t2w_window metadata. + temporal_interval_mode: How to compute the temporal interval between sampled frames. + "force_one" — always 1 (consecutive frames at original fps). + "max_30fps" — smallest interval that keeps effective fps <= 30. + "entire_chunk" — spread num_video_frames evenly across the whole window. + frame_selection_mode: Where to select frames within the window. + "center" — center-crop temporally (default). + "first" — take the first num_video_frames from the window start. + tokenizer_config: Config for the tokenizer + cfg_dropout_rate: Dropout rate for the caption + use_system_prompt: Whether to use the system prompt during tokenization + append_duration_fps_timestamps: If True, appends duration/FPS text to captions + append_resolution_info: If True, appends resolution text to captions + cfg_dropout_keep_metadata: If True, CFG dropout fires before appending + duration/FPS/resolution text so that metadata is preserved during + unconditional guidance. If False (default), dropout fires after + and clears the entire caption including metadata. + sample_by_window: If True, each t2w_window is treated as a separate + sample (the dataset length equals the total number of windows). + If False (default), each video uuid is one sample and a random + window is chosen on every access. + min_short_edge: Drop videos whose shortest spatial edge (min of width, + height) is below this value. 0 (default) disables the filter. + caption_suffix: Text appended to every caption before the + duration/FPS/resolution templates, e.g. + ``"Overall, the video is of poor quality."``. Empty string + (default) disables the suffix. + conditioning_fps: FPS value used for duration/FPS conditioning. + A positive value is used directly (default 24). A negative + value (e.g. ``-1``) means the actual effective FPS + (``original_fps / temporal_interval``) is used instead. + conditioning_fps_noise_std: Standard deviation of log-normal + multiplicative noise applied to ``conditioning_fps``. The FPS + is multiplied by ``exp(N(0, std))``. 0.0 (default) disables + the noise. + conditioning_config: Weighted distribution mapping latent-frame counts + to unnormalized probabilities for image-to-video conditioning. + Example: ``{0: 0.7, 1: 0.2, 2: 0.1}``. ``None`` disables + conditioning (all frames are generation targets). + temporal_compression_factor: VAE temporal compression factor used to + convert pixel frame count to latent frame count. + Returns: + SFTDataset instance + """ + log.info(f"Unknown kwargs for get_sft_dataset: {kwargs}") + assert resolution in VIDEO_RES_SIZE_INFO.keys(), "The provided resolution cannot be found in VIDEO_RES_SIZE_INFO." + + if isinstance(jsonl_paths, str): + jsonl_paths = [jsonl_paths] + + if INTERNAL: + with open("credentials/gcs.secret", "r") as f: + credentials = json.load(f) + else: + credentials = {} + + s3_client = boto3.client("s3", **credentials) + + metadata_list: list[dict] = [] + for idx, jsonl_url in enumerate(jsonl_paths): + prefix = f"{idx}/" if len(jsonl_paths) > 1 else "" + metadata_list.extend( + _load_sft_metadata_from_s3( + s3_client, + jsonl_url, + min_frames=61, + uuid_prefix=prefix, + min_short_edge=min_short_edge, + ) + ) + + total_windows = sum(len(m["t2w_windows"]) for m in metadata_list) + log.info( + f"Finished loading metadata from {len(jsonl_paths)} file(s). " + f"Total videos: {len(metadata_list)}, total windows: {total_windows}" + ) + + if sample_by_window: + metadata_list = _flatten_metadata_by_window(metadata_list) + log.info(f"sample_by_window=True: flattened to {len(metadata_list)} samples (one per window)") + + # Deterministic shuffle based on the sha256 hash of uuid + metadata_list.sort(key=lambda x: hashlib.sha256(x["uuid"].encode("utf-8")).hexdigest()) + + dataset = SFTDataset( + metadata=metadata_list, + num_video_frames=num_video_frames, + resolution=resolution, + s3_credentials=credentials, + temporal_interval_mode=temporal_interval_mode, + frame_selection_mode=frame_selection_mode, + tokenizer_config=tokenizer_config, + cfg_dropout_rate=cfg_dropout_rate, + use_system_prompt=use_system_prompt, + append_duration_fps_timestamps=append_duration_fps_timestamps, + append_resolution_info=append_resolution_info, + cfg_dropout_keep_metadata=cfg_dropout_keep_metadata, + caption_suffix=caption_suffix, + conditioning_fps=conditioning_fps, + conditioning_fps_noise_std=conditioning_fps_noise_std, + conditioning_config=conditioning_config, + temporal_compression_factor=temporal_compression_factor, + ) + return dataset diff --git a/cosmos_framework/data/vfm/packing_iterable_dataset.py b/cosmos_framework/data/vfm/packing_iterable_dataset.py new file mode 100644 index 0000000..847b85a --- /dev/null +++ b/cosmos_framework/data/vfm/packing_iterable_dataset.py @@ -0,0 +1,276 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +""" +Abstract base class for pool-based token-budget bin-packing over multiple datasets. + +Extracted from ``projects.cosmos3.vfm.datasets.vlm.joint_dataset_dynamic_batch_webloader`` +so that both the VLM and VFM internal dataloaders can share a single packing implementation. + +Usage +----- +Subclass and implement ``compute_sample_tokens(sample) -> int``. +Optionally override ``collate_batch(samples) -> Any`` for custom collation. +""" + +from __future__ import annotations + +import random +from abc import ABC, abstractmethod +from collections import deque +from enum import Enum +from typing import Any, Union + +import torch + +from cosmos_framework.utils.lazy_config import instantiate +from cosmos_framework.utils import log + + +class Modality(Enum): + IMAGE = "image" + VIDEO = "video" + TEXT = "text" + + +class PackingIterableDataset(torch.utils.data.IterableDataset, ABC): + """Pool-based greedy bin-packing IterableDataset. + + Maintains a pool of ``pool_size`` samples and assembles batches by + greedily selecting candidates that fit within the token budget + ``max_tokens``. Subclasses supply two hooks: + + * ``compute_sample_tokens(sample)`` — token cost of one sample (abstract). + * ``collate_batch(samples)`` — assemble a packed list into a batch + (default: identity, returns the list unchanged). + + Parameters + ---------- + datasets_cfg: + Mapping ``{name: {"dataset": , "ratio": }}``. + The *dataset* value may be a Hydra lazy config, an already-constructed + ``IterableDataset``, or a plain ``DataLoader`` (its ``.dataset`` is + unwrapped automatically). + max_tokens: + Token budget per batch (padded cost = ``cur_max_len * batch_size``). + pool_size: + Number of samples to buffer before selecting a batch. + max_batch_size: + Hard cap on items per batch (0 or None = no cap). + long_threshold: + Samples with token count ``>= long_threshold`` are emitted as + singletons regardless of budget. + batching_strategy: + ``"prefer_closest"`` (default) or ``"prefer_first"``. + apply_long_sample_halving: + When ``True`` (default), ``_max_tokens`` halves the budget for any + batch whose largest sample exceeds 1000 tokens — a memory-safety + heuristic. Set ``False`` only when memory headroom at the literal + ``max_tokens`` budget has been validated for the recipe. + """ + + def __init__( + self, + datasets_cfg: dict[str, dict[str, Union[int, object]]], + max_tokens: int, + pool_size: int, + max_batch_size: int, + long_threshold: int, + batching_strategy: str, + apply_long_sample_halving: bool = True, + ): + super().__init__() + + assert batching_strategy in ("prefer_first", "prefer_closest"), ( + f"batching_strategy must be 'prefer_first' or 'prefer_closest', got {batching_strategy!r}" + ) + + self.max_tokens = max_tokens + self.pool_size = pool_size + self.long_threshold = long_threshold + self.max_batch_size = max_batch_size + self.batching_strategy = batching_strategy + self.apply_long_sample_halving = apply_long_sample_halving + + self._pool: deque[dict] = deque() + self._dataset_names: list[str] = [] + self._ratios: list[float] = [] + self._datasets: list[torch.utils.data.IterableDataset] = [] + + for name, cfg in datasets_cfg.items(): + assert {"ratio", "dataset"} <= cfg.keys(), ( + f"Each entry must have 'dataset' and 'ratio' keys: {name} -> {cfg.keys()}" + ) + ratio = cfg["ratio"] + if ratio == 0: + log.info(f"Skipping dataset {name} with ratio {ratio}") + continue + dataset_cfg = cfg["dataset"] + + ds = ( + instantiate(dataset_cfg) + if not isinstance(dataset_cfg, (torch.utils.data.IterableDataset, torch.utils.data.DataLoader)) + else dataset_cfg + ) + if isinstance(ds, torch.utils.data.DataLoader): + ds = ds.dataset + if hasattr(ds, "build_dataset") and callable(getattr(ds, "build_dataset")): + ds = ds.build_dataset() + + assert isinstance(ds, torch.utils.data.IterableDataset), ( + f"Expected an IterableDataset, got {type(ds)} for {name}" + ) + + self._dataset_names.append(name) + self._ratios.append(float(ratio)) + self._datasets.append(ds) + log.info(f"Added dataset {name} with ratio {ratio}") + + log.info(f"added data: {list(datasets_cfg.keys())}") + assert len(self._datasets) > 0, "No datasets added" + self._data_len: int = sum(int(getattr(ds, "total_images", 0)) for ds in self._datasets) + if self._data_len == 0: + self._data_len = 10**12 + self.iterators = [iter(ds) for ds in self._datasets] + + # ------------------------------------------------------------------ + # Abstract / overridable hooks + # ------------------------------------------------------------------ + + @abstractmethod + def compute_sample_tokens(self, sample: dict) -> int: + """Return the token cost of one sample for packing budget accounting.""" + + def collate_batch(self, samples: list[dict]) -> Any: + """Assemble a packed list of samples into one batch. + + Default implementation returns the list unchanged (identity). + Override to pad, stack, or transform samples into tensors. + """ + return samples + + # ------------------------------------------------------------------ + # PyTorch Dataset API + # ------------------------------------------------------------------ + + def __len__(self) -> int: + return self._data_len + + def __iter__(self): + while True: + batch = self._best_fit_batch() + yield self.collate_batch(batch) + + # ------------------------------------------------------------------ + # Internal packing helpers (moved verbatim from _JointIterableDataset) + # ------------------------------------------------------------------ + + def _max_tokens(self, cur_max: int) -> int: + if not self.apply_long_sample_halving: + return self.max_tokens + if cur_max < 1000: + return self.max_tokens + return self.max_tokens // 2 + + def _get_next_sample(self) -> dict: + index_id = random.choices(range(len(self.iterators)), weights=self._ratios, k=1)[0] + curr_dataset = self.iterators[index_id] + try: + output = next(curr_dataset) + except StopIteration: + log.critical(f"dataset {self._dataset_names[index_id]} exhausted") + self.iterators[index_id] = iter(self._datasets[index_id]) + output = next(self.iterators[index_id]) + return output + + def _fill_pool(self): + while len(self._pool) < self.pool_size: + self._pool.append(self._get_next_sample()) + + def _padded_cost(self, cur_max: int, k: int) -> int: + return cur_max * k + + def _get_modality(self, sample: dict) -> Modality: + if "pixel_values" in sample: + return Modality.IMAGE + elif "pixel_values_videos" in sample: + return Modality.VIDEO + return Modality.TEXT + + def _best_fit_batch(self) -> list[dict]: + """Build one batch using the configured token-budget strategy.""" + self._fill_pool() + seed = self._pool.popleft() + seed_modality = self._get_modality(seed) + L0 = self.compute_sample_tokens(seed) + + if L0 >= self.long_threshold or L0 >= self._max_tokens(L0): + return [seed] + + chosen = [seed] + cur_max = L0 + + while self._pool: + if self.max_batch_size and len(chosen) >= self.max_batch_size: + break + best_idx = self._find_best_candidate(cur_max, len(chosen), seed_modality) + if best_idx is None: + break + cand = self._remove_from_pool(best_idx) + chosen.append(cand) + cur_max = max(cur_max, self.compute_sample_tokens(cand)) + + return chosen + + def _find_best_candidate(self, cur_max: int, num_chosen: int, seed_modality: Modality) -> int | None: + if self.batching_strategy == "prefer_first": + return self._find_best_candidate_prefer_first(cur_max, num_chosen, seed_modality) + return self._find_best_candidate_prefer_closest(cur_max, num_chosen, seed_modality) + + def _find_best_candidate_prefer_first(self, cur_max: int, num_chosen: int, seed_modality: Modality) -> int | None: + best_idx = None + best_new_tokens = None + for idx, cand in enumerate(self._pool): + if self._get_modality(cand) != seed_modality: + continue + L = self.compute_sample_tokens(cand) + new_max = max(cur_max, L) + new_tokens = self._padded_cost(new_max, num_chosen + 1) + if new_tokens <= self._max_tokens(cur_max): + if best_new_tokens is None or new_tokens < best_new_tokens: + best_new_tokens = new_tokens + best_idx = idx + return best_idx + + def _find_best_candidate_prefer_closest(self, cur_max: int, num_chosen: int, seed_modality: Modality) -> int | None: + best_idx = None + best_new_tokens = None + smallest_length_diff = None + for idx, cand in enumerate(self._pool): + if self._get_modality(cand) != seed_modality: + continue + L = self.compute_sample_tokens(cand) + new_max = max(cur_max, L) + new_tokens = self._padded_cost(new_max, num_chosen + 1) + if new_tokens <= self._max_tokens(cur_max): + length_diff = abs(L - cur_max) + if ( + best_new_tokens is None + or new_tokens < best_new_tokens + or (new_tokens == best_new_tokens and length_diff < smallest_length_diff) + ): + best_new_tokens = new_tokens + best_idx = idx + smallest_length_diff = length_diff + return best_idx + + def _remove_from_pool(self, idx: int) -> dict: + if idx == 0: + return self._pool.popleft() + elif idx == len(self._pool) - 1: + return self._pool.pop() + else: + self._pool.rotate(-idx) + item = self._pool.popleft() + self._pool.rotate(idx) + return item diff --git a/cosmos_framework/data/vfm/packing_iterable_dataset_test.py b/cosmos_framework/data/vfm/packing_iterable_dataset_test.py new file mode 100644 index 0000000..2fb7f4f --- /dev/null +++ b/cosmos_framework/data/vfm/packing_iterable_dataset_test.py @@ -0,0 +1,78 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Unit tests for the ``apply_long_sample_halving`` knob added to +:class:`cosmos_framework.data.vfm.packing_iterable_dataset.PackingIterableDataset` with +the wandb + DataPacker spec (2026-05-21-toml-interface-wandb-datapacker-design.md). +""" + +from __future__ import annotations + +import torch + +from cosmos_framework.data.vfm.packing_iterable_dataset import PackingIterableDataset + + +class _StubIterable(torch.utils.data.IterableDataset): + """Trivial finite IterableDataset; the halving tests don't iterate it.""" + + def __iter__(self): + yield from () + + +class _StubPacking(PackingIterableDataset): + """Minimal concrete subclass; ``_max_tokens`` is the SUT, so we just need + a constructable instance — ``compute_sample_tokens`` isn't called by these + tests.""" + + def compute_sample_tokens(self, sample: dict) -> int: # pragma: no cover - unused + return 0 + + +def _make(apply_long_sample_halving: bool = True) -> _StubPacking: + return _StubPacking( + datasets_cfg={"default": {"dataset": _StubIterable(), "ratio": 1.0}}, + max_tokens=45056, + pool_size=16, + max_batch_size=1, + long_threshold=6400, + batching_strategy="prefer_closest", + apply_long_sample_halving=apply_long_sample_halving, + ) + + +# ----- halving heuristic ---------------------------------------------------- + + +def test_default_applies_halving_above_threshold(): + """Default behavior: cur_max >= 1000 triggers ``max_tokens // 2``.""" + ds = _make() + assert ds.apply_long_sample_halving is True + assert ds._max_tokens(999) == 45056 # below threshold → full budget + assert ds._max_tokens(1000) == 22528 # at threshold → halved + assert ds._max_tokens(5000) == 22528 # well above → halved + + +def test_halving_disabled_keeps_full_budget(): + """``apply_long_sample_halving=False`` returns ``max_tokens`` literally.""" + ds = _make(apply_long_sample_halving=False) + assert ds.apply_long_sample_halving is False + assert ds._max_tokens(999) == 45056 + assert ds._max_tokens(1000) == 45056 # would have been halved with default + assert ds._max_tokens(50_000) == 45056 + + +def test_halving_default_is_true_when_unspecified(): + """Backwards compat: constructing without the new kwarg keeps the original + (halving-active) behavior bit-for-bit — every existing recipe is unchanged. + """ + ds = _StubPacking( + datasets_cfg={"default": {"dataset": _StubIterable(), "ratio": 1.0}}, + max_tokens=10_000, + pool_size=16, + max_batch_size=1, + long_threshold=6400, + batching_strategy="prefer_closest", + ) + assert ds.apply_long_sample_halving is True + assert ds._max_tokens(2000) == 5000 diff --git a/cosmos_framework/data/vfm/processors/__init__.py b/cosmos_framework/data/vfm/processors/__init__.py new file mode 100644 index 0000000..2a21fb8 --- /dev/null +++ b/cosmos_framework/data/vfm/processors/__init__.py @@ -0,0 +1,144 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +import json +import os +import sys +from types import SimpleNamespace +from typing import Optional + +from transformers import PreTrainedTokenizerFast + +from cosmos_framework.data.vfm.processors.base import BaseVLMProcessor +from cosmos_framework.data.vfm.processors.nemotron3densevl_processor import Nemotron3DenseVLProcessor +from cosmos_framework.data.vfm.processors.nemotronvl_processor import NemotronVLProcessor +from cosmos_framework.data.vfm.processors.qwen3vl_processor import Qwen3VLProcessor +from cosmos_framework.model.vfm.tokenizers.tokenization_qwen2 import Qwen2Tokenizer +from cosmos_framework.utils.vfm.vlm.pretrained_models_downloader import maybe_download_hf_model_from_s3 + +_VARIANT_TO_CREDENTIALS = { + "s3": ("credentials/s3_training.secret", "checkpoints-us-east-1"), + "gcp": ("credentials/gcp_checkpoint.secret", "nv-00-10206-checkpoint"), + # "hf" => no S3 backing store: pass empty credentials/bucket so the downloader + # falls back to a direct HuggingFace Hub download (matches the legacy + # ``download_tokenizer_files(model_name, "hf")`` behavior on origin/main, which + # simply returned the model name and let from_pretrained pull from HF). + "hf": ("", ""), +} + +# S3 prefix under which HuggingFace model files are stored in the checkpoint buckets. +_LLM_S3_PREFIX = "cosmos3/pretrained/huggingface" + + +class LLMTokenizerProcessor(BaseVLMProcessor): + """Wrapper that adapts a bare LLM tokenizer to the ``BaseVLMProcessor`` API. + + Used by LLM-only (no-vision) tokenizer configs so that all augmentors and + model code can treat LLM-only and full VLM configs uniformly through the + same ``proc.tokenizer`` / ``proc.tokenize_text`` surface. The base class + handles ``tokenize_text`` / ``encode`` / ``decode``; we only need to wire + up ``self.processor`` so ``.tokenizer`` resolves. + """ + + def __init__(self, tokenizer): + self.processor = SimpleNamespace(tokenizer=tokenizer) + + +def _patch_nemotron_llm_tokenizer_vision_tokens(destination_dir: str) -> None: + """Remap reserved placeholder tokens to vision special tokens in-place. + + The Nemotron LLM tokenizer reserves ```` / ```` + at IDs 20/21 -- the same slots the VLM tokenizer uses for + ``<|vision_start|>`` / ``<|vision_end|>``. Renaming them here keeps + every vision-token ID inside the original vocab_size (131072) so no + embedding-layer resize is needed during FSDP training. The function is + idempotent: re-applying it after the tokens are already renamed is a no-op. + """ + remap = {"": "<|vision_start|>", "": "<|vision_end|>"} + + tokenizer_json_path = os.path.join(destination_dir, "tokenizer.json") + if os.path.exists(tokenizer_json_path): + with open(tokenizer_json_path) as f: + data = json.load(f) + for entry in data.get("added_tokens", []): + if entry["content"] in remap: + entry["content"] = remap[entry["content"]] + vocab = data.get("model", {}).get("vocab", {}) + for old_name, new_name in remap.items(): + if old_name in vocab: + vocab[new_name] = vocab.pop(old_name) + with open(tokenizer_json_path, "w") as f: + json.dump(data, f) + + tokenizer_config_path = os.path.join(destination_dir, "tokenizer_config.json") + if os.path.exists(tokenizer_config_path): + with open(tokenizer_config_path) as f: + tc_data = json.load(f) + for entry in tc_data.get("added_tokens_decoder", {}).values(): + if entry.get("content") in remap: + entry["content"] = remap[entry["content"]] + with open(tokenizer_config_path, "w") as f: + json.dump(tc_data, f) + + +def _download_llm_tokenizer( + tokenizer_type: str, + credentials: str, + bucket: str, + cache_dir: Optional[str] = None, +) -> str: + return maybe_download_hf_model_from_s3( + tokenizer_type, + credentials=credentials, + bucket=bucket, + include_model_weights=False, + cache_dir=cache_dir, + s3_prefix=_LLM_S3_PREFIX, + ) + + +def build_processor( + tokenizer_type: str, + config_variant: Optional[str] = None, + credentials: Optional[str] = None, + bucket: Optional[str] = None, + cache_dir: Optional[str] = None, +): + if credentials is None or bucket is None: + if config_variant is None: + config_variant = "s3" + if config_variant not in _VARIANT_TO_CREDENTIALS: + raise ValueError(f"config_variant must be one of {list(_VARIANT_TO_CREDENTIALS)}, got {config_variant!r}") + variant_credentials, variant_bucket = _VARIANT_TO_CREDENTIALS[config_variant] + credentials = credentials if credentials is not None else variant_credentials + bucket = bucket if bucket is not None else variant_bucket + elif config_variant is not None: + raise ValueError("Provide either config_variant or (credentials, bucket), not both") + if "Qwen/Qwen3-VL" in tokenizer_type or "Siglip2-Qwen3-1.7B" in tokenizer_type: + return Qwen3VLProcessor(tokenizer_type, credentials=credentials, bucket=bucket, cache_dir=cache_dir) + elif "nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16" in tokenizer_type: + return NemotronVLProcessor(tokenizer_type, credentials=credentials, bucket=bucket, cache_dir=cache_dir) + elif "NVIDIA-Nemotron-3-Dense-VL" in tokenizer_type or "Qwen3-2B-ViT" in tokenizer_type: + return Nemotron3DenseVLProcessor(tokenizer_type, credentials=credentials, bucket=bucket, cache_dir=cache_dir) + elif "Qwen/Qwen3-0.6B" in tokenizer_type: + local_path = _download_llm_tokenizer(tokenizer_type, credentials, bucket, cache_dir) + return LLMTokenizerProcessor(Qwen2Tokenizer.from_pretrained(local_path)) + elif "Nemotron/NVIDIA-Nemotron-3-2B-BF16" in tokenizer_type: + local_path = _download_llm_tokenizer(tokenizer_type, credentials, bucket, cache_dir) + _patch_nemotron_llm_tokenizer_vision_tokens(local_path) + return LLMTokenizerProcessor(PreTrainedTokenizerFast.from_pretrained(local_path, trust_remote_code=True)) + else: + raise ValueError(f"Tokenizer type {tokenizer_type} not supported") + + +def build_processor_lazy(*args, **kwargs): + """LazyCall wrapper that resolves ``build_processor`` on this module at call time. + + LazyCall captures its target at config-construction time, so a direct + ``L(build_processor)`` would freeze the original function reference and + bypass any later ``monkeypatch.setattr`` on this module's + ``build_processor`` attribute. This wrapper performs a fresh module-level + lookup on every call, so test fixtures patching ``build_processor`` are + honored when the config is instantiated. + """ + return sys.modules[__name__].build_processor(*args, **kwargs) diff --git a/cosmos_framework/data/vfm/processors/base.py b/cosmos_framework/data/vfm/processors/base.py new file mode 100644 index 0000000..88ad4a6 --- /dev/null +++ b/cosmos_framework/data/vfm/processors/base.py @@ -0,0 +1,192 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +"""Base class and shared helpers for VFM/VLM processor wrappers. + +Each concrete processor wraps a HuggingFace ``AutoProcessor`` for a specific +model family and exposes a small surface used by dataloaders and the training +model: + +* ``apply_chat_template`` -- model-specific message templating (per subclass) +* ``add_assistant_tokens_mask`` -- model-specific loss mask construction +* ``tokenizer`` -- the underlying HF tokenizer (uniform property) +* ``tokenize_text`` / ``encode`` / ``decode`` -- simple delegations + +This module hosts the parts that were truly common across subclasses so the +concrete files only contain model-specific logic. +""" + +import os +from typing import Dict, List, Optional + +from transformers.models.auto.processing_auto import AutoProcessor + +from cosmos_framework.utils import log +from cosmos_framework.model.vfm.vlm.qwen3_vl.utils import tokenize_caption +from cosmos_framework.utils.vfm.vlm.pretrained_models_downloader import maybe_download_hf_model_from_s3 + + +def convert_string_content_to_list_content(messages: List[Dict]) -> List[Dict]: + """Normalize chat messages so ``content`` is always a list of typed dicts. + + Many HF processors do not accept ``"content": str``; they expect each + message's content to be a list of ``{"type": ..., ...}`` entries. This + helper rewrites bare-string contents into a single ``{"type": "text", ...}`` + entry in place and returns the same list for convenience. + """ + for i, message in enumerate(messages): + if isinstance(message["content"], str): + messages[i]["content"] = [{"type": "text", "text": message["content"]}] + return messages + + +def maybe_parse_video_content( + messages: List[Dict], +) -> tuple[int, Optional[list[float]], Optional[list[int]], Optional[list[list[int]]]]: + """Scan messages for video entries and return their decoding metadata. + + Returns ``(num_video, fps_per_video, total_frames_per_video, frame_indices_per_video)``. + Logs a critical warning when a video entry omits ``fps``. + """ + num_video = 0 + video_fps: list[float] = [] + video_total_num_frames: list[int] = [] + video_frames_indices: list[list[int]] = [] + for message in messages: + if isinstance(message["content"], list): + for sub_content in message["content"]: + if sub_content.get("type", "") == "video" and isinstance(sub_content["video"], list): + num_video += 1 + fps = sub_content.get("fps", None) + if fps is None: + log.critical( + f"fps is None for video {sub_content}. Better to set the fps explicitly", rank0_only=False + ) + video_fps.append(fps) + video_total_num_frames.append(len(sub_content["video"])) + video_frames_indices.append(list(range(video_total_num_frames[-1]))) + return num_video, video_fps, video_total_num_frames, video_frames_indices + + +def maybe_get_max_pixels_from_images_kwargs(messages: List[Dict]) -> tuple[Optional[int], Optional[int]]: + """Return ``(max_pixels, min_pixels)`` from the first image entry that sets ``max_pixels``.""" + for message in messages: + if isinstance(message["content"], list): + for sub_content in message["content"]: + if sub_content.get("type", "") == "image" and sub_content.get("max_pixels", None) is not None: + return sub_content["max_pixels"], sub_content.get("min_pixels", None) + return None, None + + +class BaseVLMProcessor: + """Shared skeleton for VFM/VLM processor wrappers. + + Subclasses inherit the S3-or-local model resolution, the + ``AutoProcessor`` load, and the extraction of common token IDs. They + are responsible only for: + + * the chat templating logic (``apply_chat_template``); + * the loss-mask construction (``add_assistant_tokens_mask``); + * any model-specific dataloader helper fields (e.g. ``patch_size``, + ``merge_size``, ``use_smart_resize``). + + A subclass that needs a different pad-id resolution (e.g. NemotronVL's + ```` convention) overrides :py:meth:`_resolve_pad_id`. A + subclass that needs a different vision-end marker sets the + ``VISION_END_TOKEN`` class attribute; the default ``None`` skips that + lookup entirely (used for tokenizers that lack a single-token marker). + """ + + # Override on subclasses to the model's vision-end token (e.g. ``""``). + # Leave as None when the tokenizer does not expose a single-token marker — + # ``vision_end_id`` will then be set to None and downstream consumers + # (e.g. ``debug_data_qwen.py``) will skip the check. + VISION_END_TOKEN: Optional[str] = None + + def __init__( + self, + name: str, + credentials: str = "./credentials/s3_training.secret", + bucket: str = "checkpoints-us-east-1", + cache_dir: Optional[str] = None, + ) -> None: + self.name = name + if os.path.isdir(name): + model_name_or_path_local = name + else: + model_name_or_path_local = maybe_download_hf_model_from_s3( + name, credentials, bucket, include_model_weights=False, cache_dir=cache_dir + ) + + self.processor = AutoProcessor.from_pretrained(model_name_or_path_local, trust_remote_code=True) + log.info("Successfully loaded processor from local cache") + + self.image_token_id = ( + self.processor.tokenizer.convert_tokens_to_ids(self.processor.image_token) + if hasattr(self.processor, "image_token") + else None + ) + self.video_token_id = ( + self.processor.tokenizer.convert_tokens_to_ids(self.processor.video_token) + if hasattr(self.processor, "video_token") + else None + ) + self.eos_id = self.processor.tokenizer.eos_token_id + self.pad_id = self._resolve_pad_id() + self.vision_end_id = ( + self.processor.tokenizer.convert_tokens_to_ids(self.VISION_END_TOKEN) + if self.VISION_END_TOKEN is not None + else None + ) + + # ------------------------------------------------------------------ + # Hooks for subclasses + # ------------------------------------------------------------------ + + def _resolve_pad_id(self): + """Return the pad token id. Default: ``pad_token_id`` falling back to ``eos_id``. + + Override on subclasses whose model uses a non-standard pad token (e.g. + NemotronVL uses ````). + """ + pad = self.processor.tokenizer.pad_token_id + return pad if pad is not None else self.eos_id + + # ------------------------------------------------------------------ + # Shared interfaces + # ------------------------------------------------------------------ + + @property + def tokenizer(self): + """Expose the underlying HF tokenizer uniformly. + + Lets model and test code call ``proc.tokenizer`` regardless of which + concrete processor wrapper they received. + """ + return self.processor.tokenizer + + def tokenize_text( + self, + caption: str, + is_video: bool = False, + use_system_prompt: bool = False, + system_prompt: Optional[str] = None, + ) -> list[int]: + """Tokenize a text caption via the shared ``tokenize_caption`` helper. + + Keeps VFM diffusion augmentors and VLM dataloaders on the same code + path so a single processor instance serves both. + """ + return tokenize_caption( + caption, + self.processor.tokenizer, + is_video=is_video, + use_system_prompt=use_system_prompt, + system_prompt=system_prompt, + ) + + def encode(self, *args, **kwargs): + return self.processor.tokenizer.encode(*args, **kwargs) + + def decode(self, *args, **kwargs): + return self.processor.tokenizer.decode(*args, **kwargs) diff --git a/cosmos_framework/data/vfm/processors/nemotron3densevl_processor.py b/cosmos_framework/data/vfm/processors/nemotron3densevl_processor.py new file mode 100644 index 0000000..4d27438 --- /dev/null +++ b/cosmos_framework/data/vfm/processors/nemotron3densevl_processor.py @@ -0,0 +1,181 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Optional + +import numpy as np +import torch +from PIL import Image +from qwen_vl_utils.vision_process import smart_resize + +from cosmos_framework.data.vfm.processors.base import ( + BaseVLMProcessor, + convert_string_content_to_list_content, + maybe_parse_video_content, +) + + +class Nemotron3DenseVLProcessor( + BaseVLMProcessor +): + """Wrapper around the HuggingFace ``AutoProcessor`` for Nemotron3-Dense-VL / Qwen3-2B-ViT.""" + + # Nemotron3-Dense / Qwen3-2B-ViT does not expose a single vision-end token; + # leave ``vision_end_id`` unset (None) rather than the legacy ```` + # which silently resolved to the UNK token id. + VISION_END_TOKEN: Optional[str] = None + + def __init__( + self, + name: str = "Qwen/Qwen3-VL-2B-Init", + credentials: str = "./credentials/s3_training.secret", + bucket: str = "checkpoints-us-east-1", + cache_dir: Optional[str] = None, + ): + super().__init__(name=name, credentials=credentials, bucket=bucket, cache_dir=cache_dir) + # Helper attributes consumed by the dataloader video decoding path. + shortest_edge = self.processor.image_processor.size["shortest_edge"] + self.min_height_width = int(np.sqrt(shortest_edge)) + self.patch_size = self.processor.video_processor.patch_size + self.temporal_patch_size = self.processor.video_processor.temporal_patch_size + self.merge_size = self.processor.video_processor.merge_size + self.use_smart_resize = True + + def apply_chat_template( + self, + messages, + add_generation_prompt=False, + return_tensors="pt", + tokenize=True, + **kwargs, + ): + """ + Return: + inputs: dict + input_ids: torch.Tensor, shape: (N_token) + attention_mask: torch.Tensor, shape: (N_token) + texts: str, the raw text + image_sizes: torch.Tensor, shape (N_img, 2) + pixel_values: torch.Tensor, shape (N_img_patch, 3, 224, 224) + """ + assert tokenize, "tokenize must be True" + assert return_tensors == "pt", "return_tensors must be pt" + # Note: this tokenizer does not support "content": str, it always expect "content" entry to be a list of dicts + messages = convert_string_content_to_list_content(messages) + kwargs = {} + # Pre-resize images per-message using smart_resize so the resulting + # token count matches the configured min/max-pixel budget. + for message in messages: + if isinstance(message["content"], list): + for sub_content in message["content"]: + if sub_content.get("type", "") == "image": + image = sub_content["image"] + max_pixels = sub_content.get("max_pixels", self.processor.image_processor.size["longest_edge"]) + min_pixels = sub_content.get("min_pixels", self.processor.image_processor.size["shortest_edge"]) + assert isinstance(image, Image.Image), ( + "image must be a url string for now, not support list of images for one content" + ) + width, height = image.size + resized_height, resized_width = smart_resize( + height, + width, + factor=32, + min_pixels=min_pixels, + max_pixels=max_pixels, + ) + image = image.resize((resized_width, resized_height)) + sub_content["image"] = image + + num_video, video_fps, video_total_num_frames, video_frames_indices = maybe_parse_video_content(messages) + if num_video > 0: + assert num_video == 1, "only support one video for now" + fps = video_fps[0] + total_num_frames = video_total_num_frames[0] + frames_indices = video_frames_indices[0] + kwargs.update( + { + "do_sample_frames": False, + "video_metadata": dict(fps=fps, total_num_frames=total_num_frames, frames_indices=frames_indices), + } + ) + + inputs = self.processor.apply_chat_template( + messages, + tokenize=tokenize, + add_generation_prompt=add_generation_prompt, + return_dict=True, + return_tensors=return_tensors, + **kwargs, + ) + + # Convert batch features into single features + inputs["input_ids"] = inputs["input_ids"][0] # [N_token] + inputs["attention_mask"] = inputs["attention_mask"][0] # [N_token] + return inputs + + def add_assistant_tokens_mask(self, tokens): + """ + Add a mask to the assistant tokens. + This is used to mask out tokens that are not generated by the assistant (e.g., system prompts, user prompts, chat templates), such that in the loss computation, only the tokens generated by the assistant are used. + If there are multiple turns in the conversation, the mask will mask all the assistant tokens in each turn. + + Args: + tokens (Union[List[int], torch.Tensor]): The tokens to add the mask to. + Returns: + Union[List[bool], torch.Tensor]: The mask. True for tokens generated by the assistant (i.e. should apply loss on), False for tokens not generated by the assistant. + """ + if isinstance(tokens, torch.Tensor) and tokens.ndim == 2: + mask = torch.stack( + [self.add_assistant_tokens_mask(tokens[i]) for i in range(tokens.shape[0])] + ) # [B,N_token] + assert mask.shape == tokens.shape + return mask + np_tokens = tokens.cpu().numpy() if isinstance(tokens, torch.Tensor) else np.array(tokens) + assert np_tokens.ndim == 1 + + # Constants defining bos, eos and fixed offsets. + BOS_TOKEN = "<|im_start|>" + EOS_TOKEN = "<|im_end|>" + ROLE = "assistant" + # Offsets: skip the bos + "assistant\n" (always 3 tokens) and include the eos (+1) for supervision + START_OFFSET = 3 + END_OFFSET = 1 + + # Retrieve token IDs for the markers and the role. + bos_token_id = self.processor.tokenizer.convert_tokens_to_ids(BOS_TOKEN) + eos_token_id = self.processor.tokenizer.convert_tokens_to_ids(EOS_TOKEN) + role_id = self.processor.tokenizer.convert_tokens_to_ids(ROLE) + # ``role`` may tokenize into multiple sub-tokens (e.g. Qwen3-2B-ViT + # splits "assistant"); the multi-token branch below handles that case. + role_ids = self.processor.tokenizer.encode(ROLE, add_special_tokens=False) + think_start_id = self.processor.tokenizer.convert_tokens_to_ids("") + think_end_id = self.processor.tokenizer.convert_tokens_to_ids("") + + # Locate all positions where the start and end markers appear. + start_indices = np.where(np_tokens == bos_token_id)[0] + end_indices = np.where(np_tokens == eos_token_id)[0] + + # Initialize the mask with False values. + masks = np.zeros_like(np_tokens, dtype=bool) + assert len(start_indices) == len(end_indices) + # For each pair of bos/eos, check if the role is 'assistant' + # and apply the mask accordingly. + for start, end in zip(start_indices, end_indices): + end_pos = None + if np_tokens[start + 1] == role_id: + # Mask tokens from after the assistant header (start+3) to include the end marker (end+1) + masks[start + START_OFFSET : end + END_OFFSET] = True + end_pos = start + START_OFFSET + elif all(np_tokens[start + 1 : start + 1 + len(role_ids)] == role_ids): + masks[start + START_OFFSET + len(role_ids) - 1 : end + END_OFFSET] = True + end_pos = start + START_OFFSET + len(role_ids) - 1 + if end_pos is not None and np_tokens[end_pos] == think_start_id: + masks[end_pos] = False + if np_tokens[end_pos + 1] == think_end_id: + masks[end_pos + 1] = False + + assert masks.shape == np_tokens.shape + if isinstance(tokens, torch.Tensor): + return torch.from_numpy(masks) + else: + return masks.tolist() diff --git a/cosmos_framework/data/vfm/processors/nemotronvl_processor.py b/cosmos_framework/data/vfm/processors/nemotronvl_processor.py new file mode 100644 index 0000000..a67b95c --- /dev/null +++ b/cosmos_framework/data/vfm/processors/nemotronvl_processor.py @@ -0,0 +1,472 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 + +from typing import Dict, List, Optional + +import numpy as np +import torch +from PIL import Image +from transformers.processing_utils import VideosKwargs +from transformers.video_utils import VideoMetadata + +from cosmos_framework.utils import log +from cosmos_framework.data.vfm.processors.base import BaseVLMProcessor, convert_string_content_to_list_content + +nemotron_chat_template = """ +{%- set ns = namespace(enable_thinking=false, has_sys_prompt=false, non_tool_system_content='', has_video=false, explicit_think_requested=false) -%} +{%- set msg = namespace(content='') -%} +{%- for message in messages -%} + {%- if message['role'] == 'system' -%} + {%- set ns.has_sys_prompt = true -%} + {# Extract system content without tool flags #} + {%- if message['content'] is string -%} + {%- set ns.non_tool_system_content = message['content'].replace('', '<_end_think>').replace('/think', '').replace('/no_think', '').replace('<_end_think>', '').strip() -%} + {%- else -%} + {%- set ns.non_tool_system_content = '' -%} + {%- for content in message['content'] -%} + {%- if content['type'] == 'text' -%} + {%- set ns.non_tool_system_content = ns.non_tool_system_content + content['text'].replace('', '<_end_think>').replace('/think', '').replace('/no_think', '').replace('<_end_think>', '') -%} + {%- endif -%} + {%- endfor -%} + {%- set ns.non_tool_system_content = ns.non_tool_system_content.strip() -%} + {%- endif -%} + {%- endif -%} + {# Check for video content in all messages #} + {%- if message['content'] is not string -%} + {%- for content in message['content'] -%} + {%- if content['type'] == 'video' or content['type'] == 'video_url' -%} + {%- set ns.has_video = true -%} + {%- endif -%} + {%- endfor -%} + {%- endif -%} + {%- if message['content'] is string -%} + {%- if message['role'] == 'user' or message['role'] == 'system' -%} + {%- if '/think' in message['content'].replace('', '') -%} + {%- set ns.enable_thinking = true -%} + {%- set ns.explicit_think_requested = true -%} + {%- elif '/no_think' in message['content'] -%} + {%- set ns.enable_thinking = false -%} + {%- endif -%} + {%- endif -%} + {%- else -%} + {%- for content in message['content'] -%} + {%- if content['type'] == 'text' -%} + {%- if message['role'] == 'user' or message['role'] == 'system' -%} + {%- if '/think' in content['text'].replace('', '') -%} + {%- set ns.enable_thinking = true -%} + {%- set ns.explicit_think_requested = true -%} + {%- elif '/no_think' in content['text'] -%} + {%- set ns.enable_thinking = false -%} + {%- endif -%} + {%- endif -%} + {%- endif -%} + {%- endfor -%} + {%- endif -%} +{%- endfor -%} + +{{- bos_token -}} +{%- if messages[0]['role'] != 'system' -%} + {{- 'System\n' -}} +{%- else -%} + {{- 'System\n' + ns.non_tool_system_content }} +{%- endif -%} + +{%- if tools -%} + {%- if ns.non_tool_system_content != '' -%} + {{- '\n\n' -}} + {%- endif -%} + {{- 'You can use the following tools to assist the user if required:\n' -}} + {{- '[' -}} + {%- for tool in tools -%} + {{- (tool.function if tool.function is defined else tool) | tojson -}} + {{- ', ' if not loop.last else '' -}} + {%- endfor -%} + {{- ']\n\n' -}} + + {{- 'If you decide to call any tool(s), use the following format:\n' -}} + {{- '[{"name": "tool_name1", "arguments": "tool_args1"}, ' -}} + {{- '{"name": "tool_name2", "arguments": "tool_args2"}]\n\n' -}} + + {{- 'The user will execute tool-calls and return responses from tool(s) in this format:\n' -}} + {{- '[{"response": "tool_response1"}, ' -}} + {{- '{"response": "tool_response2"}]\n\n' -}} + + {{- 'Based on the tool responses, you can call additional tools if needed, ' -}} + {{- 'correct tool calls if any errors are found, or just respond to the user.' -}} +{%- endif -%} +{{- '\n' -}} + +{%- set messages = messages[1:] if messages[0]['role'] == 'system' else messages -%} + +{# Prevent no user or assistant message #} +{%- if messages|length == 0 -%} + {%- set messages = [{'role': 'user', 'content': ''}] -%} +{%- endif -%} + +{%- for message in messages %} + {%- if message['content'] is string -%} + {%- set msg.content = message['content'].replace('', '<_end_think>').replace('/think', '').replace('/no_think', '').replace('<_end_think>', '').strip() -%} + {%- else -%} + {%- set msg.content = '' -%} + {%- set mm_content = '' -%} + {%- set counters = namespace(images=0, videos=0) -%} + + {%- for content in message['content'] -%} + {%- if content['type'] == 'image' -%} + {%- set counters.images = counters.images + 1 -%} + {%- elif content['type'] == 'video' -%} + {%- set counters.videos = counters.videos + 1 -%} + {%- elif content['type'] == 'text' -%} + {%- set msg.content = msg.content + content['text'] -%} + {%- endif -%} + {%- endfor -%} + {%- if '' in msg.content -%} + {%- set counters.images = 0 -%} + {%- endif -%} + {%- if '