docs: sync miner guide to recipe 1.4.0 - #4
Conversation
Mirror BASE docs/external-miner/prism.md: miner-chosen tokenizer, source-tree ZIP caps, G5 pretrain-only (RULER/BABILong/natural), and baseline use of ctx["tokenizer"] / vocab_size (gpt2 remains a fallback choice, not a challenge rule).
📝 WalkthroughWalkthroughThe PR updates Prism documentation and the baseline example for recipe v1.4.0. It documents source-tree submissions, injected tokenizers, v3 scoring and API routes, Zone B reporting, G5 evaluation, intake limits, and terminal cap outcomes. ChangesRecipe 1.4.0 contract
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/getting-started.md`:
- Around line 48-58: The tokenizer contract in build_tokenizer must cover the
baseline’s batch call and BatchEncoding behavior, including batched text input,
truncation, padding, return_tensors="pt", .to(device), and enc.input_ids.
Document and enforce these requirements, or update the baseline to use only the
currently documented single-text interface; keep the chosen contract consistent
throughout the baseline and getting-started guide.
In `@docs/scoring.md`:
- Around line 3-12: In docs/scoring.md, explicitly define measured_bpb as the
tokenizer-neutral bits_per_byte value used for live scoring, while
distinguishing it from the legacy per-token bpb key. In README.md lines 29-30,
replace the “bits-per-byte (bpb)” wording with the canonical
measured_bpb/bits_per_byte name and preserve the distinction from legacy bpb.
In `@docs/submit.md`:
- Line 96: Update the training-only intake description near the two-script
layout statement to identify it as the legacy training seam: training.py is
submitted with arch_id, while the architecture is retrieved from the registry.
Remove the claim that the payload is limited to a two-script layout and keep the
surrounding table and follow-up paragraph consistent.
- Around line 55-56: Add the normative ≤ 8 MiB compressed ZIP limit to the
source-tree cap summary in docs/submit.md:55-56. Also update the 400-error
checklist in docs/troubleshooting.md:5 to include both the compressed limit and
the tokenizer/ sub-cap, preserving the existing intake limits.
- Around line 35-52: Update the packaging example so the Python snippet reads
the ZIP from the path actually created by the preceding zip command, and write
its JSON output to submission.json for the subsequent curl command instead of
stdout. Keep the existing archive contents, payload fields, and submission
endpoint unchanged.
In `@examples/baseline/training.py`:
- Around line 61-62: Update the tokenizer setup around tok.pad_token assignment
so EOS is not aliased as padding when it is the only available special token.
Keep padding unset in that case and adjust the batching/loss path to use
padding=False or mask only attention-mask padding positions with
ignore_index=-100 before cross_entropy, preserving EOS targets in the loss.
In `@README.md`:
- Around line 48-50: Update the README instructions around the baseline example
so only prism_telemetry.report is described as required; identify
finish_evaluation as an optional early-stop signal while retaining the
ctx["tokenizer"] requirement.
- Line 39: Update the recipe summaries to replace the vague “natural docs” label
with the exact natural G5 pack names, “natural MCQ” and “HELMET RAG”; apply this
change in README.md lines 39-39 and docs/README.md lines 12-14 so both summaries
use identical naming.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 6354e155-91e7-43da-8f7c-34e9916afd30
📒 Files selected for processing (10)
README.mddocs/README.mddocs/api.mddocs/getting-started.mddocs/scoring.mddocs/submit.mddocs/troubleshooting.mdexamples/baseline/README.mdexamples/baseline/architecture.pyexamples/baseline/training.py
| ```python | ||
| # architecture.py | ||
| def build_tokenizer(ctx): | ||
| """Anything offline: train a BPE on ctx["dataset_path"], wrap a vendored | ||
| implementation, or hand-roll a byte-level tokenizer. Must satisfy: | ||
|
|
||
| tok(text, add_special_tokens=False)["input_ids"] -> list[int] | ||
| tok.decode(ids) -> str # roundtrips plain ASCII | ||
| len(tok) or tok.vocab_size -> int # 256 .. 262144 | ||
| tok.eos_token_id -> int | None | ||
| """ |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 3 'def build_tokenizer|ctx\["tokenizer"\]|return_tensors|padding|input_ids|attention_mask|\.to\(device\)' \
docs/getting-started.md examples/baseline/training.pyRepository: BaseIntelligence/prism
Length of output: 4343
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '--- docs/getting-started.md relevant section ---\n'
sed -n '1,90p' docs/getting-started.md
printf '\n--- examples/baseline/training.py relevant section ---\n'
sed -n '48,105p' examples/baseline/training.pyRepository: BaseIntelligence/prism
Length of output: 251
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- docs/getting-started.md relevant section ---'
sed -n '1,90p' docs/getting-started.md
printf '%s\n' ''
printf '%s\n' '--- examples/baseline/training.py relevant section ---'
sed -n '48,105p' examples/baseline/training.pyRepository: BaseIntelligence/prism
Length of output: 6191
Document the full baseline tokenizer contract.
docs/getting-started.md only specifies single-text tokenization, decode, vocab size, and eos_token_id. The baseline passes a list of texts and calls tok(batch_txt, return_tensors="pt", truncation=True, max_length=block, padding=True).to(device) before reading enc.input_ids. Either document and enforce the batch / BatchEncoding / padding contract, or change the baseline to use the documented minimum.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/getting-started.md` around lines 48 - 58, The tokenizer contract in
build_tokenizer must cover the baseline’s batch call and BatchEncoding behavior,
including batched text input, truncation, padding, return_tensors="pt",
.to(device), and enc.input_ids. Document and enforce these requirements, or
update the baseline to use only the currently documented single-text interface;
keep the chosen contract consistent throughout the baseline and getting-started
guide.
| ## Pure bpb (live leaf) | ||
|
|
||
| `final_score = score_from_bpb(measured_bpb)` on the integer lattice `[0, SCORE_MAX]` — | ||
| lower bpb, higher score. The LLM reviews are **gates, not graders**: they verify the | ||
| submission is coherent and not cheating; their quality notes never move the score. | ||
|
|
||
| **Fairness across tokenizers.** `bits_per_byte` (bits over UTF-8 bytes of the scored | ||
| region) is the tokenizer-neutral anchor reported in `METRICS_JSON`. The legacy `bpb` | ||
| key is bits per *token* and is only comparable across submissions that share a | ||
| tokenizer. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Files:\n'
git ls-files | rg '(^|/)(README\.md|docs/(scoring|README|submit|troubling|.*\.md)$)' || true
printf '\nRelevant docs excerpts:\n'
sed -n '1,80p' README.md 2>/dev/null || true
printf '\n--- docs/scoring.md ---\n'
sed -n '1,60p' docs/scoring.md 2>/dev/null || true
printf '\nSearch for exact identifiers:\n'
rg -n "score_from_bpb|measured_bpb|bits_per_byte|bits-per-byte|bits per|METRICS_JSON|bpb" README.md docs 2>/dev/null || trueRepository: BaseIntelligence/prism
Length of output: 8656
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'docs/getting-started.md excerpts:\n'
sed -n '1,90p' docs/getting-started.md 2>/dev/null || true
printf '\nFull occurrences of relevant terms with filenames:\n'
rg -n -C 2 "measured_bpb|score_from_bpb|bits_per_byte|METRICS_JSON|bits-per-byte|bits per (token|byte)|bpb" README.md docs 2>/dev/null || true
printf '\nExact source-line probe for current claimed occurrences:\n'
python3 - <<'PY'
from pathlib import Path
files = ["README.md", "docs/scoring.md", "docs/getting-started.md"]
terms = ["measured_bpb", "score_from_bpb", "bits_per_byte", "METRICS_JSON", "bits-per-byte", "bits per token"]
for path in files:
try:
text = Path(path).read_text(encoding="utf-8")
except FileNotFoundError as e:
print(f"{path}: MISSING\n")
continue
lines = text.splitlines()
print(f"\n--- {path} ({len(lines)} lines) ---")
hits, term_hits = [], {t: [] for t in terms}
for i, line in enumerate(lines, 1):
if any(t in line for t in terms):
hits.append(i)
for t in terms:
if t in line:
term_hits[t].append(i)
print("term hits:", {t: hits_ for t, hits_ in term_hits.items() if hits_})
for i in (1,29,30,39,43,50,13,63,67,68) if path=="docs/getting-started.md" else (1,5,6,9,11):
if 1 <= i <= len(lines):
print(f"{i}: {lines[i-1]}")
PYRepository: BaseIntelligence/prism
Length of output: 11122
Use one canonical name for the live-score input.
docs/scoring.md introduces measured_bpb without saying it equals the tokenizer-neutral bits_per_byte value, and README.md says live score is “bits-per-byte (bpb)” without distinguishing the legacy per-token bpb key. Define whether measured_bpb is bits_per_byte and replace the README equivalence with the canonical metric name.
📍 Affects 2 files
docs/scoring.md#L3-L12(this comment)README.md#L29-L30
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/scoring.md` around lines 3 - 12, In docs/scoring.md, explicitly define
measured_bpb as the tokenizer-neutral bits_per_byte value used for live scoring,
while distinguishing it from the legacy per-token bpb key. In README.md lines
29-30, replace the “bits-per-byte (bpb)” wording with the canonical
measured_bpb/bits_per_byte name and preserve the distinction from legacy bpb.
| ```bash | ||
| # pack the tree (paths relative to project root) | ||
| cd my-submission | ||
| zip -r ../tree.zip . -x '*.pyc' -x '__pycache__/*' -x '.git/*' | ||
|
|
||
| python3 - <<'PY' | ||
| import base64, json, pathlib | ||
| raw = pathlib.Path("tree.zip").read_bytes() | ||
| print(json.dumps({ | ||
| "miner_hotkey": "<64 lowercase hex>", | ||
| "zip_base64": base64.b64encode(raw).decode(), | ||
| "label": "optional", | ||
| })) | ||
| PY | ||
|
|
||
| curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \ | ||
| -H 'content-type: application/json' \ | ||
| -d @submission.json |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Fix the source-tree packaging example before publishing it.
After cd my-submission, the ZIP is written to ../tree.zip, but the Python snippet reads tree.zip from my-submission. The snippet also prints JSON to stdout, while the next command reads submission.json, which the snippet never creates.
Proposed fix
-python3 - <<'PY'
+python3 - <<'PY' > submission.json
import base64, json, pathlib
-raw = pathlib.Path("tree.zip").read_bytes()
+raw = pathlib.Path("../tree.zip").read_bytes()
print(json.dumps({📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| ```bash | |
| # pack the tree (paths relative to project root) | |
| cd my-submission | |
| zip -r ../tree.zip . -x '*.pyc' -x '__pycache__/*' -x '.git/*' | |
| python3 - <<'PY' | |
| import base64, json, pathlib | |
| raw = pathlib.Path("tree.zip").read_bytes() | |
| print(json.dumps({ | |
| "miner_hotkey": "<64 lowercase hex>", | |
| "zip_base64": base64.b64encode(raw).decode(), | |
| "label": "optional", | |
| })) | |
| PY | |
| curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \ | |
| -H 'content-type: application/json' \ | |
| -d @submission.json |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/submit.md` around lines 35 - 52, Update the packaging example so the
Python snippet reads the ZIP from the path actually created by the preceding zip
command, and write its JSON output to submission.json for the subsequent curl
command instead of stdout. Keep the existing archive contents, payload fields,
and submission endpoint unchanged.
| Caps: ≤ 128 files, ≤ 4 MiB/file, ≤ 16 MiB total uncompressed; `tokenizer/` ≤ 12 | ||
| files / ≤ 8 MiB. The validated tree is staged on the pod under `submission/`. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Keep all intake cap summaries complete.
The normative cap table includes an 8 MiB compressed ZIP limit, but these intake summaries omit it.
docs/submit.md#L55-L56: Add≤ 8 MiB compressedto the source-tree caps.docs/troubleshooting.md#L5-L5: Mirror the compressed limit and tokenizer-directory sub-cap in the 400-error checklist.
📍 Affects 2 files
docs/submit.md#L55-L56(this comment)docs/troubleshooting.md#L5-L5
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/submit.md` around lines 55 - 56, Add the normative ≤ 8 MiB compressed
ZIP limit to the source-tree cap summary in docs/submit.md:55-56. Also update
the 400-error checklist in docs/troubleshooting.md:5 to include both the
compressed limit and the tokenizer/ sub-cap, preserving the existing intake
limits.
|
|
||
| Training-only entries are **separate slots**: one accepted entry per `(hotkey, arch_id)` | ||
| — you may train on many published architectures, one script per arch. | ||
| Training-only intake accepts the **two-script** layout only (not a full source tree). |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Correct the training-only layout description.
Training-only intake sends training.py with arch_id; the architecture is pulled from the registry. “Two-script layout only” conflicts with the table and the next paragraph. Describe this as the legacy training seam, not as a two-script payload.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/submit.md` at line 96, Update the training-only intake description near
the two-script layout statement to identify it as the legacy training seam:
training.py is submitted with arch_id, while the architecture is retrieved from
the registry. Remove the claim that the payload is limited to a two-script
layout and keep the surrounding table and follow-up paragraph consistent.
| if getattr(tok, "pad_token", None) is None and getattr(tok, "eos_token", None) is not None: | ||
| tok.pad_token = tok.eos_token |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 3 'eos_token|pad_token|pad_token_id|attention_mask|ignore_index' \
examples/baseline/training.pyRepository: BaseIntelligence/prism
Length of output: 928
🏁 Script executed:
#!/bin/bash
set -euo pipefail
wc -l examples/baseline/training.py
sed -n '1,130p' examples/baseline/training.pyRepository: BaseIntelligence/prism
Length of output: 4423
🌐 Web query:
Transformers TokenizerBase __init__ pad_token_id eos_token padding attribute source
💡 Result:
In the Hugging Face Transformers library, PreTrainedTokenizerBase serves as the foundational class for all tokenizer implementations [1][2]. It defines and manages special tokens and padding behavior through several key attributes and properties [3][4]. Special Token Attributes The base class maintains a list of special token names in the class attribute SPECIAL_TOKENS_ATTRIBUTES, which includes bos_token, eos_token, unk_token, sep_token, pad_token, cls_token, and mask_token [1][5]. When a tokenizer is initialized, these are often set via keyword arguments [2][6]. Properties and IDs The class provides property methods to retrieve the integer ID for these tokens, such as pad_token_id and eos_token_id [4][7]. These properties dynamically look up the token's ID in the tokenizer's vocabulary [4]. Specifically: - pad_token_id: Returns the ID of the pad_token [4]. If the token has not been set, it returns None [4]. - eos_token_id: Returns the ID of the eos_token [4]. Similarly, it returns None if not set [4]. - pad_token_type_id: An integer attribute (defaulting to 0) used in conjunction with padding logic [5][6][4]. Padding and Configuration Padding configuration is managed at the tokenizer level using several attributes: - padding_side: Defines whether padding is applied to the left or right of the sequence (defaults to right) [1][6][8]. - Padding Logic: The actual padding process (e.g., during tokenizer.call or via the pad method) utilizes the pad_token_id, pad_token_type_id, and padding_side configuration to create uniform tensors for batching [3][8][9]. During init, the base class processes keyword arguments, ensures no conflicts occur between user-provided arguments and class methods, and handles backward compatibility for special token configuration [2][5][6].
Citations:
- 1: https://github.com/huggingface/transformers/blob/v5.14.0/src/transformers/tokenization_utils_base.py
- 2: https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization%5Futils%5Fbase.py
- 3: https://huggingface.co/docs/transformers/internal/tokenization_utils
- 4: https://github.com/huggingface/transformers/blob/v4.18.0/src/transformers/tokenization_utils_base.py
- 5: https://github.com/huggingface/transformers/blob/9ea2afbc/src/transformers/tokenization_utils_base.py
- 6: https://github.com/huggingface/transformers/blob/v5.0.0rc0/src/transformers/tokenization_utils_base.py
- 7: https://www.huggingface.co/transformers/v4.6.0/_modules/transformers/tokenization_utils_base.html
- 8: https://huggingface.co/docs/transformers/v5.12.0/en/internal/tokenization_utils
- 9: https://huggingface.co/docs/transformers/main/en/internal/tokenization_utils
🌐 Web query:
Transformers calling tokenizer padding=True raises RuntimeError pad_token_id is None source
💡 Result:
The RuntimeError or ValueError indicating that pad_token_id is None occurs because many modern Transformer models (such as GPT-2, LLaMA, and others) are not pre-configured with a designated padding token [1][2][3]. When you call a tokenizer with padding=True, the library attempts to pad the sequences using a specific token ID, but if that ID is not set in the tokenizer or the model configuration, the process fails [1][4][3]. To resolve this issue, you must explicitly define a padding token for your tokenizer. A common and standard practice is to use an existing special token, such as the End-of-Sentence (EOS) token, as the padding token [3][5]. Recommended Fix: You can set the pad_token and update the model configuration as follows: # Set the pad_token to the eos_token tokenizer.pad_token = tokenizer.eos_token # If you are fine-tuning, it is often necessary to resize the model's token embeddings # to account for any new tokens added to the tokenizer model.resize_token_embeddings(len(tokenizer)) # Ensure the model configuration also has the pad_token_id set model.config.pad_token_id = tokenizer.pad_token_id Alternatively, if you prefer to use a specific padding token (e.g., [PAD]), you can add it directly: tokenizer.add_special_tokens({'pad_token': '[PAD]'}) model.resize_token_embeddings(len(tokenizer)) model.config.pad_token_id = tokenizer.pad_token_id Note: Always ensure that your attention masks are correctly applied when processing padded sequences to prevent the model from attending to the padding tokens, which can otherwise lead to incorrect hidden states [6][7].
Citations:
- 1: https://discuss.huggingface.co/t/padding-token-missing-from-llama/150901
- 2: tensor mismatch when finetuning a smollm3 model huggingface/transformers#41129
- 3: ValueError: Unable to set proper padding strategy as the tokenizer does not have a padding token. huggingface/transformers#3859
- 4: fix: improve error message when
pad_token_idis not configured huggingface/trl#1152 - 5: run_lm_finetuning.py for GPT2 throw error "Using pad_token, but it is not set yet." huggingface/transformers#2648
- 6: https://huggingface.co/docs/transformers/v5.9.0/troubleshooting
- 7: https://huggingface.co/docs/transformers/en/troubleshooting
🌐 Web query:
Transformers tokenizer padding=True tokenizers backends check pad_token_id is None source fast tokenizer
💡 Result:
When using the Hugging Face Transformers library, setting padding=True instructs the tokenizer to pad sequences to the longest sequence in the batch [1][2]. If your tokenizer's pad_token_id is None (common in many base models like Llama or GPT-2), the library will raise a ValueError during batch encoding because it cannot determine which token to use for padding [3]. The ValueError typically states: "Asking to pad but the tokenizer does not have a padding token" [3]. To resolve this, you must explicitly assign a padding token to the tokenizer before performing the batch operation. Common approaches include: 1. Setting the pad token to an existing special token (e.g., the EOS token): tokenizer.pad_token = tokenizer.eos_token 2. Adding a new, dedicated padding token: tokenizer.add_special_tokens({'pad_token': '[PAD]'}) If you are using a fast tokenizer (the default TokenizersBackend), these settings are shared and propagated to the underlying Rust-based implementation [4][5]. After setting the pad_token, you should also ensure your model's configuration reflects this change if necessary (e.g., model.config.pad_token_id = tokenizer.pad_token_id), especially if you are performing generation tasks [6]. Using the call method on your tokenizer is recommended for performance, as it handles the fast backend execution efficiently [7].
Citations:
- 1: https://huggingface.co/docs/transformers/main/en/pad_truncation
- 2: https://huggingface.co/docs/transformers/en/internal/tokenization_utils
- 3: LlamaTokenizer has no
padtoken, leading to failure during batch-tokenization huggingface/transformers#22312 - 4: https://huggingface.co/docs/transformers/fast%5Ftokenizers
- 5: https://github.com/huggingface/transformers/blob/main/docs/source/en/fast_tokenizers.md
- 6: ValueError: If
eos_token_idis defined, make sure thatpad_token_idis defined. huggingface/transformers#25353 - 7: https://huggingface.co/docs/transformers/internal/tokenization_utils
🌐 Web query:
"hugging face tokenizers" "pad token" "padding is enabled but the tokenizer does not have a padding token"
💡 Result:
The error "Asking to pad but the tokenizer does not have a padding token" occurs in Hugging Face Transformers when you attempt to pad sequences (e.g., in a batch) using a tokenizer that was not configured with a dedicated pad token [1][2][3]. This is common with models that do not have a default padding token defined, such as LLaMA or GPT-2 [3]. To resolve this issue, you must explicitly define a padding token for the tokenizer [1][2][4]. You can do this using one of the following methods: Option 1: Assign an existing special token as the pad token (commonly the EOS token) [1][3]. tokenizer.pad_token = tokenizer.eos_token Option 2: Add a new, dedicated pad token to the tokenizer's vocabulary [1][2][3]. tokenizer.add_special_tokens({'pad_token': '[PAD]'}) After setting the pad token, you should ensure that your tokenizer's padding configuration is correct [4]. If you are using the underlying Tokenizers library directly, you can use the enable_padding method [5][6]: tokenizer.enable_padding(pad_token='[PAD]', pad_id=0) Important considerations: - When using the EOS token as a pad token, be mindful of whether the model's architecture or training data expects a distinct padding behavior [3]. - If you add a new special token, ensure that the model itself is aware of it, often requiring you to resize the model's token embeddings if you are continuing pre-training or fine-tuning [3]. - If you are training a tokenizer from scratch, you can include [PAD] in the special_tokens list during the initial training phase to avoid this issue later [4].
Citations:
- 1: https://discuss.huggingface.co/t/asking-to-pad-but-the-tokenizer-does-not-have-a-padding-token/85344
- 2: https://discuss.huggingface.co/t/padding-not-working-when-loading-a-tokenizer-trained-via-the-tokenizers-library-into-transformers/42326/2
- 3: LlamaTokenizer has no
padtoken, leading to failure during batch-tokenization huggingface/transformers#22312 - 4: https://discuss.huggingface.co/t/tokenizer-pad-token-what/24367
- 5: https://huggingface.co/docs/tokenizers/api/tokenizer
- 6: https://www.huggingface.co/docs/tokenizers/python/latest/api/reference.html
🌐 Web query:
HuggingFace Tokenizers Rust Padded padding enabled but tokenizer does not have a padding token source pad_token_id
💡 Result:
The error "padding enabled but tokenizer does not have a padding token" typically occurs when using the Hugging Face transformers library, which requires an explicit pad_token to be set on the tokenizer instance for its padding-related functions to work [1][2][3]. While the lower-level tokenizers (Rust) library allows you to configure padding parameters (like pad_id and pad_token) via enable_padding() [4][5][6], these settings are not always automatically recognized as the designated pad_token when the tokenizer is loaded into the higher-level transformers PreTrainedTokenizerFast class [1][2]. To resolve this error, you must explicitly assign a pad token to the tokenizer object [7][8]. Common solutions include: 1. Setting an existing token as the pad token (e.g., if using a model like GPT-2 that lacks a default pad token) [7][8]: tokenizer.pad_token = tokenizer.eos_token 2. Adding a new special token for padding [1][2][3]: tokenizer.add_special_tokens({'pad_token': '[PAD]'}) If you are working with the tokenizers (Rust) library directly, you configure padding by calling enable_padding() with the desired parameters [4][5][6]: tokenizer.enable_padding( direction=PaddingDirection::Right, // or Left pad_id=0, pad_token="[PAD]".to_string, //... other parameters); The disconnect often arises because transformers maintains its own pad_token property to manage vocabulary and model embedding consistency, which must be explicitly synced when transferring configurations from the Rust tokenizers library [1][2]. If you are loading a custom tokenizer from a file, ensure that any padding configuration saved in the JSON file is correctly mapped to the pad_token attribute upon initialization in transformers [1][2].
Citations:
- 1: https://discuss.huggingface.co/t/padding-not-transferring-when-loading-a-tokenizer-trained-via-the-tokenizers-library-into-transformers/42872
- 2: https://discuss.huggingface.co/t/padding-not-working-when-loading-a-tokenizer-trained-via-the-tokenizers-library-into-transformers/42326/2
- 3: https://discuss.huggingface.co/t/asking-to-pad-but-the-tokenizer-does-not-have-a-padding-token/85344
- 4: https://huggingface.co/docs/tokenizers/main/api/tokenizer
- 5: https://huggingface.co/docs/tokenizers/python/latest/api/reference.html?highlight=padding
- 6: https://huggingface.co/docs/tokenizers/api/tokenizer
- 7: ValueError: Unable to set proper padding strategy as the tokenizer does not have a padding token. huggingface/transformers#3859
- 8: GPT-2 asking for Padding Token huggingface/transformers#12594
🌐 Web query:
tokenizer.py tokenizers_tokenizer RustTokenizer padding pad_token_id None source PreTrainedTokenizerFast fast
💡 Result:
When using PreTrainedTokenizerFast in the Hugging Face Transformers library, encountering a pad_token_id of None is a common occurrence because the underlying Rust-based tokenizer object does not automatically infer or map special token concepts from its configuration [1][2]. Key points regarding this behavior include: 1. Manual Configuration Required: Unlike standard Transformers tokenizers that might be pre-configured for specific models, a PreTrainedTokenizerFast instantiated directly from a tokenizer object (or loaded from a file without a complete tokenizer_config.json) does not automatically recognize which token in the vocabulary is intended for padding [1][2]. You must manually assign the pad_token and other special tokens during initialization or by using methods like add_special_tokens [1][3][2]. 2. Handling pad_token_id is None: If pad_token_id is None, any attempt to perform padding operations—such as calling the tokenizer with padding=True—will fail or trigger warnings because the tokenizer does not know which ID to use for the padding operation [2][4]. 3. Technical Context: The PreTrainedTokenizerFast acts as a Python wrapper around the Rust-based tokenizers library [5][6]. While the Rust Tokenizer struct has a with_padding method to set padding parameters [7][8], these must be explicitly configured and synchronized with the Transformers wrapper's special token attributes [1][2]. To resolve this, you should explicitly set the pad_token after initialization [1][2]: tokenizer = PreTrainedTokenizerFast(tokenizer_object=rust_tokenizer) tokenizer.pad_token = "[PAD]" # Or your specific padding token tokenizer.pad_token_id = tokenizer.convert_tokens_to_ids("[PAD]") Additionally, ensure your tokenizer's post-processor is configured correctly if you are building the tokenizer manually [1]. If you are seeing warnings even when not using padding, it is often due to legacy issues where the tokenizer checks for the pad_token_id property unnecessarily; this is generally handled by the library, but ensuring the token is defined is the best practice [9][10].
Citations:
- 1: PretrainedTokenizerFast from Tokenizer Does Not Keep The Same Properties? huggingface/tokenizers#777
- 2: https://discuss.huggingface.co/t/cannot-create-an-identical-pretrainedtokenizerfast-object-from-a-tokenizer-created-by-tokenizers-library/9317/2
- 3: https://github.com/datamol-io/safe/blob/main/safe/tokenizer.py
- 4: https://huggingface.co/transformers/v3.0.2/main_classes/tokenizer.html
- 5: https://huggingface.co/docs/transformers/fast%5Ftokenizers
- 6: https://huggingface.co/docs/transformers/main_classes/tokenizer
- 7: https://docs.rs/tokenizers/latest/tokenizers/tokenizer/struct.Tokenizer.html
- 8: https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/tokenizer/mod.rs
- 9: GPT2TokenizerFast raises pad_token error even if not used huggingface/transformers#4764
- 10: Don't access pad_token_id if there is no pad_token huggingface/transformers#4773
🌐 Web query:
GPT2TokenizerFast from_pretrained "gpt2" pad_token padding behavior
💡 Result:
The GPT-2 tokenizer, including GPT2TokenizerFast, does not have a default pad_token when loaded via from_pretrained("gpt2") [1][2][3]. Because GPT-2 was not trained with a padding token, it lacks this attribute by default [3][4]. If you attempt to use padding (e.g., by calling the tokenizer with padding='max_length' or passing a batch of sequences to the model) without first setting a pad_token, the Hugging Face Transformers library will raise a ValueError [3][4][5]. To resolve this behavior, you must manually assign a pad_token [3][4]. The common recommended practice is to set the pad_token to the existing eos_token [3][4]: tokenizer = GPT2TokenizerFast.from_pretrained("gpt2") tokenizer.pad_token = tokenizer.eos_token When using this for batching, it is also standard practice to set the padding side to 'left' to avoid issues with the model's causal language modeling objective [4]: tokenizer.padding_side = 'left' This configuration allows the tokenizer to perform padding without requiring you to add a new special token to the model's vocabulary, which would otherwise necessitate resizing the model's embedding layer [4].
Citations:
- 1: https://huggingface.co/docs/transformers/v4.57.0/model_doc/gpt2
- 2: https://huggingface.co/docs/transformers/v5.0.0rc0/en/model_doc/gpt2
- 3: GPT-2 asking for Padding Token huggingface/transformers#12594
- 4: Pad token for GPT2 and OpenAIGPT models huggingface/transformers#2630
- 5: run_lm_finetuning.py for GPT2 throw error "Using pad_token, but it is not set yet." huggingface/transformers#2648
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 4 'tokenizer_contract|tokenizer contract|eos_token_id|pad_token|pad_token_id|build_tokenizer|padding_side|return_tensors="pt"|padding=' . -g '!*.pyc' -g '!node_modules' -g '!dist' -g '!build'Repository: BaseIntelligence/prism
Length of output: 10877
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
class Tok:
def __init__(self, eos_token=None, pad_token=None):
self.eos_token = eos_token
self.pad_token = pad_token
tok = Tok(eos_token="<|endoftext|>")
if getattr(tok, "pad_token", None) is None and getattr(tok, "eos_token", None) is not None:
tok.pad_token = tok.eos_token
print(tok.pad_token, getattr(tok, "pad_token_id", None), getattr(tok, "eos_token", None) is not None)
tok2 = Tok(pad_token=None, eos_token=None)
same_pad_token_id = getattr(tok2, "pad_token", None) is None and getattr(tok2, "pad_token_id", None) is None
print(same_pad_token_id)
PYRepository: BaseIntelligence/prism
Length of output: 189
Avoid aliasing EOS as the padding token before loss masking.
Lines 61-62 let any eos_token become pad_token, so pad_token_id becomes the real EOS ID. If ids[:, 1:] contains EOS targets, ignore_index=pad_id drops their loss. Keep padding unset when EOS is the only special token, and then batch with padding=False, or mask only attention-mask padding positions with ignore_index=-100 before calling cross_entropy.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/baseline/training.py` around lines 61 - 62, Update the tokenizer
setup around tok.pad_token assignment so EOS is not aliased as padding when it
is the only available special token. Keep padding unset in that case and adjust
the batching/loss path to use padding=False or mask only attention-mask padding
positions with ignore_index=-100 before cross_entropy, preserving EOS targets in
the loss.
| | Staging gateway | `http://staging.api.joinbase.ai` | | ||
| | Submit path | `/challenge/prism/v1/submissions` | | ||
| | Recipe | v1.2.0 — telemetry hooks required | | ||
| | Recipe | **v1.4.0** — miner-chosen tokenizer; G5 = RULER + BABILong + natural docs (**pretrain-only**) | |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Use exact names for the natural G5 packs.
“Natural docs” does not identify the two separate scored packs.
README.md#L39-L39: Name natural MCQ and HELMET RAG explicitly.docs/README.md#L12-L14: Use the same exact G5 names in the recipe summary.
📍 Affects 2 files
README.md#L39-L39(this comment)docs/README.md#L12-L14
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` at line 39, Update the recipe summaries to replace the vague
“natural docs” label with the exact natural G5 pack names, “natural MCQ” and
“HELMET RAG”; apply this change in README.md lines 39-39 and docs/README.md
lines 12-14 so both summaries use identical naming.
| 2. Copy [`examples/baseline/`](examples/baseline/) — required telemetry hooks | ||
| (`prism_telemetry.report` + `finish_evaluation`) and `ctx["tokenizer"]`. | ||
| 3. Zip and submit — see [Submit](docs/submit.md). |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Mark finish_evaluation() as optional.
prism_telemetry.report(...) is required. finish_evaluation() is an optional early-stop signal. Grouping both under “required telemetry hooks” gives miners an incorrect contract.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` around lines 48 - 50, Update the README instructions around the
baseline example so only prism_telemetry.report is described as required;
identify finish_evaluation as an optional early-stop signal while retaining the
ctx["tokenizer"] requirement.
|
Prod audit note: live |
|
Hold — do not merge yet. Production still runs recipe 1.2.0 ( Please leave open until after a prod pin cutover to 1.4.0; merging earlier would contradict the live recipe miners hit at Docs-only clarity for current prod (train_rows vs full-shard streaming) shipped separately in #5. |
1 participant
Summary
docs/external-miner/prism.mdonprism-better@02c1fd89(recipe 1.4.0).build_tokenizer/tokenizer/; gpt2 is fallback only), source-tree ZIP caps, and G5 pretrain-only long-context (RULER + BABILong + natural MCQ/HELMET RAG).examples/baselineto usectx["tokenizer"]/ctx["vocab_size"]instead of hardcoding hubgpt2as the challenge rule.Test plan
Summary by CodeRabbit