Add length column to tokenize_and_chunk output

[jammastergirish]

Summary

When data.chunk_length > 0, dataset preparation routes through tokenize_and_chunk (bergson/data.py:700). That function previously returned only input_ids and doc_ids. Every downstream consumer expects a length column too:

• bergson/build.py:87,100 — build worker
• bergson/score/score.py:290,313 — score worker
• bergson/hessians/hessian_approximations.py:159

The non-chunked path adds length implicitly because tokenize (data.py:633) passes return_length=True to the tokenizer. The chunked path skipped that entirely, so any build / score / hessian run with chunk_length > 0 crashed at the first batch:

ValueError: Column 'length' doesn't exist.

Fix

Every chunk produced by tokenize_and_chunk has length chunk_size by construction, so this is a one-line addition to chunk_batch's return dict:

return {
    "input_ids": token_chunks,
    "doc_ids": doc_chunks,
    "length": [chunk_size] * n_chunks,
}

How this surfaced

Hit while running the new build → score pipeline example (on the upcoming PR in add_build_score_pipeline_example_yaml), which uses chunk_length: 1024. Both the build and score steps would have crashed on the same missing column. We also saw this on earlier work.