Skip to content

Fix & enrich tokenization example script#296

Merged
timofeev1995 merged 2 commits intonextfrom
add-position-ids-to-tokenization-example
Mar 11, 2026
Merged

Fix & enrich tokenization example script#296
timofeev1995 merged 2 commits intonextfrom
add-position-ids-to-tokenization-example

Conversation

@timofeev1995
Copy link
Contributor

@timofeev1995 timofeev1995 commented Mar 10, 2026

Adds positional ids to tokenization script & fixes incorrect sequence splitting.

@timofeev1995 timofeev1995 changed the title Fix script Fix & enrich tokenization example script Mar 10, 2026
@timofeev1995 timofeev1995 requested review from artek0chumak, blainekasten and fdrose and removed request for blainekasten March 11, 2026 10:22
@timofeev1995 timofeev1995 merged commit 0b30876 into next Mar 11, 2026
8 of 9 checks passed
stainless-app bot pushed a commit that referenced this pull request Mar 11, 2026
* Fix script

* Add parquet columnt to the list
stainless-app bot added a commit that referenced this pull request Mar 11, 2026
* fix: fixed autoscaling configs in deployments api

* fix(jig): send {} for deployment autoscaling to unset if unset in config (#294)

* fix: fix autoscaling config usage in jig (#298)

* Fix & enrich tokenization example script (#296)

* Fix script

* Add parquet columnt to the list

* feat(jig): move config.dockerfile to config.image.dockerfile_path and add an config.deploy.image option so that you don't have to always pass it as a flag (#287)

* chore: fix lints (#299)

* release: 2.4.0

---------

Co-authored-by: stainless-app[bot] <142633134+stainless-app[bot]@users.noreply.github.com>
Co-authored-by: Dulaj Disanayaka <8828757+dulaj-me@users.noreply.github.com>
Co-authored-by: Egor Timofeev <43988181+timofeev1995@users.noreply.github.com>
Co-authored-by: technillogue <technillogue@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants