Refactor tokenizer with builders #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

DongyuZhao merged 1 commit into user/donz/refactor from codex/apitokenizer

Jul 21, 2025

Owner

DongyuZhao commented Jul 21, 2025

Summary

implement Markdown token builders for the new CodeTokenizer
inject the default token builders into MarkdownLanguage
add minimal tests verifying CodeTokenizer with new builders
fix inline code builder and adjust basic tokenizer tests

Testing

swift test --filter CodeTokenizerBasicTests.testSingleCharacterToken
swift test --filter CodeTokenizerBasicTests.testInlineCode
swift test

https://chatgpt.com/codex/tasks/task_e_687e0743aec4832298d00b957c12b96c


          Fix token builder and adjust tests

DongyuZhao added the codex label

— with

ChatGPT Codex Connector

DongyuZhao merged commit 7c355be into user/donz/refactor

DongyuZhao deleted the codex/apitokenizer branch

August 7, 2025 15:52

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels