{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":624523753,"defaultBranch":"main","name":"tokenizer","ownerLogin":"tiktoken-go","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2023-04-06T16:56:55.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/130084571?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1715370555.0","currentOid":""},"activityList":{"items":[{"before":"70d9b595feda6b98d6339ee4bab5f7a0c7c997f0","after":"7f65fe7d582fc7772cd74271d20dfad52fb8611a","ref":"refs/heads/main","pushedAt":"2024-05-10T19:46:48.000Z","pushType":"pr_merge","commitsCount":3,"pusher":{"login":"bluescreen10","name":"Mariano Wahlmann","path":"/bluescreen10","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/86094?s=80&v=4"},"commit":{"message":"Merge pull request #6 from espadolini/lazy-load\n\nGenerate embedded vocabulary maps lazily","shortMessageHtmlLink":"Merge pull request #6 from espadolini/lazy-load"}},{"before":"94e23ef2a7f6a2832126c7ace1001258220d3cd5","after":"70d9b595feda6b98d6339ee4bab5f7a0c7c997f0","ref":"refs/heads/main","pushedAt":"2023-04-19T13:07:29.000Z","pushType":"pr_merge","commitsCount":2,"pusher":{"login":"bluescreen10","name":"Mariano Wahlmann","path":"/bluescreen10","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/86094?s=80&v=4"},"commit":{"message":"Merge pull request #3 from philippgille/patch-1\n\nUpdate alternatives","shortMessageHtmlLink":"Merge pull request #3 from philippgille/patch-1"}},{"before":"876e66b44d595542c8a0251b0a5bbadfdbb9b205","after":"94e23ef2a7f6a2832126c7ace1001258220d3cd5","ref":"refs/heads/main","pushedAt":"2023-04-11T13:21:45.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"bluescreen10","name":"Mariano Wahlmann","path":"/bluescreen10","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/86094?s=80&v=4"},"commit":{"message":"๐ fixed bug with spaces\n\nGo's regexp doesn't allow for look-around so had to re-write the\nregexp but that had a bug with multiple spaces. Switched to regexp2\nthat supports the native tiktoken regexp","shortMessageHtmlLink":"๐ fixed bug with spaces"}},{"before":"4868b0b5dc763f5886b8956dd4a9c9749aa7ae7e","after":"876e66b44d595542c8a0251b0a5bbadfdbb9b205","ref":"refs/heads/main","pushedAt":"2023-04-11T02:30:06.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"bluescreen10","name":"Mariano Wahlmann","path":"/bluescreen10","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/86094?s=80&v=4"},"commit":{"message":"๐ moved cli into its own folder\n\nbefore it was not possible to install the cli","shortMessageHtmlLink":"๐ moved cli into its own folder"}},{"before":"12a3097aa3fad93104c64a7e774d974879e42927","after":"4868b0b5dc763f5886b8956dd4a9c9749aa7ae7e","ref":"refs/heads/main","pushedAt":"2023-04-09T21:44:32.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"bluescreen10","name":"Mariano Wahlmann","path":"/bluescreen10","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/86094?s=80&v=4"},"commit":{"message":"๐ช clean up the bpe algorithm\n\nRenamed variables struct to reflect what they do a little better.","shortMessageHtmlLink":"๐ช clean up the bpe algorithm"}},{"before":"859c61e7dc23560fefb62403e163a1afd80814cf","after":"12a3097aa3fad93104c64a7e774d974879e42927","ref":"refs/heads/main","pushedAt":"2023-04-08T15:00:03.000Z","pushType":"pr_merge","commitsCount":2,"pusher":{"login":"bluescreen10","name":"Mariano Wahlmann","path":"/bluescreen10","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/86094?s=80&v=4"},"commit":{"message":"Merge pull request #1 from MarvinJWendt/patch-1\n\n๐ fixed usage example in readme","shortMessageHtmlLink":"Merge pull request #1 from MarvinJWendt/patch-1"}},{"before":"1b538b580912c6e4e1c8e8ba72e82aece232bd9f","after":"859c61e7dc23560fefb62403e163a1afd80814cf","ref":"refs/heads/main","pushedAt":"2023-04-07T15:10:06.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"bluescreen10","name":"Mariano Wahlmann","path":"/bluescreen10","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/86094?s=80&v=4"},"commit":{"message":"๐ fixed tests\n\nAfter comfirming the expected behavior changed tests to match OpenAI's\nbehavior. It should now work as expected.","shortMessageHtmlLink":"๐ fixed tests"}},{"before":"d265c9555c4cce6affc148fb8f1446c9a31e49c9","after":"1b538b580912c6e4e1c8e8ba72e82aece232bd9f","ref":"refs/heads/main","pushedAt":"2023-04-06T21:56:27.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"bluescreen10","name":"Mariano Wahlmann","path":"/bluescreen10","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/86094?s=80&v=4"},"commit":{"message":"๐ added a cli to use the tokenizer via the command Lines\n\nthis would be helpful for other people to test and use the tokenizer.\nEven from other languages because this can be distributed as a single\nbinary.","shortMessageHtmlLink":"๐ added a cli to use the tokenizer via the command Lines"}},{"before":"b511b31fa943b0d53fdbfaf137f2ab68690a430b","after":"d265c9555c4cce6affc148fb8f1446c9a31e49c9","ref":"refs/heads/main","pushedAt":"2023-04-06T21:55:29.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"bluescreen10","name":"Mariano Wahlmann","path":"/bluescreen10","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/86094?s=80&v=4"},"commit":{"message":"๐ added a cli to use the tokenizer via the command Lines\n\nthis would be helpful for other people to test and use the tokenizer.\nEven from other languages because this can be distributed as a single\nbinary.","shortMessageHtmlLink":"๐ added a cli to use the tokenizer via the command Lines"}},{"before":"79aa80d63f09a0d74b360df077d204f7755bc714","after":"b511b31fa943b0d53fdbfaf137f2ab68690a430b","ref":"refs/heads/main","pushedAt":"2023-04-06T21:52:37.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"bluescreen10","name":"Mariano Wahlmann","path":"/bluescreen10","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/86094?s=80&v=4"},"commit":{"message":"๐ added a cli to use the tokenizer via the command Lines\n\nthis would be helpful for other people to test and use the tokenizer.\nEven from other languages because this can be distributed as a single\nbinary.","shortMessageHtmlLink":"๐ added a cli to use the tokenizer via the command Lines"}},{"before":"ea2b0eea6b85ee333cdf235e71442c8196c742ad","after":"79aa80d63f09a0d74b360df077d204f7755bc714","ref":"refs/heads/main","pushedAt":"2023-04-06T20:59:32.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"bluescreen10","name":"Mariano Wahlmann","path":"/bluescreen10","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/86094?s=80&v=4"},"commit":{"message":"๐ fixed an issue in the internal generate command\n\nThe internal generate command was using log.Fatal instead of the\ncorrect log.Fatalf.","shortMessageHtmlLink":"๐ fixed an issue in the internal generate command"}},{"before":"2e69c6a4db9547039ca536c281a73c350a20b57b","after":"ea2b0eea6b85ee333cdf235e71442c8196c742ad","ref":"refs/heads/main","pushedAt":"2023-04-06T20:11:54.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"bluescreen10","name":"Mariano Wahlmann","path":"/bluescreen10","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/86094?s=80&v=4"},"commit":{"message":"๐คซ temporarily skipping failing test until I confirm the expected output\n\nRight now emits the same output as OpenAI's tiktoken but I'm not sure\nthat's the right output.","shortMessageHtmlLink":"๐คซ temporarily skipping failing test until I confirm the expected output"}},{"before":"a4eb7ed16c9dbe62ab99a614157759f64fd99dac","after":"2e69c6a4db9547039ca536c281a73c350a20b57b","ref":"refs/heads/main","pushedAt":"2023-04-06T20:02:57.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"bluescreen10","name":"Mariano Wahlmann","path":"/bluescreen10","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/86094?s=80&v=4"},"commit":{"message":"๐ adding test results badge to README.md\n\nThis will make it more obvious the status of the build when reading the landing page","shortMessageHtmlLink":"๐ adding test results badge to README.md"}},{"before":"72848e8b530ee3bcb5f464232e39d905e06109a4","after":"a4eb7ed16c9dbe62ab99a614157759f64fd99dac","ref":"refs/heads/main","pushedAt":"2023-04-06T19:58:33.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"bluescreen10","name":"Mariano Wahlmann","path":"/bluescreen10","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/86094?s=80&v=4"},"commit":{"message":"๐ add github actions\n\nThis should help verify future releases. The test are not passing as of now because I'm not sure tiktoken's (original python library) output is correct.","shortMessageHtmlLink":"๐ add github actions"}},{"before":null,"after":"72848e8b530ee3bcb5f464232e39d905e06109a4","ref":"refs/heads/main","pushedAt":"2023-04-06T19:54:55.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"bluescreen10","name":"Mariano Wahlmann","path":"/bluescreen10","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/86094?s=80&v=4"},"commit":{"message":"๐ข Initial Release\n\nThis implementations should work for most of OpenAI's model. However\nWe are not yet dealing with the special tokens and/or gpt2 encoding.","shortMessageHtmlLink":"๐ข Initial Release"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAERuLKqwA","startCursor":null,"endCursor":null}},"title":"Activity ยท tiktoken-go/tokenizer"}