New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

关于语料问题 #26

Closed

ZenXir opened this issue Apr 1, 2023 · 3 comments

ZenXir commented Apr 1, 2023

从网盘新下载了 merge.json 语料
发现原来是 663M 现在变成389M了
是什么原因语料变小只保留了70W+条大佬？

Owner

Facico commented Apr 1, 2023

@ZenXir 改成utf-8格式了

Facico closed this as completed

Owner

Facico commented Apr 1, 2023

ascii格式表示长度要长

Author

ZenXir commented Apr 1, 2023

确实刚看了下我之前转成utf8格式的确实也是389M 哈哈

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment