Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unexpected characters changed to encoding when tried with the tool #420

Closed
mundey opened this issue Aug 10, 2021 · 1 comment
Closed

unexpected characters changed to encoding when tried with the tool #420

mundey opened this issue Aug 10, 2021 · 1 comment

Comments

@mundey
Copy link

mundey commented Aug 10, 2021

scenario: tried to translate parts of text in source (html/go/js). encoding for '=' didn't change back in output and converted to u003d

command used:

trans -brief zh-CN:en file://example.txt -o example-out.txt

input example file content:

          <el-form-item label="字典名称" prop="dictType">

actual output file:

<el-form-item labelu003d"Dictionary name" propu003d"dictType">

expected / output using web browser:

<el-form-item label="Dictionary name" prop="dictType">

i tried with -no-ansi but i guess it's not right option
here is output with -D

  16 bytes > HTTP/1.1 200 OK
  46 bytes > Content-Type: application/json; charset=utf-8
  32 bytes > X-Content-Type-Options: nosniff
  31 bytes > Access-Control-Allow-Origin: *
  62 bytes > Cache-Control: no-cache, no-store, max-age=0, must-revalidate
  17 bytes > Pragma: no-cache
  39 bytes > Expires: Mon, 01 Jan 1990 00:00:00 GMT
  36 bytes > Date: Tue, 10 Aug 2021 10:33:01 GMT
  80 bytes > Content-Disposition: attachment; filename="json.txt"; filename*=UTF-8''json.txt
 101 bytes > Content-Security-Policy: require-trusted-types-for 'script';report-uri /_/TranslateApiHttp/cspreport
 192 bytes > Content-Security-Policy: script-src 'report-sample' 'nonce-5TXzZFq8qxe3qGstrORw6g' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/TranslateApiHttp/cspreport;worker-src 'self'
  43 bytes > Cross-Origin-Resource-Policy: cross-origin
  40 bytes > Cross-Origin-Opener-Policy: same-origin
  12 bytes > Server: ESF
  20 bytes > X-XSS-Protection: 0
  28 bytes > X-Frame-Options: SAMEORIGIN
  20 bytes > Accept-Ranges: none
  22 bytes > Vary: Accept-Encoding
  18 bytes > Connection: close
  27 bytes > Transfer-Encoding: chunked
   1 bytes >
   4 bytes > 352
 823 bytes > [[["\u003cel-form-item label\u003d\"Dictionary name\" prop\u003d\"dictType\"\u003e","\u003cel-form-item label\u003d\"字典名称\" prop\u003d\"dictType\"\u003e",null,null,3,null,null,[[]],[[["51171d70a5b6c2485a24361038890141","zh_en_2020q2.md"]]]],[null,null,null,"\u003cEl-form-item label\u003d\"zìdiǎn míngchēng\" prop\u003d\"dictType\"\u003e"]],null,"zh-CN",null,null,[["\u003cel-form-item label\u003d\"字典名称\" prop\u003d\"dictType\"\u003e",null,[["\u003cel-form-item label\u003d\"Dictionary name\" prop\u003d\"dictType\"\u003e",0,true,false,[3]],["\u003cEl-form-item label \u003d \"dictionary name\" prop \u003d \"dictType\"\u003e",0,true,false,[0]]],[[0,43]],"\u003cel-form-item label\u003d\"字典名称\" prop\u003d\"dictType\"\u003e",0,0]],1,[],[["zh-CN"],null,[1],["zh-CN"]],null,null,null,null,null,null,null,null,null,[null,2]]
   2 bytes > 0
   1 bytes >
content = '''
352
[[["\u003cel-form-item label\u003d\"Dictionary name\" prop\u003d\"dictType\"\u003e","\u003cel-form-item label\u003d\"字典名称\" prop\u003d\"dictType\"\u003e",null,null,3,null,null,[[]],[[["51171d70a5b6c2485a24361038890141","zh_en_2020q2.md"]]]],[null,null,null,"\u003cEl-form-item label\u003d\"zìdiǎn míngchēng\" prop\u003d\"dictType\"\u003e"]],null,"zh-CN",null,null,[["\u003cel-form-item label\u003d\"字典名称\" prop\u003d\"dictType\"\u003e",null,[["\u003cel-form-item label\u003d\"Dictionary name\" prop\u003d\"dictType\"\u003e",0,true,false,[3]],["\u003cEl-form-item label \u003d \"dictionary name\" prop \u003d \"dictType\"\u003e",0,true,false,[0]]],[[0,43]],"\u003cel-form-item label\u003d\"字典名称\" prop\u003d\"dictType\"\u003e",0,0]],1,[],[["zh-CN"],null,[1],["zh-CN"]],null,null,null,null,null,null,null,null,null,[null,2]]
0

'''
tokens = ["352", "[", "[", "[", "\"\\u003cel-form-item label\\u003d\\\"Dictionary name\\\" prop\\u003d\\\"dictType\\\"\\u003e\"", ",", "\"\\u003cel-form-item label\\u003d\\\"字典名称\\\" prop\\u003d\\\"dictType\\\"\\u003e\"", ",", "null", ",", "null", ",", "3", ",", "null", ",", "null", ",", "[", "[", "]", "]", ",", "[", "[", "[", "\"51171d70a5b6c2485a24361038890141\"", ",", "\"zh_en_2020q2.md\"", "]", "]", "]", "]", ",", "[", "null", ",", "null", ",", "null", ",", "\"\\u003cEl-form-item label\\u003d\\\"zìdiǎn míngchēng\\\" prop\\u003d\\\"dictType\\\"\\u003e\"", "]", "]", ",", "null", ",", "\"zh-CN\"", ",", "null", ",", "null", ",", "[", "[", "\"\\u003cel-form-item label\\u003d\\\"字典名称\\\" prop\\u003d\\\"dictType\\\"\\u003e\"", ",", "null", ",", "[", "[", "\"\\u003cel-form-item label\\u003d\\\"Dictionary name\\\" prop\\u003d\\\"dictType\\\"\\u003e\"", ",", "0", ",", "true", ",", "false", ",", "[", "3", "]", "]", ",", "[", "\"\\u003cEl-form-item label \\u003d \\\"dictionary name\\\" prop \\u003d \\\"dictType\\\"\\u003e\"", ",", "0", ",", "true", ",", "false", ",", "[", "0", "]", "]", "]", ",", "[", "[", "0", ",", "43", "]", "]", ",", "\"\\u003cel-form-item label\\u003d\\\"字典名称\\\" prop\\u003d\\\"dictType\\\"\\u003e\"", ",", "0", ",", "0", "]", "]", ",", "1", ",", "[", "]", ",", "[", "[", "\"zh-CN\"", "]", ",", "null", ",", "[", "1", "]", ",", "[", "\"zh-CN\"", "]", "]", ",", "null", ",", "null", ",", "null", ",", "null", ",", "null", ",", "null", ",", "null", ",", "null", ",", "null", ",", "[", "null", ",", "2", "]", "]", "0"]
ast = {
"0,0,0,0"	"\"\\u003cel-form-item label\\u003d\\\"Dictionary name\\\" prop\\u003d\\\"dictType\\\"\\u003e\""
"0,0,0,1"	"\"\\u003cel-form-item label\\u003d\\\"字典名称\\\" prop\\u003d\\\"dictType\\\"\\u003e\""
"0,0,0,2"	"null"
"0,0,0,3"	"null"
"0,0,0,4"	"3"
"0,0,0,5"	"null"
"0,0,0,6"	"null"
"0,0,0,8,0,0,0"	"\"51171d70a5b6c2485a24361038890141\""
"0,0,0,8,0,0,1"	"\"zh_en_2020q2.md\""
"0,0,1,0"	"null"
"0,0,1,1"	"null"
"0,0,1,2"	"null"
"0,0,1,3"	"\"\\u003cEl-form-item label\\u003d\\\"zìdiǎn míngchēng\\\" prop\\u003d\\\"dictType\\\"\\u003e\""
"0,1"	"null"
"0,2"	"\"zh-CN\""
"0,3"	"null"
"0,4"	"null"
"0,5,0,0"	"\"\\u003cel-form-item label\\u003d\\\"字典名称\\\" prop\\u003d\\\"dictType\\\"\\u003e\""
"0,5,0,1"	"null"
"0,5,0,2,0,0"	"\"\\u003cel-form-item label\\u003d\\\"Dictionary name\\\" prop\\u003d\\\"dictType\\\"\\u003e\""
"0,5,0,2,0,1"	"0"
"0,5,0,2,0,2"	"true"
"0,5,0,2,0,3"	"false"
"0,5,0,2,0,4,0"	"3"
"0,5,0,2,1,0"	"\"\\u003cEl-form-item label \\u003d \\\"dictionary name\\\" prop \\u003d \\\"dictType\\\"\\u003e\""
"0,5,0,2,1,1"	"0"
"0,5,0,2,1,2"	"true"
"0,5,0,2,1,3"	"false"
"0,5,0,2,1,4,0"	"0"
"0,5,0,3,0,0"	"0"
"0,5,0,3,0,1"	"43"
"0,5,0,4"	"\"\\u003cel-form-item label\\u003d\\\"字典名称\\\" prop\\u003d\\\"dictType\\\"\\u003e\""
"0,5,0,5"	"0"
"0,5,0,6"	"0"
"0,6"	"1"
"0,8,0,0"	"\"zh-CN\""
"0,8,1"	"null"
"0,8,2,0"	"1"
"0,8,3,0"	"\"zh-CN\""
"0,9"	"null"
"0,10"	"null"
"0,11"	"null"
"0,12"	"null"
"0,13"	"null"
"0,14"	"null"
"0,15"	"null"
"0,16"	"null"
"0,17"	"null"
"0,18,0"	"null"
"0,18,1"	"2"
"0"	"0"
}
❯ trans -V
Translate Shell       0.9.6.12-release

platform              Linux
terminal type         xterm-256color
bi-di emulator        [N/A]
gawk (GNU Awk)        5.1.0
fribidi (GNU FriBidi) 1.0.10
audio player          mpg123
terminal pager        less
web browser           xdg-open
user locale           en_CA.UTF-8 (English)
home language         en
source language       auto
target language       en
translation engine    google
proxy                 [NONE]
user-agent            Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36
ip version            [DEFAULT]
theme                 default
init file             [NONE]

Report bugs to:       https://github.com/soimort/translate-shell/issues

Thank you for peeking into the issue.

@mundey mundey changed the title unexpected characters changed when tried with the tool unexpected characters changed to encoding when tried with the tool Aug 10, 2021
@soimort soimort closed this as completed in d3e0836 Sep 1, 2021
@soimort
Copy link
Owner

soimort commented Sep 1, 2021

Thanks for reporting. Fixed now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants