Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`split': invalid byte sequence in US-ASCII (ArgumentError) on ripper.rb #596

Closed
github0013 opened this issue Jul 19, 2020 · 11 comments · Fixed by #617
Closed

`split': invalid byte sequence in US-ASCII (ArgumentError) on ripper.rb #596

github0013 opened this issue Jul 19, 2020 · 11 comments · Fixed by #617

Comments

@github0013
Copy link

Metadata

node -v
  v14.5.0

ruby -v
  ruby 2.7.1p83 (2020-03ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-darwin19]

uname -a
  Darwin mbp 19.6.0 Darwin Kernel Version 19.6.0: Sun Jul  5 00:43:10 PDT 2020; root:xnu-6153.141.1~9/RELEASE_X86_64 x86_64

cat package.json
  {
    "devDependencies": {
      "@prettier/plugin-ruby": "^0.19.0",
      "prettier": "^2.0.5"
    }
  }

Visual Studio Code
  Version: 1.47.0
  Commit: d5e9aa0227e057a60c82568bf31c04730dc15dcd
  Date: 2020-07-09T08:01:54.115Z (1 wk ago)
  Electron: 7.3.2
  Chrome: 78.0.3904.130
  Node.js: 12.8.1
  V8: 7.8.279.23-electron.0
  OS: Darwin x64 19.6.0

https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode
  v5.1.3

Input

# あ 

      d=[30644250780,9003106878,
  30636278846,66641217692,4501790980,
671_24_603036,131_61973916,66_606629_920,
 30642677916,30643069058];a,s=[],$*[0]
    s.each_byte{|b|a<<("%036b"%d[b.
       chr.to_i]).scan(/\d{6}/)}
        a.transpose.each{ |a|
          a.join.each_byte{\
           |i|print i==49?\
             ($*[1]||"#")\
               :32.chr}
                 puts
                  }
class RipperJS < Ripper
  attr_reader :source, :lines, :__end__

  def initialize(source, *args)
    super(source, *args)

    @source = source # <= source.encoding
    @lines = source.split("\n")
    @__end__ = nil
  end

It seems if ran through VSCode, source.encoding would be recognized as "US-ASCII." And I get an error like below.

ripper.rb:25:in `split': invalid byte sequence in US-ASCII (ArgumentError)

When I run bundle exec rbprettier --write on the same file, source.encoding is "UTF-8", and the format is done correctly.

github0013 added a commit to github0013/plugin-ruby that referenced this issue Jul 19, 2020
@kddnewton
Copy link
Member

For the life of me I cannot get this thing to replicate. How are you running it through VSCode? Are you clicking "Format Document"? Or is it something else?

@github0013
Copy link
Author

Screen Shot 2020-07-22 at 10 14 16

Because I didn't point out clearly (my bad), you might have missed the Japanese char in the sample code.

# あ  <= HERE!!

      d=[30644250780,9003106878,
  30636278846,66641217692,4501790980,
671_24_603036,131_61973916,66_606629_920,
 30642677916,30643069058];a,s=[],$*[0]
    s.each_byte{|b|a<<("%036b"%d[b.
       chr.to_i]).scan(/\d{6}/)}
        a.transpose.each{ |a|
          a.join.each_byte{\
           |i|print i==49?\
             ($*[1]||"#")\
               :32.chr}
                 puts
                  }

I recently reinstalled OS and everything, and I just started having this issue.

Screen_Shot_2020-07-22_at_10_30_32

Screen_Shot_2020-07-22_at_10_31_12

@stmichael
Copy link

stmichael commented Aug 14, 2020

I'm having the same issue but in a different setup. I don't use any editor integration, just plain command line. My setup:

OS: Debian
Language: JS (NodeJS v12.18.3)
Plugin-Ruby: 0.19.0

Running yarn prettier -l '**/*.rb' in my project prints a whole bunch of errors identical to the ones @github0013 mentioned.

Error: /app/node_modules/@prettier/plugin-ruby/src/ripper.rb:25:in `split': invalid byte sequence in US-ASCII (ArgumentError)

I can't count them all, there's too many.

To track the issue I came up with a minimal example. Create a file test.rb with content

'ä'

That's a valid Ruby program although it doesn't make any sense. Running yarn prettier -l test.rb will produce the very same issue.

I'm not an expert on character encodings. In my understanding since Ruby 2.0 the default encoding is UTF-8. Then why does this plugin read the files in US-ASCII?

@kddnewton
Copy link
Member

I really have no idea to be honest, and it's not replicating on my machine at all. I've got the exact same version of stuff as y'all do. Can one of you run ruby -e 'pp RbConfig::CONFIG' and we'll see if there's some kind of weird configuration different between our rubies?

Also please run ruby -rripper -e "pp Ripper.sexp('ä')" and see what the returns. Also maybe put that into a file and run ripper over it.

@csalvato
Copy link

csalvato commented Aug 15, 2020

I was facing this issue and had a ¢ character in a comment in my code.

I removed the ¢ character and prettier started working again.

H/T to @stmichael for pointing out the special character, which made me look for special characters like ¢ in my code.

@github0013
Copy link
Author

╰─➤  ruby -v
ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-darwin19]
╰─➤  ruby -e 'pp RbConfig::CONFIG'
{"DESTDIR"=>"",
 "MAJOR"=>"2",
 "MINOR"=>"7",
 "TEENY"=>"1",
 "PATCHLEVEL"=>"83",
 "INSTALL"=>"/usr/bin/install -c",
 "EXEEXT"=>"",
 "prefix"=>"/Users/ac/.rbenv/versions/2.7.1",
 "ruby_install_name"=>"ruby",
 "RUBY_INSTALL_NAME"=>"ruby",
 "RUBY_SO_NAME"=>"ruby.2.7",
 "exec"=>"exec",
 "ruby_pc"=>"ruby-2.7.pc",
 "CC_WRAPPER"=>"",
 "PACKAGE"=>"ruby",
 "BUILTIN_TRANSSRCS"=>" enc/trans/newline.c",
 "MANTYPE"=>"doc",
 "vendorarchhdrdir"=>
  "/Users/ac/.rbenv/versions/2.7.1/include/ruby-2.7.0/vendor_ruby/x86_64-darwin19",
 "sitearchhdrdir"=>
  "/Users/ac/.rbenv/versions/2.7.1/include/ruby-2.7.0/site_ruby/x86_64-darwin19",
 "rubyarchhdrdir"=>
  "/Users/ac/.rbenv/versions/2.7.1/include/ruby-2.7.0/x86_64-darwin19",
 "vendorhdrdir"=>
  "/Users/ac/.rbenv/versions/2.7.1/include/ruby-2.7.0/vendor_ruby",
 "sitehdrdir"=>"/Users/ac/.rbenv/versions/2.7.1/include/ruby-2.7.0/site_ruby",
 "rubyhdrdir"=>"/Users/ac/.rbenv/versions/2.7.1/include/ruby-2.7.0",
 "RUBY_SEARCH_PATH"=>"",
 "UNIVERSAL_INTS"=>"",
 "UNIVERSAL_ARCHNAMES"=>"",
 "configure_args"=>
  " '--prefix=/Users/ac/.rbenv/versions/2.7.1' '--with-openssl-dir=/Users/ac/.rbenv/versions/2.7.1/openssl' '--enable-shared' '--with-readline-dir=/usr/local/opt/readline' 'CC=clang' 'LDFLAGS=-L/Users/ac/.rbenv/versions/2.7.1/lib ' 'CPPFLAGS=-I/Users/ac/.rbenv/versions/2.7.1/include '",
 "CONFIGURE"=>"configure",
 "vendorarchdir"=>
  "/Users/ac/.rbenv/versions/2.7.1/lib/ruby/vendor_ruby/2.7.0/x86_64-darwin19",
 "vendorlibdir"=>"/Users/ac/.rbenv/versions/2.7.1/lib/ruby/vendor_ruby/2.7.0",
 "vendordir"=>"/Users/ac/.rbenv/versions/2.7.1/lib/ruby/vendor_ruby",
 "sitearchdir"=>
  "/Users/ac/.rbenv/versions/2.7.1/lib/ruby/site_ruby/2.7.0/x86_64-darwin19",
 "sitelibdir"=>"/Users/ac/.rbenv/versions/2.7.1/lib/ruby/site_ruby/2.7.0",
 "sitedir"=>"/Users/ac/.rbenv/versions/2.7.1/lib/ruby/site_ruby",
 "rubyarchdir"=>
  "/Users/ac/.rbenv/versions/2.7.1/lib/ruby/2.7.0/x86_64-darwin19",
 "rubylibdir"=>"/Users/ac/.rbenv/versions/2.7.1/lib/ruby/2.7.0",
 "ruby_version"=>"2.7.0",
 "sitearch"=>"x86_64-darwin19",
 "arch"=>"x86_64-darwin19",
 "sitearchincludedir"=>
  "/Users/ac/.rbenv/versions/2.7.1/include/x86_64-darwin19",
 "archincludedir"=>"/Users/ac/.rbenv/versions/2.7.1/include/x86_64-darwin19",
 "sitearchlibdir"=>"/Users/ac/.rbenv/versions/2.7.1/lib/x86_64-darwin19",
 "archlibdir"=>"/Users/ac/.rbenv/versions/2.7.1/lib/x86_64-darwin19",
 "libdirname"=>"libdir",
 "RUBY_EXEC_PREFIX"=>"/Users/ac/.rbenv/versions/2.7.1",
 "RUBY_LIB_VERSION"=>"",
 "RUBY_LIB_VERSION_STYLE"=>"3\t/* full */",
 "RI_BASE_NAME"=>"ri",
 "ridir"=>"/Users/ac/.rbenv/versions/2.7.1/share/ri",
 "rubysitearchprefix"=>
  "/Users/ac/.rbenv/versions/2.7.1/lib/ruby/x86_64-darwin19",
 "rubyarchprefix"=>"/Users/ac/.rbenv/versions/2.7.1/lib/ruby/x86_64-darwin19",
 "MAKEFILES"=>"Makefile GNUmakefile",
 "PLATFORM_DIR"=>"",
 "THREAD_MODEL"=>"pthread",
 "SYMBOL_PREFIX"=>"_",
 "EXPORT_PREFIX"=>"",
 "COMMON_HEADERS"=>"",
 "COMMON_MACROS"=>"",
 "COMMON_LIBS"=>"",
 "MAINLIBS"=>"-lpthread -ldl -lobjc",
 "ENABLE_SHARED"=>"yes",
 "DLDSHARED"=>"clang -dynamiclib",
 "DLDLIBS"=>"",
 "SOLIBS"=>"-lpthread -ldl -lobjc",
 "LIBRUBYARG_SHARED"=>"-lruby.2.7",
 "LIBRUBYARG_STATIC"=>
  "-lruby.2.7-static -framework Security -framework Foundation -lpthread -ldl -lobjc",
 "LIBRUBYARG"=>"-lruby.2.7",
 "LIBRUBY"=>"libruby.2.7.dylib",
 "LIBRUBY_ALIASES"=>"libruby.dylib",
 "LIBRUBY_SONAME"=>"libruby.2.7.dylib",
 "LIBRUBY_SO"=>"libruby.2.7.dylib",
 "LIBRUBY_A"=>"libruby.2.7-static.a",
 "RUBYW_INSTALL_NAME"=>"",
 "rubyw_install_name"=>"",
 "EXTDLDFLAGS"=>"",
 "EXTLDFLAGS"=>"",
 "strict_warnflags"=>"-std=gnu99",
 "warnflags"=>
  "-Wall -Wextra -Wdeprecated-declarations -Wdivision-by-zero -Wimplicit-function-declaration -Wimplicit-int -Wpointer-arith -Wshorten-64-to-32 -Wwrite-strings -Wmissing-noreturn -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wunused-variable -Wextra-tokens",
 "debugflags"=>"-ggdb3",
 "optflags"=>"-O3",
 "NULLCMD"=>":",
 "ENABLE_DEBUG_ENV"=>"",
 "DLNOBJ"=>"dln.o",
 "INSTALL_STATIC_LIBRARY"=>"no",
 "MJIT_SUPPORT"=>"yes",
 "EXECUTABLE_EXTS"=>"",
 "ARCHFILE"=>"",
 "LIBRUBY_RELATIVE"=>"no",
 "EXTOUT"=>".ext",
 "PREP"=>"miniruby exe/$(PROGRAM)",
 "CROSS_COMPILING"=>"no",
 "TEST_RUNNABLE"=>"yes",
 "rubylibprefix"=>"/Users/ac/.rbenv/versions/2.7.1/lib/ruby",
 "setup"=>"Setup",
 "ENCSTATIC"=>"",
 "EXTSTATIC"=>"",
 "STRIP"=>"strip -A -n",
 "SOEXT"=>"dylib",
 "TRY_LINK"=>"",
 "PRELOADENV"=>"DYLD_INSERT_LIBRARIES",
 "LIBPATHENV"=>"DYLD_FALLBACK_LIBRARY_PATH",
 "RPATHFLAG"=>"",
 "LIBPATHFLAG"=>" -L%s",
 "LINK_SO"=>
  "\n" +
  "dsymutil $@; { test -z '$(RUBY_CODESIGN)' || codesign -s '$(RUBY_CODESIGN)' -f $@; }",
 "ASMEXT"=>"S",
 "LIBEXT"=>"a",
 "DLEXT2"=>"",
 "DLEXT"=>"bundle",
 "LDSHAREDXX"=>"clang++ -dynamic -bundle",
 "LDSHARED"=>"clang -dynamic -bundle",
 "CCDLFLAGS"=>"-fno-common",
 "STATIC"=>"",
 "ARCH_FLAG"=>"",
 "DLDFLAGS"=>
  "-L/Users/ac/.rbenv/versions/2.7.1/lib  -Wl,-undefined,dynamic_lookup -Wl,-multiply_defined,suppress",
 "ALLOCA"=>"",
 "MATHN"=>"yes",
 "dsymutil"=>"dsymutil",
 "codesign"=>"codesign",
 "POSTLINK"=>
  "dsymutil $@; { test -z '$(RUBY_CODESIGN)' || codesign -s '$(RUBY_CODESIGN)' -f $@; }",
 "WERRORFLAG"=>"-Werror",
 "CHDIR"=>"cd -P",
 "RMALL"=>"rm -fr",
 "RMDIRS"=>"rmdir -p",
 "RMDIR"=>"rmdir",
 "CP"=>"cp",
 "RM"=>"rm -f",
 "PKG_CONFIG"=>"pkg-config",
 "DOXYGEN"=>"",
 "DOT"=>"",
 "MAKEDIRS"=>"mkdir -p",
 "MKDIR_P"=>"mkdir -p",
 "INSTALL_DATA"=>"/usr/bin/install -c -m 644",
 "INSTALL_SCRIPT"=>"/usr/bin/install -c",
 "INSTALL_PROGRAM"=>"/usr/bin/install -c",
 "SET_MAKE"=>"",
 "LN_S"=>"ln -s",
 "NM"=>"nm",
 "DLLWRAP"=>"",
 "WINDRES"=>"",
 "OBJCOPY"=>":",
 "OBJDUMP"=>"objdump",
 "ASFLAGS"=>"",
 "AS"=>"as",
 "ARFLAGS"=>"-no_warning_for_no_symbols -o ",
 "AR"=>"libtool -static",
 "RANLIB"=>":",
 "try_header"=>"",
 "CC_VERSION_MESSAGE"=>
  "Apple clang version 11.0.3 (clang-1103.0.32.62)\n" +
  "Target: x86_64-apple-darwin19.5.0\n" +
  "Thread model: posix\n" +
  "InstalledDir: /Library/Developer/CommandLineTools/usr/bin",
 "CC_VERSION"=>"clang --version",
 "MJIT_CC"=>"/usr/bin/clang",
 "CSRCFLAG"=>"",
 "COUTFLAG"=>"-o ",
 "OUTFLAG"=>"-o ",
 "CPPOUTFILE"=>"-o conftest.i",
 "GNU_LD"=>"no",
 "LD"=>"ld",
 "GCC"=>"yes",
 "EGREP"=>"/usr/bin/grep -E",
 "GREP"=>"/usr/bin/grep",
 "CPP"=>"clang -E",
 "CXXFLAGS"=>"-g -O2",
 "CXX"=>"clang++",
 "OBJEXT"=>"o",
 "CPPFLAGS"=>
  "-I/Users/ac/.rbenv/versions/2.7.1/include  -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT  ",
 "LDFLAGS"=>
  "-L. -L/Users/ac/.rbenv/versions/2.7.1/lib  -fstack-protector-strong -L/usr/local/lib",
 "CFLAGS"=>
  "-O3 -ggdb3 -Wall -Wextra -Wdeprecated-declarations -Wdivision-by-zero -Wimplicit-function-declaration -Wimplicit-int -Wpointer-arith -Wshorten-64-to-32 -Wwrite-strings -Wmissing-noreturn -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wunused-variable -Wextra-tokens  -fno-common -pipe",
 "CC"=>"clang",
 "target_os"=>"darwin19",
 "target_vendor"=>"apple",
 "target_cpu"=>"x86_64",
 "target"=>"x86_64-apple-darwin19",
 "host_os"=>"darwin19.5.0",
 "host_vendor"=>"apple",
 "host_cpu"=>"x86_64",
 "host"=>"x86_64-apple-darwin19.5.0",
 "RUBY_VERSION_NAME"=>"ruby-2.7.0",
 "RUBYW_BASE_NAME"=>"rubyw",
 "RUBY_BASE_NAME"=>"ruby",
 "build_os"=>"darwin19.5.0",
 "build_vendor"=>"apple",
 "build_cpu"=>"x86_64",
 "build"=>"x86_64-apple-darwin19.5.0",
 "RUBY_PROGRAM_VERSION"=>"2.7.1",
 "RUBY_API_VERSION"=>"2.7",
 "HAVE_GIT"=>"yes",
 "GIT"=>"git",
 "cxxflags"=>"",
 "cppflags"=>"",
 "cflags"=>
  "-O3 -ggdb3 -Wall -Wextra -Wdeprecated-declarations -Wdivision-by-zero -Wimplicit-function-declaration -Wimplicit-int -Wpointer-arith -Wshorten-64-to-32 -Wwrite-strings -Wmissing-noreturn -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wunused-variable -Wextra-tokens",
 "target_alias"=>"",
 "host_alias"=>"",
 "build_alias"=>"",
 "LIBS"=>"",
 "ECHO_T"=>"",
 "ECHO_N"=>"",
 "ECHO_C"=>"\\\\c",
 "DEFS"=>"",
 "mandir"=>"/Users/ac/.rbenv/versions/2.7.1/share/man",
 "localedir"=>"/Users/ac/.rbenv/versions/2.7.1/share/locale",
 "libdir"=>"/Users/ac/.rbenv/versions/2.7.1/lib",
 "psdir"=>"/Users/ac/.rbenv/versions/2.7.1/share/doc/ruby",
 "pdfdir"=>"/Users/ac/.rbenv/versions/2.7.1/share/doc/ruby",
 "dvidir"=>"/Users/ac/.rbenv/versions/2.7.1/share/doc/ruby",
 "htmldir"=>"/Users/ac/.rbenv/versions/2.7.1/share/doc/ruby",
 "infodir"=>"/Users/ac/.rbenv/versions/2.7.1/share/info",
 "docdir"=>"/Users/ac/.rbenv/versions/2.7.1/share/doc/ruby",
 "oldincludedir"=>"/usr/include",
 "includedir"=>"/Users/ac/.rbenv/versions/2.7.1/include",
 "runstatedir"=>"/Users/ac/.rbenv/versions/2.7.1/var/run",
 "localstatedir"=>"/Users/ac/.rbenv/versions/2.7.1/var",
 "sharedstatedir"=>"/Users/ac/.rbenv/versions/2.7.1/com",
 "sysconfdir"=>"/Users/ac/.rbenv/versions/2.7.1/etc",
 "datadir"=>"/Users/ac/.rbenv/versions/2.7.1/share",
 "datarootdir"=>"/Users/ac/.rbenv/versions/2.7.1/share",
 "libexecdir"=>"/Users/ac/.rbenv/versions/2.7.1/libexec",
 "sbindir"=>"/Users/ac/.rbenv/versions/2.7.1/sbin",
 "bindir"=>"/Users/ac/.rbenv/versions/2.7.1/bin",
 "exec_prefix"=>"/Users/ac/.rbenv/versions/2.7.1",
 "PACKAGE_URL"=>"",
 "PACKAGE_BUGREPORT"=>"",
 "PACKAGE_STRING"=>"",
 "PACKAGE_VERSION"=>"",
 "PACKAGE_TARNAME"=>"",
 "PACKAGE_NAME"=>"",
 "PATH_SEPARATOR"=>":",
 "SHELL"=>"/bin/sh",
 "UNICODE_VERSION"=>"12.1.0",
 "UNICODE_EMOJI_VERSION"=>"12.1",
 "SDKROOT"=>"",
 "archdir"=>"/Users/ac/.rbenv/versions/2.7.1/lib/ruby/2.7.0/x86_64-darwin19",
 "topdir"=>"/Users/ac/.rbenv/versions/2.7.1/lib/ruby/2.7.0/x86_64-darwin19"}
╰─➤  ruby -rripper -e "pp Ripper.sexp('ä')"
[:program, [[:vcall, [:@ident, "ä", [1, 0]]]]]
╰─➤  ruby -rripper -e "pp Ripper.sexp('あ')"
[:program, [[:vcall, [:@ident, "あ", [1, 0]]]]]

@stmichael
Copy link

➜  ruby -e 'pp RbConfig::CONFIG'
{"DESTDIR"=>"",
 "MAJOR"=>"2",
 "MINOR"=>"6",
 "TEENY"=>"3",
 "PATCHLEVEL"=>"62",
 "INSTALL"=>"/usr/bin/install -c",
 "EXEEXT"=>"",
 "prefix"=>"/usr/local",
 "ruby_install_name"=>"ruby",
 "RUBY_INSTALL_NAME"=>"ruby",
 "RUBY_SO_NAME"=>"ruby",
 "exec"=>"exec",
 "ruby_pc"=>"ruby-2.6.pc",
 "CC_WRAPPER"=>"",
 "PACKAGE"=>"ruby",
 "BUILTIN_TRANSSRCS"=>" enc/trans/newline.c",
 "MANTYPE"=>"man",
 "vendorarchhdrdir"=>"/usr/local/include/ruby-2.6.0/vendor_ruby/x86_64-linux",
 "sitearchhdrdir"=>"/usr/local/include/ruby-2.6.0/site_ruby/x86_64-linux",
 "rubyarchhdrdir"=>"/usr/local/include/ruby-2.6.0/x86_64-linux",
 "vendorhdrdir"=>"/usr/local/include/ruby-2.6.0/vendor_ruby",
 "sitehdrdir"=>"/usr/local/include/ruby-2.6.0/site_ruby",
 "rubyhdrdir"=>"/usr/local/include/ruby-2.6.0",
 "RUBY_SEARCH_PATH"=>"",
 "UNIVERSAL_INTS"=>"",
 "UNIVERSAL_ARCHNAMES"=>"",
 "configure_args"=>
  " '--build=x86_64-linux-gnu' '--disable-install-doc' '--enable-shared' 'build_alias=x86_64-linux-gnu'",
 "CONFIGURE"=>"configure",
 "vendorarchdir"=>"/usr/local/lib/ruby/vendor_ruby/2.6.0/x86_64-linux",
 "vendorlibdir"=>"/usr/local/lib/ruby/vendor_ruby/2.6.0",
 "vendordir"=>"/usr/local/lib/ruby/vendor_ruby",
 "sitearchdir"=>"/usr/local/lib/ruby/site_ruby/2.6.0/x86_64-linux",
 "sitelibdir"=>"/usr/local/lib/ruby/site_ruby/2.6.0",
 "sitedir"=>"/usr/local/lib/ruby/site_ruby",
 "rubyarchdir"=>"/usr/local/lib/ruby/2.6.0/x86_64-linux",
 "rubylibdir"=>"/usr/local/lib/ruby/2.6.0",
 "ruby_version"=>"2.6.0",
 "sitearch"=>"x86_64-linux",
 "arch"=>"x86_64-linux",
 "sitearchincludedir"=>"/usr/local/include/x86_64-linux",
 "archincludedir"=>"/usr/local/include/x86_64-linux",
 "sitearchlibdir"=>"/usr/local/lib/x86_64-linux",
 "archlibdir"=>"/usr/local/lib/x86_64-linux",
 "libdirname"=>"libdir",
 "RUBY_EXEC_PREFIX"=>"/usr/local",
 "RUBY_LIB_VERSION"=>"",
 "RUBY_LIB_VERSION_STYLE"=>"3\t/* full */",
 "RI_BASE_NAME"=>"ri",
 "ridir"=>"/usr/local/share/ri",
 "rubysitearchprefix"=>"/usr/local/lib/ruby/x86_64-linux",
 "rubyarchprefix"=>"/usr/local/lib/ruby/x86_64-linux",
 "MAKEFILES"=>"Makefile GNUmakefile",
 "PLATFORM_DIR"=>"",
 "THREAD_MODEL"=>"pthread",
 "SYMBOL_PREFIX"=>"",
 "EXPORT_PREFIX"=>"",
 "COMMON_HEADERS"=>"",
 "COMMON_MACROS"=>"",
 "COMMON_LIBS"=>"",
 "MAINLIBS"=>"-lz -lpthread -lrt -lrt -lgmp -ldl -lcrypt -lm ",
 "ENABLE_SHARED"=>"yes",
 "DLDSHARED"=>"gcc -shared",
 "DLDLIBS"=>" -lc",
 "SOLIBS"=>"-lz -lpthread -lrt -lrt -lgmp -ldl -lcrypt -lm ",
 "LIBRUBYARG_SHARED"=>"-Wl,-rpath,/usr/local/lib -L/usr/local/lib -lruby",
 "LIBRUBYARG_STATIC"=>
  "-Wl,-rpath,/usr/local/lib -L/usr/local/lib -lruby-static -lz -lpthread -lrt -lrt -lgmp -ldl -lcrypt -lm ",
 "LIBRUBYARG"=>"-Wl,-rpath,/usr/local/lib -L/usr/local/lib -lruby",
 "LIBRUBY"=>"libruby.so.2.6.3",
 "LIBRUBY_ALIASES"=>"libruby.so.2.6 libruby.so",
 "LIBRUBY_SONAME"=>"libruby.so.2.6",
 "LIBRUBY_SO"=>"libruby.so.2.6.3",
 "LIBRUBY_A"=>"libruby-static.a",
 "RUBYW_INSTALL_NAME"=>"",
 "rubyw_install_name"=>"",
 "EXTDLDFLAGS"=>"",
 "EXTLDFLAGS"=>"",
 "strict_warnflags"=>"-std=gnu99",
 "warnflags"=>
  "-Wall -Wextra -Wdeclaration-after-statement -Wdeprecated-declarations -Wduplicated-cond -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wrestrict -Wwrite-strings -Wimplicit-fallthrough=0 -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-packed-bitfield-compat -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wsuggest-attribute=format -Wsuggest-attribute=noreturn -Wunused-variable",
 "debugflags"=>"-ggdb3",
 "optflags"=>"-O3",
 "NULLCMD"=>":",
 "ENABLE_DEBUG_ENV"=>"",
 "DLNOBJ"=>"dln.o",
 "INSTALL_STATIC_LIBRARY"=>"no",
 "MJIT_SUPPORT"=>"yes",
 "EXECUTABLE_EXTS"=>"",
 "ARCHFILE"=>"",
 "LIBRUBY_RELATIVE"=>"no",
 "EXTOUT"=>".ext",
 "PREP"=>"miniruby",
 "CROSS_COMPILING"=>"no",
 "TEST_RUNNABLE"=>"yes",
 "rubylibprefix"=>"/usr/local/lib/ruby",
 "setup"=>"Setup",
 "ENCSTATIC"=>"",
 "EXTSTATIC"=>"",
 "STRIP"=>"strip -S -x",
 "SOEXT"=>"so",
 "TRY_LINK"=>"",
 "PRELOADENV"=>"LD_PRELOAD",
 "LIBPATHENV"=>"LD_LIBRARY_PATH",
 "RPATHFLAG"=>" -Wl,-rpath,%1$-s",
 "LIBPATHFLAG"=>" -L%1$-s",
 "LINK_SO"=>"",
 "ASMEXT"=>"S",
 "LIBEXT"=>"a",
 "DLEXT2"=>"",
 "DLEXT"=>"so",
 "LDSHAREDXX"=>"g++ -shared",
 "LDSHARED"=>"gcc -shared",
 "CCDLFLAGS"=>"-fPIC",
 "STATIC"=>"",
 "ARCH_FLAG"=>"",
 "DLDFLAGS"=>"-Wl,--compress-debug-sections=zlib",
 "ALLOCA"=>"",
 "MATHN"=>"yes",
 "dsymutil"=>"",
 "codesign"=>"",
 "POSTLINK"=>":",
 "WERRORFLAG"=>"-Werror",
 "CHDIR"=>"cd -P",
 "RMALL"=>"rm -fr",
 "RMDIRS"=>"rmdir --ignore-fail-on-non-empty -p",
 "RMDIR"=>"rmdir --ignore-fail-on-non-empty",
 "CP"=>"cp",
 "RM"=>"rm -f",
 "PKG_CONFIG"=>"pkg-config",
 "DOXYGEN"=>"",
 "DOT"=>"",
 "MAKEDIRS"=>"/bin/mkdir -p",
 "MKDIR_P"=>"/bin/mkdir -p",
 "INSTALL_DATA"=>"/usr/bin/install -c -m 644",
 "INSTALL_SCRIPT"=>"/usr/bin/install -c",
 "INSTALL_PROGRAM"=>"/usr/bin/install -c",
 "SET_MAKE"=>"",
 "LN_S"=>"ln -s",
 "NM"=>"nm",
 "DLLWRAP"=>"",
 "WINDRES"=>"",
 "OBJCOPY"=>":",
 "OBJDUMP"=>"objdump",
 "ASFLAGS"=>"",
 "AS"=>"as",
 "ARFLAGS"=>"rcD ",
 "AR"=>"ar",
 "RANLIB"=>"ranlib",
 "try_header"=>"",
 "CC_VERSION_MESSAGE"=>
  "gcc (Debian 8.3.0-6) 8.3.0\n" +
  "Copyright (C) 2018 Free Software Foundation, Inc.\n" +
  "This is free software; see the source for copying conditions.  There is NO\n" +
  "warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.",
 "CC_VERSION"=>"gcc --version",
 "CSRCFLAG"=>"",
 "COUTFLAG"=>"-o ",
 "OUTFLAG"=>"-o ",
 "CPPOUTFILE"=>"-o conftest.i",
 "GNU_LD"=>"yes",
 "LD"=>"ld",
 "GCC"=>"yes",
 "EGREP"=>"/bin/grep -E",
 "GREP"=>"/bin/grep",
 "CPP"=>"gcc -E",
 "CXXFLAGS"=>
  "-O3 -ggdb3 -Wall -Wextra -Wdeclaration-after-statement -Wdeprecated-declarations -Wduplicated-cond -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wrestrict -Wwrite-strings -Wimplicit-fallthrough=0 -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-packed-bitfield-compat -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wsuggest-attribute=format -Wsuggest-attribute=noreturn -Wunused-variable",
 "CXX"=>"g++",
 "OBJEXT"=>"o",
 "CPPFLAGS"=>"  ",
 "LDFLAGS"=>"-L. -fstack-protector-strong -rdynamic -Wl,-export-dynamic",
 "CFLAGS"=>
  "-O3 -ggdb3 -Wall -Wextra -Wdeclaration-after-statement -Wdeprecated-declarations -Wduplicated-cond -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wrestrict -Wwrite-strings -Wimplicit-fallthrough=0 -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-packed-bitfield-compat -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wsuggest-attribute=format -Wsuggest-attribute=noreturn -Wunused-variable  -fPIC",
 "CC"=>"gcc",
 "target_os"=>"linux",
 "target_vendor"=>"pc",
 "target_cpu"=>"x86_64",
 "target"=>"x86_64-pc-linux-gnu",
 "host_os"=>"linux-gnu",
 "host_vendor"=>"pc",
 "host_cpu"=>"x86_64",
 "host"=>"x86_64-pc-linux-gnu",
 "RUBY_VERSION_NAME"=>"ruby-2.6.0",
 "RUBYW_BASE_NAME"=>"rubyw",
 "RUBY_BASE_NAME"=>"ruby",
 "build_os"=>"linux-gnu",
 "build_vendor"=>"pc",
 "build_cpu"=>"x86_64",
 "build"=>"x86_64-pc-linux-gnu",
 "RUBY_API_VERSION"=>"2.6",
 "RUBY_PROGRAM_VERSION"=>"2.6.3",
 "HAVE_GIT"=>"yes",
 "GIT"=>"git",
 "cxxflags"=>
  "-O3 -ggdb3 -Wall -Wextra -Wdeclaration-after-statement -Wdeprecated-declarations -Wduplicated-cond -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wrestrict -Wwrite-strings -Wimplicit-fallthrough=0 -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-packed-bitfield-compat -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wsuggest-attribute=format -Wsuggest-attribute=noreturn -Wunused-variable",
 "cppflags"=>"",
 "cflags"=>
  "-O3 -ggdb3 -Wall -Wextra -Wdeclaration-after-statement -Wdeprecated-declarations -Wduplicated-cond -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wrestrict -Wwrite-strings -Wimplicit-fallthrough=0 -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-packed-bitfield-compat -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wsuggest-attribute=format -Wsuggest-attribute=noreturn -Wunused-variable",
 "target_alias"=>"",
 "host_alias"=>"",
 "build_alias"=>"x86_64-linux-gnu",
 "LIBS"=>"-lm ",
 "ECHO_T"=>"",
 "ECHO_N"=>"-n",
 "ECHO_C"=>"",
 "DEFS"=>"",
 "mandir"=>"/usr/local/share/man",
 "localedir"=>"/usr/local/share/locale",
 "libdir"=>"/usr/local/lib",
 "psdir"=>"/usr/local/share/doc/ruby",
 "pdfdir"=>"/usr/local/share/doc/ruby",
 "dvidir"=>"/usr/local/share/doc/ruby",
 "htmldir"=>"/usr/local/share/doc/ruby",
 "infodir"=>"/usr/local/share/info",
 "docdir"=>"/usr/local/share/doc/ruby",
 "oldincludedir"=>"/usr/include",
 "includedir"=>"/usr/local/include",
 "runstatedir"=>"/usr/local/var/run",
 "localstatedir"=>"/usr/local/var",
 "sharedstatedir"=>"/usr/local/com",
 "sysconfdir"=>"/usr/local/etc",
 "datadir"=>"/usr/local/share",
 "datarootdir"=>"/usr/local/share",
 "libexecdir"=>"/usr/local/libexec",
 "sbindir"=>"/usr/local/sbin",
 "bindir"=>"/usr/local/bin",
 "exec_prefix"=>"/usr/local",
 "PACKAGE_URL"=>"",
 "PACKAGE_BUGREPORT"=>"",
 "PACKAGE_STRING"=>"",
 "PACKAGE_VERSION"=>"",
 "PACKAGE_TARNAME"=>"",
 "PACKAGE_NAME"=>"",
 "PATH_SEPARATOR"=>":",
 "SHELL"=>"/bin/bash",
 "UNICODE_VERSION"=>"12.1.0",
 "UNICODE_EMOJI_VERSION"=>"12.0",
 "archdir"=>"/usr/local/lib/ruby/2.6.0/x86_64-linux",
 "topdir"=>"/usr/local/lib/ruby/2.6.0/x86_64-linux"}

➜  ruby -rripper -e "pp Ripper.sexp('ä')"
-e:1: invalid multibyte char (US-ASCII)
-e:1: invalid multibyte char (US-ASCII)

I'm running Prettier as well as these commands in a docker container. The base container is ruby:2.6.3.

@stmichael
Copy link

I found out a few more things and possibly a solution. I'll describe how I got there and thus hopefully help @github0013 in finding a solution for his setup.

I printed the encoding of my test string in the terminal with ruby -e "pp 'ä'.encoding" and always got US-ASCII. When I put the same code pp 'ä'.encoding in a file and then ran it with ruby my_file.rb I got UTF-8. So the encoding differs when running something in the shell compared to running the same code from a file. Then I put pp Ripper.sexp('ä') in the same Ruby file, ran it and Ripper parsed the expression and produced the correct output. This tells me that the problem might be in my terminal.

Running locale showed me that the encoding in my docker container was POSIX. I also noticed that I couldn't actually type ä in the terminal. So the issue lies actually in my systems config. Once I installed an UTF8 locale I was able to run ruby -rripper -e "pp Ripper.sexp('ä')" from the terminal and also Prettier with the Ruby plugin.

@github0013
Copy link
Author

@stmichael

This is how I fixed this problem currently.
#597

@kddnewton
Copy link
Member

@stmichael that's the ticket! Thank you for finding that. Because of that investigation, I was finally able to track it down to:

LC_ALL="en_US.US-ASCII" ruby -e 'p "ä"'

That will reliably fail. I will add something that will detect the default encoding and handle it appropriately.

@kddnewton
Copy link
Member

Figured it out. Going out in the next release. #617

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants