Every repository with this icon (
Every repository with this icon (
| Description: | Johnson wraps JavaScript in a loving Ruby embrace. edit |
-
Unicode javascript strings are not transferred to ruby correctly. Here's an example. I create a javascript string consisting of a single Euro-Sign (see http://www.fileformat.info/info/unicode/char/20ac/index.htm)
irb(main):001:0> require 'rubygems' => true irb(main):002:0> require 'johnson' => true irb(main):007:0> s = Johnson.evaluate("'\\u20AC'") => "\254"In ruby, we're getting a single byte with the value 254 (octal), which is 172 decimal, or 0xAC. So it looks like we're only getting the low-byte of our 16-bit Unicode char. After scanning the Johnson code, I think I found the culprit - JS_GetStringBytes returns the bytes of a Unicode-16 String by stripping off the high-bytes.
Note that for non-ASCII strings, if JS_CStringsAreUTF8 is false, these functions can return a corrupted copy of the contents of >the string. Use JS_GetStringChars to access the 16-bit characters of a JavaScript string without conversions or copying.
A similar problem probably exists in the other direction (ruby -> js) too.
I suggest trying JS_CStringsAreUTF8 (which may solve both problems). If this fails, then johnson would have to extract the Unicode-16 chars from spidermonkey and convert them to a ruby-friendly encoding.
Comments
-
3 comments Created 3 months ago by tarcieriLarge arithmetic expressions overflowjbarnettex>> Johnson::Parser.parse "1000 * 60 * 60 * 24 * 30" => [[:lit, -2147483648]]Comments
Argh, good catch. This constant folding may be happening in SpiderMonkey's parser. If it's in the translation bits on our side, what do you think would be the least surprising behavior? Bignum?
Yes, Bignum would be fine. We're just round-tripping everything while doing some AST munging, so as long as what goes in comes out I don't really care what the intermediate representation is.
-
Are there any dependencies that I am forgetting? Here is the error,
ERROR: Error installing johnson:
ERROR: Failed to build gem native extension./opt/ruby-enterprise-1.8.7-20090928/bin/ruby extconf.rb make: Fatal error in reader: config.mk, line 41: Unexpected end of line seen
extconf.rb failed Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers. Check the mkmf.log file for more
details. You may need configuration options.Provided configuration options:
--with-opt-dir --without-opt-dir --with-opt-include --without-opt-include=${opt-dir}/include --with-opt-lib --without-opt-lib=${opt-dir}/lib --with-make-prog --without-make-prog --srcdir=. --curdir --ruby=/opt/ruby-enterprise-1.8.7-20090928/bin/rubyextconf.rb:25:in `<<': can't convert nil into String (TypeError)
from extconf.rb:25Comments
As far as I know, you're the first person to even try compiling Johnson on Solaris, especially with REE. :) The devs are all on Mac OS X or Linux. Line 25, the one with the error, is manipulating an extconf global variable that should certainly exist.
We currently bundle (and statically link) SpiderMonkey with Johnson's native extension, so it's probably not a missing dependency.
I'd try playing with extconf.rb: you can re-run it from the ext/spidermonkey directory (ruby extconf.rb) to regenerate a Makefile.
If I can successfully install spidermonkey, can I remove the vendor folder of the gem and use the install software on the system? What would I need to change in the gem?
Thanks,
Sam -
Hello, Please help me:
$ sudo gem install johnson Building native extensions. This could take a while...
ERROR: Error installing johnson:
ERROR: Failed to build gem native extension./System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby extconf.rb cat: ../../dist/Darwin_DBG.OBJ/nspr/Version: No such file or directory
cd editline; make -f Makefile.ref all
cc -o Darwin_DBG.OBJ/editline.o -c -Wall -Wno-format -MMD -DXP_UNIX -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DJSFILE -DANSI_ARROWS -DHAVE_TCGETATTR -DHIDE -DUSE_DIRENT -DSYS_UNIX -DHAVE_STDLIB -DUNIQUE_HISTORY editline.c
editline.c: In function ‘readline’:
editline.c:1038: warning: implicit declaration of function ‘getpid’
editline.c: In function ‘add_history’:
editline.c:1051: warning: pointer targets in passing argument 2 of ‘strcmp’ differ in signedness
cc -o Darwin_DBG.OBJ/sysunix.o -c -Wall -Wno-format -MMD -DXP_UNIX -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DJSFILE -DANSI_ARROWS -DHAVE_TCGETATTR -DHIDE -DUSE_DIRENT -DSYS_UNIX -DHAVE_STDLIB -DUNIQUE_HISTORY sysunix.c
ar rv Darwin_DBG.OBJ/libedit.a Darwin_DBG.OBJ/editline.o Darwin_DBG.OBJ/sysunix.o
r - Darwin_DBG.OBJ/editline.o
r - Darwin_DBG.OBJ/sysunix.o
ranlib Darwin_DBG.OBJ/libedit.a
make -f Makefile.ref Darwin_DBG.OBJ/libjs.a Darwin_DBG.OBJ/libjs.dylib Darwin_DBG.OBJ/js Darwin_DBG.OBJ/jsautocfg.h Darwin_DBG.OBJ/jscpucfg Darwin_DBG.OBJ/jscpucfg.o
cat: ../../dist/Darwin_DBG.OBJ/nspr/Version: No such file or directory
make[1]: Circular jscpucfg.h <- Darwin_DBG.OBJ/jsautocfg.h dependency dropped.
cc -o Darwin_DBG.OBJ/jscpucfg.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jscpucfg.c
cc -o Darwin_DBG.OBJ/jscpucfg Darwin_DBG.OBJ/jscpucfg.o
make[1]: Circular Darwin_DBG.OBJ/jsautocfg.h <- Darwin_DBG.OBJ/jsautocfg.h dependency dropped.
rm -f Darwin_DBG.OBJ/jsautocfg.h
Darwin_DBG.OBJ/jscpucfg > Darwin_DBG.OBJ/jsautocfg.h
cc -o Darwin_DBG.OBJ/jsapi.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsapi.c
jsapi.c: In function ‘JS_ConvertArgumentsVA’:
jsapi.c:272: warning: passing argument 5 of ‘TryArgumentFormatter’ from incompatible pointer type
jsapi.c: In function ‘JS_PushArgumentsVA’:
jsapi.c:380: warning: passing argument 5 of ‘TryArgumentFormatter’ from incompatible pointer type
cc -o Darwin_DBG.OBJ/jsarena.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsarena.c
cc -o Darwin_DBG.OBJ/jsarray.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsarray.c
cc -o Darwin_DBG.OBJ/jsatom.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsatom.c
cc -o Darwin_DBG.OBJ/jsbool.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsbool.c
cc -o Darwin_DBG.OBJ/jscntxt.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jscntxt.c
cc -o Darwin_DBG.OBJ/jsdate.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsdate.c
cc -o Darwin_DBG.OBJ/jsdbgapi.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsdbgapi.c
cc -o Darwin_DBG.OBJ/jsdhash.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsdhash.c
cc -o Darwin_DBG.OBJ/jsdtoa.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsdtoa.c
cc -o Darwin_DBG.OBJ/jsemit.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsemit.c
cc -o Darwin_DBG.OBJ/jsexn.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsexn.c
cc -o Darwin_DBG.OBJ/jsfun.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsfun.c
cc -o Darwin_DBG.OBJ/jsgc.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsgc.c
cc -o Darwin_DBG.OBJ/jshash.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jshash.c
cc -o Darwin_DBG.OBJ/jsinterp.o -c -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsinterp.c
cc -o Darwin_DBG.OBJ/jsinvoke.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsinvoke.c
cc -o Darwin_DBG.OBJ/jsiter.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsiter.c
cc -o Darwin_DBG.OBJ/jslock.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jslock.c
cc -o Darwin_DBG.OBJ/jslog2.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jslog2.c
cc -o Darwin_DBG.OBJ/jslong.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jslong.c
cc -o Darwin_DBG.OBJ/jsmath.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsmath.c
cc -o Darwin_DBG.OBJ/jsnum.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsnum.c
cc -o Darwin_DBG.OBJ/jsobj.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsobj.c
cc -o Darwin_DBG.OBJ/jsopcode.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsopcode.c
cc -o Darwin_DBG.OBJ/jsparse.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsparse.c
cc -o Darwin_DBG.OBJ/jsprf.o -c -Wall -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DDARWIN -DX86_LINUX -DDEBUG -DDEBUG_root -DEDITLINE -IDarwin_DBG.OBJ jsprf.c
jsprf.c: In function ‘BuildArgArray’:
jsprf.c:644: error: incompatible types in assignment
make[1]: [Darwin_DBG.OBJ/jsprf.o] Error 1
make: [all] Error 2
extconf.rb failed Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers. Check the mkmf.log file for more
details. You may need configuration options.Provided configuration options:
--with-opt-dir --without-opt-dir --with-opt-include --without-opt-include=${opt-dir}/include --with-opt-lib --without-opt-lib=${opt-dir}/lib --with-make-prog --without-make-prog --srcdir=. --curdir --ruby=/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/rubyextconf.rb:26:in `<<': can't convert nil into String (TypeError)
from extconf.rb:26Gem files will remain installed in /Library/Ruby/Gems/1.8/gems/johnson-1.1.2 for inspection.
Results logged to /Library/Ruby/Gems/1.8/gems/johnson-1.1.2/ext/spidermonkey/gem_make.out
Comments
You could try my fork: http://github.com/smparkes/johnson. I build on OS X, both 32 bit and 64 bit.












Note: this issue has also been posted at https://johnson.lighthouseapp.com/projects/8759-johnson/tickets/52
2a316a5:
Not quite sure what's going wrong here, but:
This, of course, should be "A"
Hrmm. That's the UTF-8 encoding \u4100, rather than \u0041... but the same line for me returns "A".
My previous commit also means you now can't pass non-UTF-8 ruby strings into JS, because of the conversion step -- which isn't brilliant.
Perhaps the best move would be to put it back how it was before, and recommend use of UTF-8 in the JS environment? I'll look at how easily we can add -DJS_C_STRINGS_ARE_UTF8 to our SM build process.
Well, that just moves the encoding step into SpiderMonkey... which leaves us with the same problem.
It's starting to feel like as long as ruby isn't tracking the encoding of a string (read: 1.8), we can't really interoperate with JS using the high byte in its characters... or at least, not with correct round-tripping.
The only other option that comes to mind at the moment would be to stop doing the native conversion, and instead proxy strings in each direction until we /really/ have to convert. That would also give JS the ability to modify Ruby strings, which it currently lacks.
I wonder why the string conversion works differently for me. Some difference in the environment?
in ruby_land_proxy.rb, I changed "UTF-16" to "UCS-2-INTERNAL":
This seems to fix the problem - now all tests pass on my machine. According to the libiconv docs, UCS-2-INTERNAL means "Full Unicode ...with machine dependent endianness and alignment", so I guess this makes sense.
I just tried this on debian/lenny: UCS-2-INTERNAL isn't available here...sigh.
Using UCS-2LE (little-endian) works on both my machines (Mac/Intel und Debian/Intel), but probably won't on big-endian plattforms. Looks like we'll need a compile-time #ifdef here.
Or maybe we should let spidermonkey do the UCS-2 to UTF-8 translation for us. That way, no endien-dependent strings will leave spidermonkey.
Okay, I've just pushed a change to use JS_SetCStringsAreUTF8() and friends to let SpiderMonkey do the conversion. As you say, that should cover any endianness issues.
It does still leave the issue that non-utf-8 strings (e.g., binary data) cannot be passed from Ruby to JS.
I personally don't think this is an issue. Trying to store binary data in a javascript string is generally a bad idea. A javascript string is simply not an array of bytes. At the latest, when manipulating (concatenation or substr), spidermonkey will garble non-textual data.
The javascript standard even defines strings as 16-bit textual data. This is also spidermonkey's internal representation.
The closest you can get to an array-of-bytes in javascript is an array-of-numbers. This, of course, would not let you manipulate raw pixel data or anything....