Limited encodings available to Ruby

[tsurezuregusa]

Perhaps related to #1549, running rb puts Encoding.list in dfhack gives only very few encodings, making it very annoying to translate from IBM437.

On Mac 10.15.5 (installed ruby 2.6.3p62, but I guess dfhack runs its own).

UTF-8
US-ASCII
UTF-16BE
UTF-16LE
UTF-32BE
UTF-32LE
UTF-16
UTF-32
UTF8-MAC
EUC-JP
Windows-31J

[lethosor]

The 64-bit Mac build should use your system Ruby, actually (not one you've installed through a package manager, though). Can you run devel/lsmem and paste its output to see what Ruby is loaded?

[tsurezuregusa]

Thank you.

I do not find any external ruby in this.

lsmem.txt

enc.txt

[lethosor]

That's weird, it should definitely list something else with "ruby" in its name, besides hack/plugins/ruby.plug.dylib. Did you run devel/lsmem before all of the plugins had loaded? Could you try running it after running some Ruby code in DFHack?

Here's the list of files the Ruby plugin tries to load on macOS:

dfhack/plugins/ruby/ruby.cpp

Lines 330 to 331 in 289e0df

	"hack/libruby.dylib",
	"/System/Library/Frameworks/Ruby.framework/Ruby",

You could try to see if either of these exist, and what Ruby version they correspond to if so.

[tsurezuregusa]

/System/Library/Frameworks/Ruby.framework/Ruby exists. But no hack/libruby.dylib.

After running ruby code, I tried running unmodified dfhack, also in /Applications, and same result.

How do I run devel/lsmem before plugins loaded? Running reload ruby causes error: dfhack: line 27: 36512 Bus error: 10 DYLD_INSERT_LIBRARIES=./hack/libdfhack.dylib ./dwarfort.exe "$@"

/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby --version

ruby 2.6.3p62 (2019-04-16 revision 67580) [universal.x86_64-darwin19]

[lethosor]

Sorry, I was asking you to run devel/lsmem after plugins had loaded. And by "run Ruby code", I mean run it in DFHack. For example, run the lever command in DFHack, or run :rb puts 1 and see if it prints 1. Then run devel/lsmem after that and see if anything else Ruby-related shows up (specifically anything under /System).

The "bus error" is a crash, which is definitely not supposed to happen, and isn't good. Does it also occur if you try to run Ruby code in DFHack (last paragraph)?

hack/libruby.dylib looks like it's only distributed for 32-bit macOS builds, so the fact that it's missing for you is expected (and fine). I'm wondering if macOS 10.15 is blocking access to the Ruby framework for some reason, though. Can you try cp /System/Library/Frameworks/Ruby.framework/Ruby hack/libruby.dylib and see if that changes anything?

[tsurezuregusa]

I did run after plugins loaded I think, in dfhack prompt. I guess the way to run before is in init file. I do not know how it would be possible to not run after plugins load.

Copying the system library to hack/ does show the library in devel/lsmem. reload ruby does not crash, but it does not change output of rb puts Encoding.list.

[lethosor]

Yeah, I'm also not sure what I meant about running something before plugins load. I know it's possible to interact with the DF window before all plugins have loaded, since it takes some time, but I don't think you can interact with the console that early. Disregard.

Out of curiosity, what does :rb puts RUBY_VERSION print?

It seems that we still use Ruby 1.8.7 on Linux (and 32-bit Windows, I think), which doesn't even have the built-in Encoding class:

[DFHack]# :rb puts Encoding.list
E: NameError: (eval): uninitialized constant Encoding
 (eval)

We should probably standardize on a single version of Ruby across platforms (update: see #1718). Evidently we can't rely on the system Ruby on macOS behaving properly either. I'm not sure how complicated it'll be to build librubys that are portable enough for our needs.

I assume "IBM437" is what DF+DFHack refer to as CP437, yes? We do have existing functions to convert between that and UTF-8, but they're currently only available to C++ and Lua. It shouldn't be hard to expose them to Ruby, although I'm far from an expert on how Ruby encodings work.

[tsurezuregusa]

I see. Thank you.

Is there a difference between at the prompt rb and :rb?

rb puts RUBY_VERSION > 2.6.3
This is the system ruby version. I thought there was another installed in /usr/local.

I have been using encoding conversion in Lua, but Lua is frustrating. If I write conversion in Ruby I will share.

[lethosor]

The : treats everything after the command as a single argument (including whitespace verbatim): https://docs.dfhack.org/en/stable/docs/Core.html#using-dfhack-commands
The lua command needs this for expressions that have spaces in it (although running lua with no arguments starts an interpreter, which is easier to use), but it looks like rb usually joins multiple arguments without needing the :.

The Ruby plugin will only look for the two libraries I linked earlier on macOS (the one in hack and the one in /System). If you change the plugin to look somewhere else and recompile, it could pick up one from somewhere else, though.

If you're interested in adding encoding conversion, these are the relevant functions:

dfhack/library/include/MiscUtils.h

Lines 399 to 401 in 289e0df

	DFHACK_EXPORT std::string UTF2DF(const std::string &in);
	DFHACK_EXPORT std::string DF2UTF(const std::string &in);
	DFHACK_EXPORT std::string DF2CONSOLE(const std::string &in);

And here's an example of a C function in the ruby plugin that takes a string:

dfhack/plugins/ruby/ruby.cpp

Lines 566 to 573 in 289e0df

	static VALUE rb_dfprint_str(VALUE self, VALUE s)
	{
	if (r_console)
	r_console->print("%s", rb_string_value_ptr(&s));
	else
	console_proxy->print("%s", rb_string_value_ptr(&s));
	return Qnil;
	}

and one that returns a string:

dfhack/plugins/ruby/ruby.cpp

Lines 856 to 860 in 289e0df

	static VALUE rb_dfmemory_read_stlstring(VALUE self, VALUE addr)
	{
	std::string *s = (std::string*)rb_num2ulong(addr);
	return rb_str_new(s->c_str(), s->length());
	}