-
Notifications
You must be signed in to change notification settings - Fork 20
Home
Welcome to the Ruby + OMR Technology Preview! The Eclipse OMR team put this preview together to show how the Eclipse OMR technology could be integrated into Ruby.
We are releasing this technology preview for a couple of reasons:
- So that people can try out the Eclipse OMR technology in Ruby
- To get feedback on how Eclipse OMR was integrated into Ruby
This preview is not intended to become “another” Ruby runtime with its own community. Our sincerest hope is that this preview facilitates us working with and within the existing Ruby community to integrate those parts of OMR that the Ruby community finds beneficial.
The Eclipse OMR project is an open source community growing around reusable and easily consumable core components (like thread library, platform abstraction library, garbage collector, etc.) for building all kinds of language runtimes, from Java to Ruby to Smalltalk and beyond. Many of the components were contributed by IBM from the IBM J9 Java Virtual Machine (JVM), an enterprise class JVM implementation representing hundreds of person years of development creating scalable, high performance runtime technology. But going forward, this open project is accepting contributions from anyone.
The OMR components have 3 features that distinguish the Eclipse OMR project from other projects that aim to reuse technology for building language runtimes: 1) OMR has no language semantics, 2) OMR is not a language runtime, and 3) OMR components can be consumed by any language runtime. The first two features distinguish OMR from efforts to implement languages on top of a mature language runtime like the Java Virtual Machine (by implementing in Java) or the CoreCLR project (by implementing via CLR bytecodes). Both these other efforts require mapping a language's semantics into the semantics of another language (either Java or CLR). In contrast, OMR components can be brought into an existing language runtime with its own bytecode set and bytecode semantics (or abstract syntax tree semantics). OMR can therefore be used to build any kind of language runtime, free from the influence of Java or another language's semantics.
One of the most important motivators for the Eclipse OMR project is the increasing importance of cloud computing platforms, where the polyglot of programming languages must all cooperate seamlessly in what can really be a diverse environment. Language runtimes must be monitored and managed adaptively to adjust to the current operating conditions in the cloud platform. But not all runtimes have the same kinds of responsiveness built in because most language runtimes today actually share very little technology. can hurt or even sabotage longer term success. But every language runtime community needs to go through this process, and as existing runtimes become more and more sophisticated, the path to success for new languages becomes even more difficult.
By introducing shareable core technology components for building all kinds of language runtimes, the Eclipse OMR project hopes to both make the process easier for new languages to bootstrap themselves as well as to provide mechanisms for existing runtimes to fill gaps or to reduce maintenance costs so that communities can focus more of their efforts on the opportunities and problems that are specific to their language.
To learn more about the Eclipse OMR project, please visit us at the Eclipse OMR Github page.
To make sure the Eclipse OMR technology really could work in runtimes other than Java, the OMR team has been working to create several proof points to use the technology with different language runtimes (like Ruby!). The Ruby + Eclipse OMR Technology Preview technology represents the current state of our Ruby proof point, with new GC, JIT compiler, and method profiling capabilities. It's not a toy: we've got it running Rails applications. We cannot claim it is ready for use in production, but we think it's good enough to be able to meet the goals outlined at the beginning of this document. You can try it out on Linux on x86, Linux on Z (IBM mainframes), or Linux on OpenPOWER. We look forward to hearing what you think about it!
We presented talks about our Ruby + OMR proof point at Ruby Kaigi 2015:
-
Matthew Gaudet at Ruby Kaigi: Experiments in sharing Java VM technology with CRuby
-
Robert Young and Craig Lehmann at Ruby Kaigi: It's dangerous to GC alone. Take this!
Our longer term goal is for the community to help us structure the changes we've made to the Ruby runtime to maximize the benefit and so that they are acceptable for contribution to Ruby. We will not maintain the Ruby + OMR Technology Preview as a Ruby fork, although we will make updates to it over time as more OMR source code becomes available, or to address / accommodate feedback we get from the Ruby community. As we update the technology, we'll also update this documentation and bump the release number r<N> in the ruby --version output.
The Ruby + OMR Technology preview is a medium to facilitate communicating with the Ruby community.
The Ruby + OMR Technology Preview release can be built via:
$ git clone https://github.com/rubyomr-preview/ruby.git --branch ruby_2_2_omr --recursive
$ cd ruby
$ autoconf
$ ./configure SPEC=<specname>
$ make
$ make install
Since the Ruby + OMR code has only been tested on Linux x86-64, Linux PPC-LE-64, Linux PPC-BE-64
and Linux 390-64 the acceptable values for <specname> are:
1. linux_x86-64
2. linux_ppc-64_le_gcc
3. linux_ppc-64
4. linux_390-64
Building with this simple sequence of commands will download the latest version of the Ruby + OMR Technology Preview and Eclipse OMR, build them together and install them onto your system. That's it!
There are more configuration options than the simple set listed above, which is just a high level description. Search for "OMR" and hopefully you'll find the tags we used self explanatory. If not, feel free to ask questions!
Unfortunately, not all of the Eclipse OMR project is available in the open (we're working on it really hard). The place to look for the most up to date source code is at The Ruby + Eclipse OMR Ruby repo. In the meantime you can test out these features by using the Docker images.
To make it even easier for people to try out the Ruby + OMR Technology Preview without having to clone repositories or build, we also have three docker images available that come with the Ruby + OMR Technology Preview already built and pre-installed. Docker images make it easy to avoid fussing with platform specifics. The images have a preinstalled Ruby 2.2.x with built-in OMR technology. Also included is a monitoring agent which can be used with IBM Health Centre to visualize Ruby method profiles and garbage collection performance while your Ruby application is running.
The Eclipse OMR technology itself is included only as part of the prebuilt binaries which you can access by simply running ruby (/usr/local/bin/ruby).
The directions are very similar for different platforms, but we have provided directions for each platform separately below. Please see the appropriate section below if you're working on Linux on X86, Linux on Z, or Linux on OpenPOWER. After following the directions, you can read the following sections to learn how to activate the various OMR capabilities within the Ruby + OMR Technology Preview. The rubyomrpreview Docker image comes with the sqlite3, pg, and rails GEMs installed. The Linux X86 Docker image has been updated to the latest version and we will be updating the Linux on Z and Linux on OpenPOWER soon.
-
If you do not already have docker installed, follow these directions to get started:
-
Download the rubyomrpreview docker image from Box.com with the command:
$ wget https://ibm.box.com/shared/static/sy7pgu7pqbyht9j3g3tvko77ska1k8x1.tgz -O rubyomrpreview-x86_64.tgz -
Load the docker image locally:
$ docker load -i rubyomrpreview-x86_64.tgz -
Now that you have successfully downloaded the image you are ready to run it:
$ docker run -p 1883:1883 -it rubyomrpreview/rubyomrpreview /bin/bash(The port-publishing option "
-p 1883:1883" is needed to enable monitoring Ruby applications using Health Center. You can omit this option if you don't want to use Health Centre or if you are not running the broker inside the container). -
You are now running in an Ubuntu 15.10 image. To verify that the correct rubyomrpreview version of Ruby is on the path, use the command:
$ ruby --version ruby 2.2.5p285 (Eclipse OMR Preview r1) (2016-03-29) [x86_64-linux] -
You're ready to go!
-
If you do not already have docker installed, follow these directions to get started: Installing docker - Linux on Z
-
Download the rubyomrpreview docker image from Box.com with the command:
$ wget https://ibm.box.com/shared/static/1bzikt7fmfdejpvp4zqglnss64tcwls2.tgz -O rubyomrpreview-s390x.tgz -
Load the docker image locally:
$ docker load -i rubyomrpreview-s390x.tgz -
Now that you have successfully downloaded the image you are ready to run it:
$ docker run -p 1883:1883 -it rubyomrpreview/rubyomrpreview /bin/bash(The port-publishing option "
-p 1883:1883" is needed to enable monitoring Ruby applications using Health Center. You can omit this option if you don't want to use Health Centre or if you are not running the broker inside the container). -
You are now running inside a ClefOS 7.1 image. To verify that the correct rubyomrpreview version of Ruby is on the path, use the command:
$ ruby --version ruby 2.2.3p97 (OMR Preview r1) (2015-04-14) [s390x-linux] -
You're ready to go!
-
If you do not already have docker installed, follow these directions to get started:
-
Download the rubyomrpreview docker image from Box.com with the command:
$ wget https://ibm.box.com/shared/static/hdqgfvvnnq1idf6qvlsv9r0rw5dzwbhc.tgz -O rubyomrpreview-powerpc64le.tgz -
Load the docker image locally:
$ docker load -i rubyomrpreview-powerpc64le.tgz -
Now that you have successfully downloaded the image you are ready to run it:
$ docker run -p 1883:1883 -it rubyomrpreview/rubyomrpreview /bin/bash(The port-publishing option "
-p 1883:1883" is needed to enable monitoring Ruby applications using Health Center. You can omit this option if you don't want to use Health Centre or if you are not running the broker inside the container). -
You are now running inside a Ubuntu 14.04 image. To verify that the correct rubyomrpreview version of Ruby is on the path, use the command:
$ ruby --version ruby 2.2.3p97 (OMR Preview r1) (2015-04-14) [powerpc64le-linux] -
You're ready to go!
IBM Monitoring and Diagnostic Tools - Health Center is a freely-available diagnostic tool that can monitor your applications while using only small amounts of processor time and memory. The tool can also provide suggestions for improving the efficiency of your application.
Health Center consists of the following parts:
- An agent which runs in the same environment as your application and collects data. An agent is already included in the Ruby + OMR Technology Preview.
- A client which connects to the agent to retrieve and display the data. You must download and install the client, as described in http://www.ibm.com/developerworks/java/jdk/tools/healthcenter/.
The agent and client communicate through an MQTT broker, which is not part of Health Center. The Mosquitto MQTT broker has been pre-installed in the Linux on x86 Docker image, but has not been installed in the Linux on Z Docker image (for the Z platform, it makes more sense to start the broker on another machine and specify the broker machine's host name and the port to use in the healthcenter.properties file as described in the next section).
-
Set the following environment variables:
export OMR_TRACE_OPTIONS=none
export OMR_AGENT_OPTIONS=/usr/local/lib/healthcenter/healthcenter=/usr/local/lib/healthcenter/healthcenter.properties/usr/local/libis the location where the Ruby+OMR libraries have been installed. -
(Optional) Customize the healthcenter.properties file.
For the MQTT broker host name, port, user name and password, refer to your broker documentation to find out where these values are set for the broker. The following properties are required:-
com.ibm.diagnostics.healthcenter.mqtt=on: enables the connector for the client and agent to communicate by using an MQTT broker. -
com.ibm.diagnostics.healthcenter.mqtt.broker.host=host_name: the host name of the machine on which the MQTT broker is running. The default value is localhost. -
com.ibm.diagnostics.healthcenter.mqtt.broker.port=port_number: the port number on which the MQTT broker is listening. The default value is 1883. This number should be the same as the port number that was published in the "docker run" command if the broker is running inside the container. -
com.ibm.diagnostics.healthcenter.mqtt.broker.user=user_name: the user name for the MQTT broker. -
com.ibm.diagnostics.healthcenter.mqtt.broker.pass=password: the password for the MQTT broker.
The following properties are optional:
-
com.ibm.diagnostics.healthcenter.logging.level=none|warning|info|fine|finest|debug: the level of messages output by the agent. The default is info. These messages show when the agent connects to the MQTT broker, when it loses a connection, and when it reconnects if the broker restarts. -
com.ibm.diagnostics.healthcenter.mqtt.topic.namespace=topic_namespace: a prefix to add to the string that is used to identify your agent in the available agents list in the client. You can assign to this property the name of your machine, to help distinguish your agent from other agents that are running on different machines, but sharing the same MQTT broker. -
com.ibm.diagnostics.healthcenter.mqtt.application.id=application_ID: a suffix to add to the string that is used to identify your agent in the available agents list in the client. You can assign to this property the name of your application, to help distinguish your agent from other agents that are running in different applications but sharing the same MQTT broker.
-
-
Start your MQTT broker:
mosquitto & -
Start your application. The agent attempts to connect to the broker. If the connection is successful, you should see a message similar to the following:
[Tue 21 Oct 2014 14:24:30 BST] com.ibm.diagnostics.healthcenter.mqtt INFO: Connected to broker localhost:1883 -
Start the Health Center client.
-
In the Health Center client, click File > New Connection.... A connection wizard is displayed.
-
Click Next.
-
Click the MQTT tab.
-
Specify the host name and port number. The host name is the name of the Docker host. The port number is the value of the host port that was passed to the "'docker run'" command (1883 in the example above).
-
If you want a secure connection, select the appropriate option and provide the required credentials. The user name and password must match the properties that you set earlier in the healthcenter.properties file.
-
Click Next to search for available agents, then select one from the list of agents that were found by the client.
-
Click Finish to connect to the selected agent.
The Health Center client attempts to connect to the agent that you selected. If the connection is successful, the Health Center client starts showing the data that is being collected by the agent. You can view the collected data by using the perspectives in the Health Center client. The following perspectives are available:
- CPU
- Garbage collection
- Native memory
- Profiling
For more information about Health Center, use the help in the Health Center client.
You can use a trace facility to help you diagnose problems in your applications and the runtime.
By default, no trace options are active. You specify options by using the environment variable OMR_TRACE_OPTIONS. If OMR_TRACE_OPTIONS is not set, or has an empty value, tracing is disabled. For more information about the OMR_TRACE_OPTIONS environment variable, see the Options table below.
You can print tracepoints to stderr at runtime by using the print or iprint options. To use the print or iprint options, you also need to specify the directory that contains the *TraceFormat.dat files, in addition to the print option. By default, the .dat files have been installed in the /usr/local/bin directory:
$ OMR_TRACE_OPTIONS=print=all:format=/usr/local/bin ./ruby -e 'print "Hello World\n"'
Note: If the trace engine can't find the formatting data for a tracepoint that it is attempting to print, the trace engine skips the tracepoint. Omitting the format= parameter does not cause trace to fail or display an error message, but you won't see any output.
Tracepoints are currently available for the following OMR components: all, hash table, j9thr, j9utilcore, omrmm, omrport, omrti, and pool. Over time, the j9 component names will be migrated to omr component names.
You can use the environment variable OMR_TRACE_OPTIONS to specify trace options:
OMR_TRACE_OPTIONS=[[<key>[=<value>]][:<key>[=<value>]]*]
Replace <key> and <value> with the required information.
To specify more than one key/value pair, separate the values with colons.
For some trace options only a key is needed without an associated value.
For example, the suspend and resume options do not have associated values.
If no values are specified, tracing is disabled.
| Purpose | <key> | <value> | Example |
|---|---|---|---|
| Modify the size of the buffers used to store trace points. | buffers |
|
To limit trace buffers to two buffers per thread, each of 128 KB: buffers={128k,nodynamic} |
| Specify the directory where trace format .dat files are located. This value is needed to print tracepoints to stderr. | format | <path> where <path> is the location of .dat files from OMR | Assuming .dat files are located in /usr/local/bin: format=/usr/local/bin |
| Print the output to stderr in real time, with indentation. | iprint | <tracepoint>|all where <tracepoint> is a valid tracepoint specification. | To print all tracepoints with indentation: iprint=all |
| Trace minimal data. The minimal option records only the time stamp and tracepoint identifier. | minimal | <tracepoint>|all where <tracepoint> is a valid tracepoint specification. | minimal=omrmm.0-1 |
| Trace maximum data. The maximal option specifies that all associated data is traced. Maximal includes the parameter data for the tracepoint. Select all, a range, or individual tracepoints in a component. | maximal | <tracepoint>|all where <tracepoint> is a valid tracepoint specification. | maximal=omrti |
| Disable all tracepoints. | none | - | none |
| Print the output to stderr in real time. | <tracepoint>|all where <tracepoint> is a valid tracepoint specification. | To print all tracepoints: print=all |
|
| Resume tracing globally. | resume | - | To resume tracing: resume |
| Suspends tracing globally (for all threads and all forms of tracing) but leaves tracepoints activated. | suspend | - | To suspend tracing: suspend |
The following example specifies trace options that route trace output to stderr:
OMR_TRACE_OPTIONS=format=/usr/local/bin:print=all
The environment variable OMR_GC_OPTIONS enables you to configure the OMR Garbage Collector. Currently you can configure the heap size and verbose GC output.
-Xms - Specifies the initial amount of memory the Ruby+OMR VM vm will consume. By default the value is 4M
-Xmx - Specifies the maximum amount of memory the Ruby+OMR VM vm will consume. By default the value is 1G
-Xverbosegclog: - Causes verbose GC output to be written to <filename>
If the <filename> cannot be found, it will try to create the file, and then continue as normal if it is successful. If it cannot create the file it will redirect the output to stderr.
Verbose GC output can be visualized using the Eclipse Garbage Collection Memory Visualizer (GCMV)
You can use stdout and stderr as <filename> if you would like the output directed to the console.
The following tokens can be used in <filename>
Token Description
%Y Year (4 digits)
%y Year (2 digits)
%m Month (2 digits)
%d Day of the month (2 digits)
%H Hour (2 digits)
%M Minute (2 digits)
%S Second (2 digits)
%pid Process id
Set the initial and maximum heap size to 1G and writes verbose GC output to stderr :
OMR_GC_OPTIONS="-Xms1g -Xmx1g -Xverbosegclog:stderr"
Set the initial heap size to 64M, the maximum heap size to 1G and write verbose GC output to a file called verbosegc with the current time stamp and process ID appended :
OMR_GC_OPTIONS="-Xms64m -Xmx128m -Xverbosegclog:verbosegc.%Y%m%d.%H%M%S.%pid.txt"
By default, the JIT compiler is not enabled so Ruby applications will run in the interpreter just like you're used to. To enable JIT compilation in the Ruby+OMR technology preview, use the environment variable OMR_JIT_OPTIONS as described below. The JIT compiler support is considered experimental at this point, and not all applications may be stable when the JIT compiler is enabled.
The JIT compiler currently supports most, but not all, Ruby bytecodes so not all methods can be JIT compiled. Method compilation is currently synchronous, which means that Ruby code execution is paused when a method is being compiled. When the method finishes compiling, Ruby code execution resumes by invoking the compiled method and subsequent invocations to that method from either the interpreter or other compiled methods will also dispatch to the method's compiled code, though not yet directly (compiled method dispatch still shares Ruby runtime code paths with the interpreter). When the JIT is enabled, a shared library called librbjit.so will be loaded into the Ruby process.
The JIT compiler is disabled by default unless the OMR_JIT_OPTIONS environment has been set. Enable JIT compilation by setting:
export OMR_JIT_OPTIONS="-Xjit"
This option enables JIT compilation for methods/blocks that do not contain unsupported bytecodes. Methods/blocks will be selected automatically based on method invocation count. Methods/blocks containing currently unsupported bytecodes will appear to fail compilation if verbose logging of the JIT compiler's activity has been requested. Methods that fail JIT compilation will continue to operate correctly using the Ruby interpreter even though JIT compilation failed.
export OMR_JIT_OPTIONS="-Xjit:verbose"
This option enables JIT compilation as well as verbose logging of the JIT compiler activity. Ruby methods are described by ::. When a method is selected for JIT compilation, a line will be displayed:
compiling /opt/rails/railties/lib/rails/generators/named_base.rb:165:block in parse_attributes!
If this method is successfully compiled, a line will be displayed:
+ /opt/rails/railties/lib/rails/generators/named_base.rb:165:block in parse_attributes! @ 0x7f8dac889034 37.341ms cold
The address after the is the instruction address for the entry point for the compiled method; it will be used transparently when dispatching subsequent invocations to this method. The time (27.341ms) is how long the method took to compile, and "cold" reflects the optimization strategy being used, which is currently always called "cold". This verbose output can also be directed to a file using the following option:
export OMR_JIT_OPTIONS="-Xjit:verbose,vlog=<filename>"
The ruby command's process ID will be appended to the filename provided.
The default invocation count used to select methods for JIT compilation is 1000. If you would like to experiment with a different count, you can use the count option:
export OMR_JIT_OPTIONS="-Xjit:count=N"
The value N can be any (but typically a small) integer. Because methods are compiled synchronously, lower count values do not always mean performance will improve. Using count=0 will force the JIT compiler to compile as many methods as it can, but that does not always provide the fastest performance, especially when using frameworks that repeatedly generate new Ruby methods on-the-fly. We have usually used low counts (<= 10) rather than zero to avoid effort spent compiling methods that won't run enough to produce a pay back.
To run without the JIT compiler enabled, the OMR_JIT_OPTIONS environment variable must be unset. If the OMR_JIT_OPTIONS environment variable is set to an invalid string, the JIT compiler will fail to initialize and the following output will be displayed before the ruby command aborts, as it does for any fatal error:
[FATAL] invalid OMR_JIT_OPTIONS <[invalid_option]>