Roadmap proposal for Opal v2

[hmdne]

This is more of a discussion I want to start, but it more fits to be an issue than discussion, because it's more related to development (also discussions don't back-notify issues and pull requests). I could add some issues to milestones, but I want to rather start a discussion for now because I'm certainly missing some parts.

We are reaching a point of time when we (or at least some of us) deliberately want to break some old APIs, some issues are kind of urgent, some are better to be solved sooner than later. Let's call this point of time "2.0", but also we can do some interim period when we notify users that they should change their behavior and possibly backport some of those new things in a safe way. Let's call this period "1.x".

If we decide that this plan is too big for "2.0", because we want to release "2.0" soon enough, we may want to split it into two parts, one "2.0", one "3.0".

The Great Symbolization

Target: 2.0. Pull request: #2223 (previous attempt: #2144). Status: early progress.

There's quite an urgent need to hide our Ruby methods behind a JS Symbol wall. From what I know, it involves compatibility between Asciidoctor and some MongoDB integration. This integration uses "$" attributes for its own use.

We could then possibly bridge the JS Symbols to our Symbols (instead of aliasing them to Strings). The issue here is twofold - using JS symbols incurs a small performance penalty and Ruby 1.8 (1.9?) used to have a similar issue with their Symbol tables, that every time user uses a new Symbol the memory usage grows (maybe we can find some solution in WeakRefs or FinalizationRegistry? https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WeakRef https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/FinalizationRegistry https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WeakMap https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WeakSet ). As of now, we have a similar issue with stubs, so maybe this is not such a big concern.

I don't see really a deprecation path for this going forward, but we may postpone (or abandon) the JS Symbol + Ruby Symbol bridge until somewhere later, for example "3.0". The main issue will remain as an ABI break that we may document.

Bridge JS Promises to Opal Promises

Target: 1.x. Pull request: #2220. Status: early finalization.

When Opal was first written and Promises were planned, we were in much harder times. I may be wrong about this, but maybe JS didn't have their own Promises back then - and most certainly - they weren't in use this much as they are today. They certainly weren't such a crucial part of the APIs as they are today. Async/await wasn't a thing and we rather foolishly thought about use of Generator functions to achieve something similar to Fiber.

Now we need to deal with two types of things that we try to link with a code like:

  # {Buffer} view into the blob
  #
  # If block is given it will be called with a parameter once we receive
  # the buffer. Otherwise return a {Promise} which will resolve once we
  # receive it.
  def buffer
    promise = nil
    unless block_given?
      promise = Promise.new
      block = proc { |i| promise.resolve(i) }
    end
    resblock = proc { |i| block.call(Buffer.new(i)) }
    `#@native.arrayBuffer().then(#{resblock.to_n})`
    promise
  end

Certainly it can be written better, but why not just:

def buffer
  `#@native.arrayBuffer()`
end

(I know the current code also accepts a block, but let's skip that for now).

The problem is that we won't be able to make a full compatibility between those two. The biggest issue is that Ruby promises resolve in the same tick, while JS promises are delayed. So a code like:

a = 10
Promise.value(true).then { a = 20 }
expect(a).to be 20

Simply won't work. In my opinion this isn't that big of an issue, but can break some assumptions. We also won't be able to use methods like Promise#resolved?, because JS offers no such API for inspection (I wrote some code to facilitate that for the current usage of Opal native promises, but it won't scale for JS native promises).

Please also take note that Ruby has no Promises in corelib/stdlib whatsoever.

So, if we want to have a deprecation path going forward, I would suggest the following (also this will be relevant to the further sections): current Promise becomes Promise1, while the new Promise becomes Promise2. Opal 1.x aliases Promise to Promise1, while Opal 2.0 will alias Promise to Promise2. In the meantime, using Promise would warn a developer to explicitly use either Promise1 or Promise2. We may then drop Promise1 or move to a Gem sometime after 2.0 release.

Native not feeling native enough

Target: 1.x?, Status: early planing

I view the current Native (all Native) as an afterthought. My suggestion is to revisit all this module and try to improve on it with a better plan. As above, it can become Native1, while Native2 is being worked on.

Maybe it could be a good idea to rename it. As a newcomer I couldn't understand, native to what, native to Opal or to JS?

First, Native.convert, Native.try_convert, Native::Wrapper#to_n, which one is to be used? I may know (I don't really without reading native.rb again), but try to document it. Maybe instead of that we should try to figure out a better idea, one public interface. (We're brainstorming now, aren't we? So it may be a silly idea, but possibly we can implicitly integrate some of those conversion in the ``/#{} calls, for instance new `hello(#{world})` becomes old Native(`hello(#{world.to_n})`)? If only we could reserve a new %SOMETHING{} block... But that's not part of this proposal itself)

Similarly, Native::Wrapper#{alias_native,native_reader,native_writer,native_accessor}. In my opinion they hide quite a lot of magic. Which one is for method and which one is for properties? Well, alias suggests something related to alias so it's for methods, reader/writer/accessor is for generating getters/setters for properties, right? This is what a common sense would suggest, but that's not necessarily true:

    alias_native :offset_left, :offsetLeft
    alias_native :offset_top, :offsetTop
    alias_native :page_left, :pageLeft
    alias_native :page_top, :pageTop

Thing is, you can't even do this kind of renaming with native_{reader,writer,accessor} method.

And this example will also be a good idea to look at in a different perspective. But please bear with me for a moment, I will write another example, now rather touching Native::Object:

  widget = Native(`WidgetApp`)
  widget.toggleWidget(
    callback: proc do |pt|
      pt = Native(pt)
      lid.value = pt[:value]
    end
  )

Pardon my ugly code, but does this feel "native"? I would say, it would feel more native if it was this:

  widget = Native(`WidgetApp`)
  widget.toggle_widget(
    callback: proc do |pt|
      lid.value = pt[:value]
    end
  )

(We won't be able to make it look 100% native, since obviously we have blocks and JS land doesn't have those. Also this library isn't the highest quality out there).

The difference - autoconversion of procs and camelCase to snake_case performed transparently. Most modern JS libraries use camelCase, while it feels very unnatural for Ruby. As I noted, most, some don't, but I believe we can do something about it (Native(WidgetApp, convert_names: false) for instance).

A small thing now, @davispuh was surprised that Native::Object is a descendant of BasicObject (#2113). I understand the reason behind it, though BasicObjects are generally not so friendly. The downside is that a JS object may be likely to have a send property/method, but it can also have an each property/method, yet each is implemented. I don't view Native::Object as a pure delegation/proxy object, because it always has an alternative (hash access) for those possible methods/properties. I don't have a strong opinion here though.

OK. We are coming to the last part now. Some things in Opal are bridged, some are wrapped, some are (re)implemented. I think we can ease things out for newcomers at least a little bit. #to_n is quite a good interface in the part that we never need to think in those terms (we all know that Hash is an implementation, String is bridged, but Buffer is wrapped. opal-jquery deals with bridged objects, opal-browser with wrapped... and all those have their good reasons, but some people don't know that). So we have a thing called Kernel#Native, which basically aims to convert anything to a sort of Ruby friendly object, but it doesn't. I'll go for a moment to opal-browser land, to describe an issue.

In a web browser DOM basically consists of Nodes, that we wrap, but some nodes are Elements (like <div>text</div>), some are Text nodes (the text part in the previous div), some are other kinds of data, like <!--comments-->. Easy class structure, right? Well, then some Elements are Inputs, some are Textareas, and so on - let's ignore it though. In general, if I have a JS native DOM node, I can call DOM(node) on it to wrap it with the best opal-browser wrapping class. But for now, let's assume this, a JS function can return a DOM node, or something else, let's say a File, Blob or a Buffer - those aren't DOM nodes. Kernel#DOM expects a JS native DOM node. Native to the rescue! Except that Native won't return a value wrapped with our wrappers, but with Native::Object (even Buffers! though I'm not sure if Buffers won't be wrapped with Native::Array). So my suggestion would be to introduce an interface of this form:

class File
  include Native2::Wrapper
  wrapping `File`
end

Or something similar. We could dispatch then Native() to an existing method with similar semantics like DOM() if a DOM Node is encountered. (Thing is, it would mean that this method changes semantics depending on the libraries loaded, but Opal kind of does that with bridging... We could add another kwarg to force it to make a Native::{Object,Array}).

And even though I promised it was the last part, while writing this I also found out something else. The interface for Wrapped values is as follows: Node.new(js_native_value). This is what we are used to, but I think .new has rather a semantic of creating new things, not wrapping, and we often want to create a new JS object then wrap it. This ends up with a messy #initialize method. Maybe it would be a good idea to replace this interface with Node.wrap(js_native_value)?

The main issue we see with all this is performance (and backwards compatibility :D). For that I have quite an easy answer: we already have the JS alternative interface and %x{}/`` if we value performance over code readability.

Buffer, Buffer::View, Buffer::Array

Target: 1.x, 2.x, 3.0?, Status: idea

This is a similar argument to what we had previously. I have a suspicion that when Opal was conceived those types weren't used that much. Those are mostly corespondent to Ruby binary string, but maybe JS offers somewhat a better interface (it offers a Float view for instance, in Ruby world we are left with pack/unpack which is really ugly). My early idea would be to make them bridged, not wrapped and rethink their Ruby interface, because what we have is kind of lacking.

String as a wrapped class, not bridged? A hybrid one?

Target: 3.0?, Status: idea

I won't expand on this too much, but as an idea, a new String could correspond to a Buffer or a String (it also doesn't cancel the previous idea, we may have a separate Buffer). Also this would allow us to make them mutable. If string mutability is what we need to have, I don't think we have a much better choice anyway.

Bignum support

Target: 1.x/2.x/3.0?, Status: early implementation, Pull request: #2219

I found out at some point, that JS (at least all the browsers we support) actually has Bignums, actually named BigInt. I toyed with this thing for a little bit, one of the biggest limitation of this type is that it only supports 5 basic operations and casting, but no support for those basic operation between a regular Number and BigInt (one must be casted). Ruby used to have a separate type for Bignum, now Bignum and Fixnum are merged to become an Integer - not good for our purposes I think (but I may not know something), anyway I don't think this is such a big issue.

And after a few hours of playing with this, I was genuinely surprised how powerful Ruby (and Opal) number system is. In almost no time I made them work with Rational, Complex and (probably) the entire Math library. And this alone makes Opal corelib more powerful than JS!

This implementation gives a non-standard #to_bm method, if we were to use this thing seriously, the numbers should automatically be casted to Bignum when they overflow (and back to Number when they would fit it), but that's incuring a performance penalty. Also this PR misses compiler support, so currently you would need to do things like "2134123412341234123412341234".to_bn. Also, we should be able to somehow gracefully fail on some environments that don't support BigInt (this PR has some preliminary support for that).

I understand that's not a most requested feature, but nevertheless it could expand Opal feasibility among scientific communities and that's what we may want to have. Also, it brings us closer to upstream Ruby.

update: we need to explore the idea of using BigInt as the native class for Ruby Integer

Final words

I think that Opal is seriously underappreciated in the Ruby community. I think we can abuse this situation to correct what is wrong, though as always a lot of users won't upgrade due to their legacy code. But still I think that it's important to improve the interfaces we provide.

[hmdne]

Possibly an issue related to Native refactor: #1236

[hmdne]

Regarding the String refactor, I would say that JS String could be bridged to Native::String which could have all methods the current String has (for example abstracted to a module). This would break the convention slightly though, as Native is about wrapping in general. But just throwing in an idea.

[hmdne]

There's an idea that we can start with symbolizing just the $$ properties and even release it sooner (in the 1.x series). As those are the internal interface, we can get away with those being changed. But that's just an idea. It could also be a config option, like

if (!Opal.config.symbolize_properties || typeof Symbol === 'undefined') {
  Opal.s = function(str) { return "$"+str; }
}
else {
  ....
}

[hmdne]

For Ruby compatibility we have agreed with @elia that one of the breaking changes could be to disable x-strings by default and re-enable them with some kind of a magic comment. Nothing is set in stone as of yet though.

The rationale behind this is that we can implement Kernel.` in certain environments like Node (see: #2313 - a very early draft) and then we can pass some tests. This would be a step forward for Opal on Node to become a full Ruby implementation (along JRuby, Rubinius, MRI and TruffleRuby) maybe even being able to run things like Roda someday (and - in a much more distant future - Rails). But this is a song of the future.

[hmdne]

By the way, we probably shouldn't break a lot of compatibility in a single major release (we probably won't be able to get all in time). And if we decide to proceed with changes like using BigInt as a native Ruby Integer, we may even decide to keep the previous major branch with upcoming minor releases.

[elia]

Notes from the core-team meeting of 2022-06-01

# - ::JS generic opal/javascript API (e.g. `::JS.function`, `::JS.global`, `::JS.new`, …)
# - ::Opal runtime opal/javascript API (e.g. `::Opal.bridge`, `::Opal.hash2`, …)
# - need to be referenced with the double colon
# - deprecate usage without double colon, then remove support in opal2
# - add support for "helper" magic comments inside methods for locally used helpers
# - move runtime helpers to corelib/helpers when possible with def_helpers (probably create a directory structure)

# JSI - JS interface for high-level interaction with JS
# - it's JSWrap
# - will replace Native


# contents of corelib/helpers.rb
# helpers: type_error, coerce_to

::Opal.def_helper :bridge, %{
  function(native_klass, klass) {
    if (native_klass.hasOwnProperty('$$bridge')) {
      throw Opal.ArgumentError.$new("already bridged");
    }

    // constructor is a JS function with a prototype chain like:
    // - constructor
    //   - super
    //
    // What we need to do is to inject our class (with its prototype chain)
    // between constructor and super. For example, after injecting ::Object
    // into JS String we get:
    //
    // - constructor (window.String)
    //   - Opal.Object
    //     - Opal.Kernel
    //       - Opal.BasicObject
    //         - super (window.Object)
    //           - null
    //
    $prop(native_klass, '$$bridge', klass);
    $set_proto(native_klass.prototype, (klass.$$super || Opal.Object).$$prototype);
    $prop(klass, '$$prototype', native_klass.prototype);

    $prop(klass.$$prototype, '$$class', klass);
    $prop(klass, '$$constructor', native_klass);
    $prop(klass, '$$bridge', true);
  };
}

::Opal.def_helper :coerce_to_or_raise, %{
  function() {
    `$coerce_to(object, type, method, args)`

    unless type === coerced
      ::Kernel.raise `$type_error(object, type, method, coerced)`
    end

    coerced
  }
}

[rubyFeedback]

Also, while not related to the API, please don't forget about tutorials.

There are basic examples, but I would love to see something "from zero to
a SIMPLE application". This can be a totally dead-simple one, a single
user formular with 3 input areas, but written in ruby. And then a single
.html file is used to show how this works. Ideally in a step-by-step
tutorial.

I may get dumber as I get older but I really learn best from working
examples. I got the simple opal examples to work but I don't quite
understand how it works together. Once I understand it I can add
support via opal for my ruby-webframework. But I don't quite fully
understand how things interconnect, which is why I'd love to
see a simple .html example that demonstrates ONE such use case.

[hmdne]

@rubyFeedback thank you for your feedback, we have plans to redesign the website, this will be taken into account

Regarding the examples, there is not a single way to connect Opal to your website, but I made quite a lot of opal-browser examples, describing different build processes (see integrations directory). All are valid and working, but you need to pick a specific one.

• https://github.com/opal/opal-browser/tree/master/examples

(To help you pick a correct one: we are slowly going to deprecate Sprockets support to mainly focus on Opal::Builder and all future integrations are going to be based on that)

[hmdne]

By the way, we probably shouldn't break a lot of compatibility in a single major release (we probably won't be able to get all in time). And if we decide to proceed with changes like using BigInt as a native Ruby Integer, we may even decide to keep the previous major branch with upcoming minor releases.

BigInts are unfortunately about 10 times slower than Integer-Floats...

[al6x]

We could then possibly bridge the JS Symbols to our Symbols

Hmm, I would say Ruby made mistake by making :a != "a", it caused problems, like the need for hash.stringify_keys. Opal made it right by making :a == "a", it would be better to keep it that way.

String as a wrapped class, not bridged? A hybrid one?

Sting is very frequently used, using custom class for String would make it harder to write JS integrations, where you use strings from Opal.

Keeping integration between Ruby and JS as close as possible could be huge benefit.

[al6x]

Async/await wasn't a thing and we rather foolishly thought about use of Generator functions to achieve something similar to Fiber.

I would say Fiber (or Actor/Coroutine) is superior to Async/Await. The Erlang/Go way of doing efficient IO is better than Node/Deno. JS stuck with async because it can't do better. If Opal could do a good Fiber/Actor/Coroutine atop of JS async stuff, that would be plus, not minus.