Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for assigning to dfree in native extensions #2830

Closed
nirvdrum opened this issue Jan 17, 2023 · 6 comments · Fixed by #2841
Closed

Add support for assigning to dfree in native extensions #2830

nirvdrum opened this issue Jan 17, 2023 · 6 comments · Fixed by #2841
Assignees
@eregon
Labels
cexts
Milestone
23.0.0 Release (J...

Comments

@nirvdrum
Copy link
Collaborator

nirvdrum commented Jan 17, 2023 •
edited by eregon

I'm working on a project that uses Commonmarker 0.23.6 to process text in the CommonMark format. It errors out because it tries to use the dfree field in rb_data_type_struct, which it looks like we don't currently implement.

https://github.com/gjtorikian/commonmarker/blob/v0.23.6/ext/commonmarker/commonmarker.c#L115

> ruby -v
truffleruby 23.0.0-dev-f3db7ba6, like ruby 3.1.3, GraalVM CE Native [aarch64-darwin]

RuntimeError: Neutered Exception Polyglot::ForeignException: Member 'dfree' not found.
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/ext/commonmarker/commonmarker.c:490:in `rb_node_unlink'
    /Users/nirvdrum/dev/workspaces/truffleruby-ws/graal/sdk/mxbuild/darwin-aarch64/GRAALVM_5080722BFA_JAVA19/graalvm-5080722bfa-java19-23.0.0-dev/Contents/Home/languages/ruby/lib/truffle/truffle/cext_ruby.rb:41:in `<unknown>'
    /Users/nirvdrum/dev/workspaces/truffleruby-ws/graal/sdk/mxbuild/darwin-aarch64/GRAALVM_5080722BFA_JAVA19/graalvm-5080722bfa-java19-23.0.0-dev/Contents/Home/languages/ruby/lib/truffle/truffle/cext_ruby.rb:41:in `CommonMarker::Node#delete'
    <... replaced ...>
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:16:in `CommonMarker::Node#walk'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:18:in `block in CommonMarker::Node#walk'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:72:in `CommonMarker::Node#each'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:17:in `CommonMarker::Node#walk'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:18:in `block in CommonMarker::Node#walk'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:72:in `CommonMarker::Node#each'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:17:in `CommonMarker::Node#walk'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:18:in `block in CommonMarker::Node#walk'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:72:in `CommonMarker::Node#each'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:17:in `CommonMarker::Node#walk'
    <... replaced ...>
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:44:in `CommonMarker::Renderer#render'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:28:in `block in CommonMarker::Renderer#out'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:21:in `Array#each'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:21:in `CommonMarker::Renderer#out'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:24:in `block (2 levels) in CommonMarker::Renderer#out'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:72:in `CommonMarker::Node#each'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:24:in `block in CommonMarker::Renderer#out'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:21:in `Array#each'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:21:in `CommonMarker::Renderer#out'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:53:in `CommonMarker::Renderer#document'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer/html_renderer.rb:6:in `CommonMarker::HtmlRenderer#document'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:38:in `CommonMarker::Renderer#render'
@eregon
Copy link
Member

eregon commented Jan 17, 2023

Seems a duplicate of #2732.
I'd think #2771 which was fixed would fix this too, but maybe it's slightly different.
I'll take a look.

@eregon eregon self-assigned this Jan 17, 2023
@eregon eregon added the cexts label Jan 17, 2023
@nirvdrum
Copy link
Collaborator Author

Oops. This is a dupe of #2732. I'm sorry about that. Just note that the version number is significant. commonmarker main tracks a pre-release version. I think it might not use dfree any longer, but I had some difficulties getting it to load at all. I'll have to look into that more some other time.

@nirvdrum
Copy link
Collaborator Author

I didn't look at the fix for #2771, but one thing that stands out is #2771 is about reading dfree whereas this issue and #2732 are about writing to the field.

@nirvdrum nirvdrum changed the title Add support for dfree in native extensions Add support for assigning to dfree in native extensions Jan 17, 2023
@eregon eregon mentioned this issue Jan 26, 2023
Closed
@eregon
Copy link
Member

eregon commented Jan 26, 2023 •
edited

Running the test suite I get:

dead handle 0xbad00000004a8b0 (com.oracle.truffle.api.CompilerDirectives.ShouldNotReachHere)
	from com.oracle.truffle.api.CompilerDirectives.shouldNotReachHere(CompilerDirectives.java:574)
	from com.oracle.truffle.api.CompilerDirectives.shouldNotReachHere(CompilerDirectives.java:520)
	from org.truffleruby.cext.UnwrapNode$UnwrapNativeNode.raiseError(UnwrapNode.java:95)
	from org.truffleruby.cext.UnwrapNode$UnwrapNativeNode.unwrapTaggedObject(UnwrapNode.java:80)
	from org.truffleruby.cext.UnwrapNodeGen$UnwrapNativeNodeGen.execute(UnwrapNodeGen.java:482)
	from org.truffleruby.cext.UnwrapNode.longToWrapper(UnwrapNode.java:253)
	from org.truffleruby.cext.UnwrapNodeGen.execute(UnwrapNodeGen.java:100)
	from org.truffleruby.cext.CExtNodes$CallWithCExtLockAndFrameAndUnwrapNode.callWithCExtLockAndFrame(CExtNodes.java:249)
	from org.truffleruby.cext.CExtNodesFactory$CallWithCExtLockAndFrameAndUnwrapNodeFactory$CallWithCExtLockAndFrameAndUnwrapNodeGen.execute(CExtNodesFactory.java:598)
	from org.truffleruby.language.locals.WriteLocalVariableNode.execute(WriteLocalVariableNode.java:28)
	from org.truffleruby.language.RubyNode.doExecuteVoid(RubyNode.java:64)
	from org.truffleruby.language.control.SequenceNode.execute(SequenceNode.java:34)
	from org.truffleruby.core.module.ModuleNodes$DefineMethodNode$CallMethodWithLambdaBody.execute(ModuleNodes.java:1482)
	from org.truffleruby.language.RubyLambdaRootNode.execute(RubyLambdaRootNode.java:84)
/home/eregon/.rubies/truffleruby-dev/lib/truffle/truffle/cext_ruby.rb:22:in `next'
	from /home/eregon/code/commonmarker/lib/commonmarker/node.rb:71:in `each'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block in out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:133:in `strong'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:44:in `render'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:28:in `block in out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block (2 levels) in out'
	from /home/eregon/code/commonmarker/lib/commonmarker/node.rb:72:in `each'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block in out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:23:in `block (2 levels) in paragraph'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:84:in `container'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:22:in `block in paragraph'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:78:in `block'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:21:in `paragraph'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:44:in `render'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:28:in `block in out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block (2 levels) in out'
	from /home/eregon/code/commonmarker/lib/commonmarker/node.rb:72:in `each'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block in out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:53:in `document'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:6:in `document'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:38:in `render'
	from /home/eregon/code/commonmarker/test/test_spec.rb:26:in `test_sourcepos_example_404'

So the dree doesn't seem the only issue.

BTW I noticed the gem switched from libcmark-gfm for 0.x to comrak for 1.0, which is Rust-based.

@eregon
Copy link
Member

eregon commented Jan 26, 2023 •
edited

I have a PR to support writing to dfree: #2841
With that, the test suite fails like so (note: it's actually a lot faster to run the test suite on jvm-ce than jvm):

$ chruby truffleruby-jvm-ce
$ bundle exec rake test TESTOPTS="-s0 -v"
...
TestSpec#test_to_html_example_553 = 0.00 s = .
TestSpec#test_sourcepos_example_436 = 
truffleruby: Unexpected internal exception in at_exit,
please report it to https://github.com/oracle/truffleruby/issues.

dead handle 0xbad000000002280 (com.oracle.truffle.api.CompilerDirectives.ShouldNotReachHere)
	from com.oracle.truffle.api.CompilerDirectives.shouldNotReachHere(CompilerDirectives.java:574)
	from com.oracle.truffle.api.CompilerDirectives.shouldNotReachHere(CompilerDirectives.java:520)
	from org.truffleruby.cext.UnwrapNode$UnwrapNativeNode.raiseError(UnwrapNode.java:95)
	from org.truffleruby.cext.UnwrapNode$UnwrapNativeNode.unwrapTaggedObject(UnwrapNode.java:80)
	from org.truffleruby.cext.UnwrapNodeGen$UnwrapNativeNodeGen.execute(UnwrapNodeGen.java:482)
	from org.truffleruby.cext.UnwrapNode.longToWrapper(UnwrapNode.java:253)
	from org.truffleruby.cext.UnwrapNodeGen.execute(UnwrapNodeGen.java:100)
	from org.truffleruby.cext.CExtNodes$CallWithCExtLockAndFrameAndUnwrapNode.callWithCExtLockAndFrame(CExtNodes.java:249)
	from org.truffleruby.cext.CExtNodesFactory$CallWithCExtLockAndFrameAndUnwrapNodeFactory$CallWithCExtLockAndFrameAndUnwrapNodeGen.execute(CExtNodesFactory.java:599)
	from org.truffleruby.language.locals.WriteLocalVariableNode.execute(WriteLocalVariableNode.java:28)
	from org.truffleruby.language.RubyNode.doExecuteVoid(RubyNode.java:64)
	from org.truffleruby.language.control.SequenceNode.execute(SequenceNode.java:34)
	from org.truffleruby.core.module.ModuleNodes$DefineMethodNode$CallMethodWithLambdaBody.execute(ModuleNodes.java:1482)
	from org.truffleruby.language.RubyLambdaRootNode.execute(RubyLambdaRootNode.java:84)
/home/eregon/code/truffleruby-ws/graal/sdk/mxbuild/linux-amd64/GRAALVM_A33290E019_JAVA20/graalvm-a33290e019-java20-23.0.0-dev/languages/ruby/lib/truffle/truffle/cext_ruby.rb:22:in `next'
	from /home/eregon/code/commonmarker/lib/commonmarker/node.rb:71:in `each'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block in out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:133:in `strong'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:44:in `render'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:28:in `block in out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block (2 levels) in out'
	from /home/eregon/code/commonmarker/lib/commonmarker/node.rb:72:in `each'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block in out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:23:in `block (2 levels) in paragraph'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:84:in `container'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:22:in `block in paragraph'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:78:in `block'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:21:in `paragraph'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:44:in `render'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:28:in `block in out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block (2 levels) in out'
	from /home/eregon/code/commonmarker/lib/commonmarker/node.rb:72:in `each'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block in out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:53:in `document'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:6:in `document'
	from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:38:in `render'
	from /home/eregon/code/commonmarker/test/test_spec.rb:26:in `test_sourcepos_example_436'

I'll merge that first and we can investigate this dead handle issue later, they don't seem related.

Command-line to repro similar error: ruby -rbundler/setup -Ilib:test test/test_spec.rb -s0 -v

The whole test suite passes if we just keep all handles (obviously just a workaround):
TRUFFLERUBYOPT="--experimental-options --keep-handles-alive" bundle exec rake test TESTOPTS="-s0 -v"

@graalvmbot graalvmbot mentioned this issue Jan 26, 2023
Merged
@eregon eregon added this to the 23.0.0 Release milestone Jan 26, 2023
@eregon eregon mentioned this issue Jan 26, 2023
Open
@eregon
Copy link
Member

eregon commented Jan 26, 2023

Tracking the dead handle issue in #2842

nirvdrum pushed a commit to Shopify/truffleruby that referenced this issue Jan 30, 2023
@eregon @nirvdrum
Support writing to RData.dfree for native extensions
b3aabbf 
* Fixes oracle#2830
* Fixes oracle#2732
* Fixes oracle#2165
sophie-kaleba pushed a commit to sophie-kaleba/truffleruby that referenced this issue Feb 16, 2023
@eregon @sophie-kaleba
Support writing to RData.dfree for native extensions
d8d2e2c 
* Fixes oracle#2830
* Fixes oracle#2732
* Fixes oracle#2165
sophie-kaleba pushed a commit to sophie-kaleba/truffleruby that referenced this issue Feb 16, 2023
@eregon @sophie-kaleba
Support writing to RData.dfree for native extensions
d4f9fc5 
* Fixes oracle#2830
* Fixes oracle#2732
* Fixes oracle#2165
john-heinnickel pushed a commit to thermofisher-jch/truffleruby that referenced this issue Aug 16, 2023
@eregon @john-heinnickel
Support writing to RData.dfree for native extensions
7273bcb 
* Fixes oracle#2830
* Fixes oracle#2732
* Fixes oracle#2165
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@eregon eregon

Labels
cexts
Projects
None yet
Milestone
23.0.0 Release (June 13, 2023)
Development

Successfully merging a pull request may close this issue.

[GR-18163] Support writing to RData.dfree for native extensions oracle/truffleruby
2 participants
@nirvdrum @eregon