JSON::ParserError: unexpected characters after the JSON document () at line 30, column 2 [sparse.c:642]

[swiknaba]

using oj_mimic_json together with rails 6 action_text fails with JSON::ParserError: unexpected characters after the JSON document () at line 30, column 2 [sparse.c:642].

how to reproduce:

rails _6.0.2.1_ new testojmimic

add the following gems:

gem 'oj'
gem 'oj_mimic_json'

yields in:

testojmimic ry$ bundle exec rails action_text:install
rails aborted!
JSON::ParserError: unexpected characters after the JSON document () at line 30, column 2 [sparse.c:642]
/Users/ry/Sites/test/testojmimic/bin/rails:9:in `<top (required)>'
/Users/ry/Sites/test/testojmimic/bin/spring:15:in `require'
/Users/ry/Sites/test/testojmimic/bin/spring:15:in `<top (required)>'
./bin/rails:3:in `load'
./bin/rails:3:in `<main>'
Tasks: TOP => app:template
(See full trace by running task with --trace)
Copied migration 20200112140326_create_active_storage_tables.active_storage.rb from active_storage
Copied migration 20200112140327_create_action_text_tables.action_text.rb from action_text

remove gem 'oj_mimic_json' (keep gem oj, again bundle install) then run bundle exec rails action_text:install again, runs without error:

Copying actiontext.scss to app/assets/stylesheets
      create  app/assets/stylesheets/actiontext.scss
Copying fixtures to test/fixtures/action_text/rich_texts.yml
      create  test/fixtures/action_text/rich_texts.yml
Copying blob rendering partial to app/views/active_storage/blobs/_blob.html.erb
      create  app/views/active_storage/blobs/_blob.html.erb
Installing JavaScript dependencies
         run  yarn add trix@^1.0.0 @rails/actiontext@^6.0.2-1 from "."
yarn add v1.9.0-20180627.0947
[1/4] 🔍  Resolving packages...
⠁ (node:3868) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
warning " > webpack-dev-server@3.10.1" has unmet peer dependency "webpack@^4.0.0 || ^5.0.0".
warning "webpack-dev-server > webpack-dev-middleware@3.7.2" has unmet peer dependency "webpack@^4.0.0".
[4/4] 📃  Building fresh packages...

success Saved lockfile.
success Saved 2 new dependencies.
info Direct dependencies
├─ @rails/actiontext@6.0.2-1
└─ trix@1.2.2
info All dependencies
├─ @rails/actiontext@6.0.2-1
└─ trix@1.2.2
✨  Done in 15.46s.
Adding trix to app/javascript/packs/application.js
      append  app/javascript/packs/application.js
Adding @rails/actiontext to app/javascript/packs/application.js
      append  app/javascript/packs/application.js

Tried again with a fresh rails app, found here: https://github.com/swiknaba/testojmimic
ruby 2.6.5
rails 6.0.2.1

update (July 15, 2021)
updated the repo to:
ruby 2.7.3
rails 6.1.4
-> the error is still present

update (Sept 30, 2022)
minor updates:
ruby 2.7.5
rails 6.1.7
-> and also added a docker image to test the dummy app without messing with the install on your local machine; see this comment further below: #7 (comment)

[ohler55]

Any way you can show me what the string you are trying to parse looks like?

[swiknaba]

Not sure, what is being parsed. Sorry, my description is not very clear.

I've done:
Shell: Rails new app-name
Ad the oj-gems
Shell: install rich text ---> parsing error

So the error must happen inside the install script from Action-Text..

[ohler55]

Ok, will try running with your repo.

[ohler55]

How do I run the repo. I tried rails console and that was okay or sort of. It complained about yarn being out of date. I also tried bundle exec rails action_text:install.

[swiknaba]

bundle install should be enough. Maybe yarn install as well. I can check out the repo on a different machine tomorrow and try again (it's getting late in Europe^^).

[ohler55]

I managed to get past the yarn issue. You can turn off integrity checking. Anyway. What is going on is that Oj is given a stream to parse. That stream contains more than one JSON object so Oj complains when it sees the second object. That is expected behaviour so I need to figure out how that differs from the JSON gem.

If I have Oj return without an error the process continues but then runs into other errors later in the process.

Options I am considering are adding an option to not raise but just return and then turning on that option for the JSON gem mimicking.

[michaelperret]

Any progress on this? Also seeing this behavior when installing action-text

Rails 6.0.3.2
ruby 2.7.0p0

[ohler55]

Sorry, I stopped it after not getting a response. I can pick it up again this weekend. To summarize and get my head back into the issue, you are parsing a stream that contains multiple JSON documents and you would like to see Oj return without errors after the first document is read. Is that correct?

[ohler55]

Are you able to describe what you expect? Was my description in the previous comment correct?

[swiknaba]

Options I am considering are adding an option to not raise but just return and then turning on that option for the JSON gem mimicking.

this sounds like a good idea to me. It seems to be wrong to just return an ignore errors, but if that is what the JSON gem is doing, then that's what should be expected from a "mimic-json" gem I guess?

[ohler55]

I guess the real question is what is returned, the first or the last JSON document in the string or file. As long as I have a sounding board I'll take a look at what makes the most sense and throw it up here for discussion.

Thanks

[ohler55]

okay, you repo has been cloned and I ran bundle install. I tried bundle exec rails action_text:install but even after installing the yarn gem I get complaints about yarn not being included in the bundle. If you have time would you mind updating your repo with what you would like to to work with and I'll give it another try.

[ramaboo]

Just ran into this today. Any solutions?

[ohler55]

I need more to go on to stand any chance of fixing what ever you are encountering. Some sample code or project would be great.

[talhajunaid63]

Just ran into this today, Any solutions? seeing this behavior when installing action-text

Rails: 6.1.4.1
Ruby: 3.0.2

[swiknaba]

I updated the repo linked above (on Jul 15, so already a while ago. Sry. forgot to ping you here about the update).
@talhajunaid63 maybe you can push a scaffold for a fresh rails app using rails 6.1 and Ruby 3?

[talhajunaid63]

Thanks for the quick reply. Sure let me do it.

[hhassan01]

Any update guys? Still stuck on this issue :/

[ohler55]

I have not done anything in regard to this issue since I am not able to get the described environment working. If someone can give me a smaller test case I'm happy to work on a solution but trying to recreate the environment has not worked for me.

[swiknaba]

I've further slimmed down the dummy app: https://github.com/swiknaba/testojmimic/blob/master/README.md

And also hacked together a docker image; you can download it here: https://hub.docker.com/layers/lud11/swiknaba/testojmimic/images/sha256-18822523defe24a7937a397acf6b0d0d5401604e4d0e527ab41dde45a120ddc7?context=repo

Here is a short demo of how to boot the docker image, including showing that the bug is present when using oj-mimic-json and is resolved by removing the gem: https://www.loom.com/share/3304286f50ff47c79228b58e0482eea4

I hope that helps the investigation :)

[ohler55]

thanks, will try to get to it this week.

[ohler55]

I know this has been going on for a long time. At this point I still don't know what the JSON file looks like that is causing the issue. If you want to solve this and are willing to zoom maybe we can get on a call and try to see what is going on together. Reach me at peter@ohler.com.

[jgarth]

Hi all, hope I can add some relevant info to this. I encountered the same error when using HTTPX in combination with Oj. Upon investigation it turns out that given a ruby object body, which prints as #<HTTPX::Response::Body:20400 @state=memory @length=339>, I can do Oj.load(body.to_s), but not Oj.load(body). I can however do JSON.parse(body).

AFAICT, Oj checks if something is a ruby String here, and if it isn't, uses a different method of parsing. It doesn't call .to_s or inspect. Arguably it's not Oj's job to call to_s on anything before parsing it, either. Maybe the default JSON implementation does that. Either way, some 3rd party code depends on this behavior.

My C-fu is not strong enough to figure out what exactly Oj is doing after the call to oj_pi_sparse, so I'm not sure how to progress from here. But I can now give you a really simple reproduction case and hope it helps:

require 'httpx'
require 'oj'

# Works as intended
puts "Without Oj"
response = HTTPX.get('https://dummyjson.com/products/1')
puts JSON.parse(response.body)

# Raises "JSON::ParserError: unexpected characters after the JSON document…"
puts "With Oj"
Oj.mimic_JSON()
response = HTTPX.get('https://dummyjson.com/products/1')
puts JSON.parse(response.body)

Let me know if there's anything I can do to assist further, and thanks for making this gem @ohler55 🎉

[ohler55]

I'll look into it. It may be that the https response is a steam or has the read methods to act like a stream. Anyway, let me do some digging.

[ohler55]

It turns out the https response does respond to the read method but it always returns the body as a full string so repeated calls to read always return a full string which makes the response appear to be an infinitely long string. Of course that is not how the read method is supposed to work. I'll have to look for a way to detect that broken behaviour and work around it.

[ohler55]

Created an issue on httpx: https://gitlab.com/os85/httpx/-/issues/237

[ohler55]

@honeyryderchuck made the fix so using the latest https should work fine.