Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error from psych after sup-config #577

Closed
krerkkiat opened this issue Jul 16, 2020 · 9 comments · Fixed by #598
Closed

Error from psych after sup-config #577

krerkkiat opened this issue Jul 16, 2020 · 9 comments · Fixed by #598

Comments

@krerkkiat
Copy link

exception-log.txt content

--- Psych::SyntaxError from thread: main
(/home/krerkkiat/.sup/sources.yaml): did not find expected whitespace or line break while scanning a tag at line 2 column 3
/home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:456:in `parse'
/home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:456:in `parse_stream'
/home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:390:in `parse'
/home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:277:in `load'
/home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:578:in `block in load_file'
/home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:577:in `open'
/home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:577:in `load_file'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup.rb:145:in `load_yaml_obj'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:227:in `load_sources'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/util.rb:605:in `method_missing'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/index.rb:102:in `load'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/util.rb:605:in `method_missing'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/bin/sup:146:in `<module:Redwood>'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/bin/sup:76:in `<top (required)>'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/sup:23:in `load'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/sup:23:in `<main>'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `eval'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `<main>'

content of .sup/sources.yml

---
- !supmua.org,2006-10-01/Redwood/Maildir
  uri: maildir:/home/krerkkiat/mail/personal
  usual: true
  archived: false
  sync_back: true
  id: 1
  labels: []
@IPv2
Copy link
Member

IPv2 commented Jul 16, 2020

Thanks for reporting @krerkkiat.

I haven't managed to reproduce this yet - your .sup/sources.yaml is accepted by psych (and so also by sup) when I tried on my machine.

Please could you try running these two commands and let me know the full output for both? These should provide some extra diagnostic information.

ruby -e 'require "yaml"; puts RUBY_VERSION; puts YAML::load_file "/home/krerkkiat/.sup/sources.yaml"'

hexdump -C /home/krerkkiat/.sup/sources.yaml

@krerkkiat
Copy link
Author

krerkkiat commented Jul 17, 2020

$ hexdump -C /home/krerkkiat/.sup/sources.yaml
00000000  25 59 41 4d 4c 20 31 2e  31 0a 2d 2d 2d 0a 2d 20  |%YAML 1.1.---.- |
00000010  21 6d 61 73 61 6e 6a 69  6e 2e 6e 65 74 2c 32 30  |!masanjin.net,20|
00000020  30 36 2d 31 30 2d 30 31  2f 52 65 64 77 6f 6f 64  |06-10-01/Redwood|
00000030  2f 4d 61 69 6c 64 69 72  0a 20 20 75 72 69 3a 20  |/Maildir.  uri: |
00000040  6d 61 69 6c 64 69 72 3a  2f 68 6f 6d 65 2f 6b 72  |maildir:/home/kr|
00000050  65 72 6b 6b 69 61 74 2f  70 65 72 73 6f 6e 61 6c  |erkkiat/personal|
00000060  0a 20 20 75 73 75 61 6c  3a 20 74 72 75 65 0a 20  |.  usual: true. |
00000070  20 61 72 63 68 69 76 65  64 3a 20 66 61 6c 73 65  | archived: false|
00000080  0a 20 20 69 64 3a 20 31  0a 20 20 6c 61 62 65 6c  |.  id: 1.  label|
00000090  73 3a 20 5b 5d 0a                                 |s: [].|
00000096
$ ruby -e 'require "yaml"; puts RUBY_VERSION; puts YAML::load_file "/home/krerkkiat/.sup/sources.yaml"'
2.7.0
Traceback (most recent call last):
	7: from -e:1:in `<main>'
	6: from /home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:577:in `load_file'
	5: from /home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:577:in `open'
	4: from /home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:578:in `block in load_file'
	3: from /home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:277:in `load'
	2: from /home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:390:in `parse'
	1: from /home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:456:in `parse_stream'
/home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:456:in `parse': (/home/krerkkiat/.sup/sources.yaml): did not find expected whitespace or line break while scanning a tag at line 2 column 3 (Psych::SyntaxError)

The same error occurs even with ruby 2.6.5.

However, if I modify the sources.yaml to be

---
- !<masanjin.net,2006-10-01/Redwood/Maildir>
  uri: maildir:/home/krerkkiat/personal
  usual: true
  archived: false
  id: 1
  labels: []

It works.

$ ruby -e 'require "yaml"; puts RUBY_VERSION; puts YAML::load_file "/home/krerkkiat/.sup/sources.yaml"'
2.7.0
{"uri"=>"maildir:/home/krerkkiat/personal", "usual"=>true, "archived"=>false, "id"=>1, "labels"=>[]}

Does it have to do with changing spec in YAML? This comment on libyaml mentioned that comma is invalid in tags. I did try to specify the version to be 1.1, but that still produce the same error. The libyaml on my machine is 0.2.5.

Although, if I use <> in the tag, I got this error instead.

$ sup-config
Traceback (most recent call last):
	11: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `<main>'
	10: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `eval'
	 9: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/sup-config:23:in `<main>'
	 8: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/sup-config:23:in `load'
	 7: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/bin/sup-config:109:in `<top (required)>'
	 6: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/util.rb:605:in `method_missing'
	 5: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:228:in `load_sources'
	 4: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:228:in `synchronize'
	 3: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:229:in `block in load_sources'
	 2: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:229:in `map'
	 1: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:229:in `block (2 levels) in load_sources'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup.rb:20:in `id': wrong id called on {"uri"=>"maildir:/home/krerkkiat/personal", "usual"=>true, "archived"=>false, "id"=>1, "labels"=>[]} (RuntimeError)

@IPv2
Copy link
Member

IPv2 commented Jul 17, 2020

Thank you @krerkkiat - that is extremely helpful.

I have been able to reproduce on my machine now, after upgrading my libyaml from 0.2.2 to 0.2.5

So I agree, yes, this is directly linked to yaml/libyaml#196 (comment) that you found, and has started to fail because of the YAML spec change (the comma in !supmua.org,2006-10-01/Redwood/Maildir was valid in 1.1 but not 1.2, and libyaml 0.2.5 only supports YAML 1.2 with no backwards compatibility for 1.1).

If you want a quick fix to get up and running, you could manually change your sources.yaml and your installed Sup to use a non-standard modified YAML source, without the comma. For example:

# ~/.sup/sources.yaml
- !supmua.org/2006-10-01/Redwood/Maildir
  uri: maildir:/home/krerkkiat/personal
  usual: true
  archived: false
  sync_back: true
  id: 1
  labels: []
# ~/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup.rb - line 29
# OLD: yaml_tag "!#{Redwood::YAML_DOMAIN},#{Redwood::YAML_DATE}/#{path}"
yaml_tag "!#{Redwood::YAML_DOMAIN}/#{Redwood::YAML_DATE}/#{path}"

But we'll need to workaround this in Sup itself, and this might cause you a headache down the line, because you may need to undo your sources.yaml change later when we release a new Sup version with a workaround.

I'll have a bit of a think on this one about how to tackle this. I suspect we'll need to introduce code in Sup to migrate from the existing YAML tag to a new YAML tag (without a comma).

@IPv2
Copy link
Member

IPv2 commented Jul 17, 2020

I had a better idea for a different quick-fix workaround, which will get you up-and-running for now, and without modifying your configuration in ways that may cause headaches in future (for when later have a permanent workaround in Sup itself).

The idea is: download your own copy of an older version of libyaml (perhaps 0.2.4), and set the LD_LIBRARY_PATH environment variable when launching Sup, so that Sup alone uses it. You don't need to install the older libyaml system-wide, so no other programs would be affected.

Steps would be something like:

mkdir ~/libyaml
cd ~/libyaml
curl -o libyaml-dist-0.2.4.tar.gz https://codeload.github.com/yaml/libyaml/tar.gz/dist-0.2.4
tar -zxvf libyaml-dist-0.2.4.tar.gz
cd libyaml-dist-0.2.4/
./configure
make

# Then launch sup with an appropriate LD_LIBRARY_PATH
LD_LIBRARY_PATH=~/libyaml/libyaml-dist-0.2.4/src/.libs:$LD_LIBRARY_PATH sup

Note that you'll need to use a standard sources.yaml file, similar to the one in your first comment (not your later comment). This should work:

# sources.yaml
---
- !supmua.org,2006-10-01/Redwood/Maildir
  uri: maildir:/home/krerkkiat/mail/personal
  usual: true
  archived: false
  sync_back: true
  id: 1
  labels: []

@IPv2
Copy link
Member

IPv2 commented Jul 20, 2020

Alternative quick fix (per YAML 1.2 spec, section 6.8.2):

  • In sources.yaml, replace (e.g.) !supmua.org,2006-10-01/Redwood/Maildir with !<!supmua.org,2006-10-01/Redwood/Maildir>
  • Note that the same fix may need to be repeated ever time Sup rewrites sources.yaml

@krerkkiat
Copy link
Author

Thank you for the help!

I went with the source code modifying method (the one at line lib/sup.rb:29), and I think I should be all good now in terms of the sources.yaml file.

I think I did try the last method already (using !<!supmua.org,2006-10-01/Redwood/Maildir>) unless I miss-understanding. Okay. I tried it again, and this is the result:

--- RuntimeError from thread: main
wrong id called on {"uri"=>"maildir:/home/krerkkiat/mail/personal", "usual"=>true, "archived"=>false, "sync_back"=>true, "id"=>1, "labels"=>["gmail"]}
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup.rb:20:in `id'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:229:in `block (2 levels) in load_sources'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:229:in `map'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:229:in `block in load_sources'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:228:in `synchronize'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:228:in `load_sources'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/util.rb:605:in `method_missing'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/index.rb:102:in `load'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/util.rb:605:in `method_missing'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/bin/sup:146:in `<module:Redwood>'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/bin/sup:76:in `<top (required)>'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/sup:23:in `load'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/sup:23:in `<main>'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `eval'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `<main>'

@IPv2
Copy link
Member

IPv2 commented Jul 21, 2020

Thanks @krekkiat!

The "wrong id" error occurs when there's a mismatch between the tag expected by Sup in lib/sup.rb, and the tag in sources.yaml.

If you revert your manual changes to lib/sup.rb first (or reinstall the sup gem), and then make the sources.yaml change, then you should avoid that error.

(This is also why you might prefer to go down this route - because otherwise when you install a new Sup, you are likely to see that error message, if you use a non-standard tag.)

@danc86
Copy link
Contributor

danc86 commented Feb 16, 2021

I don't think there is anything wrong with using the comma character in tag URIs. The issue seems to be just with the syntax that sup was using in sources.yaml. Since upgrading to libyaml 0.2.5 I updated my sources.yaml to look like this, which I think is using the YAML tag support in the way that it was intended:

%TAG ! tag:supmua.org,2006-10-01/Redwood/
---
- !Maildir
  uri: ...

I also needed a corresponding fix in sup, to make it declare its actual complete tag URI (using tag:... instead of the ! syntax from YAML):

diff --git a/lib/sup.rb b/lib/sup.rb
index e90c971d..209ddea9 100644
--- a/lib/sup.rb
+++ b/lib/sup.rb
@@ -26,7 +26,7 @@ class Module
     props = props.map { |p| p.to_s }

     path = name.gsub(/::/, "/")
-    yaml_tag "!#{Redwood::YAML_DOMAIN},#{Redwood::YAML_DATE}/#{path}"
+    yaml_tag "tag:#{Redwood::YAML_DOMAIN},#{Redwood::YAML_DATE}/#{path}"

     define_method :init_with do |coder|
       initialize(*coder.map.values_at(*props))

What I haven't had time to figure out yet is, whether that patch will break on older versions of libyaml, and also how we can gracefully and safely migrate existing sources.yaml files with the previous incorrect syntax.

@danc86
Copy link
Contributor

danc86 commented Apr 17, 2022

I used podman run -it centos:7 to easily get an old Ruby environment. It has Psych 2.0.0 and libyaml 0.1.4. I confirmed that the change to declare the proper tag URI I wrote above works fine with these old versions.

It causes sup-add to emit sources.yaml with syntax like this, which appears to be the proper way of declaring a "global tag" and was probably what sup always intended:

---
- !<tag:supmua.org,2006-10-01/Redwood/Maildir>
  uri: maildir:///some/path
  usual: true
  archived: false
  sync_back: true
  id: 1
  labels: []

That also parses fine and works as intended on later libyaml.

The only remaining issue is how we can automatically fix up older sources.yaml when sup encounters them.

danc86 added a commit to danc86/sup that referenced this issue Apr 17, 2022
danc86 added a commit to danc86/sup that referenced this issue Apr 17, 2022
danc86 added a commit to danc86/sup that referenced this issue Apr 17, 2022
danc86 added a commit to danc86/sup that referenced this issue Apr 17, 2022
danc86 added a commit that referenced this issue Apr 17, 2022
danc86 added a commit that referenced this issue Apr 17, 2022
This was referenced Apr 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants