incompatible character encodings: UTF-8 and ASCII-8BIT

[schmunk42]

I am having the following error, when editing pages via gollum, which were modified with an editor (PHPstorm or Mou) and pushed back into the wiki repo - and there as usually located two levels (folders) below root.

Encoding::CompatibilityError at /edit/my-page
incompatible character encodings: UTF-8 and ASCII-8BIT

file: index.rb location: % line: 176

I've looked through several encoding issues I've found here. And came to the conclusion, that this is an error with grit (fork from gitlab).

I've tried different version of ruby (1.9.3, 2.1.2) and gollum (2.1.0, 2.5.0, 3.0.0) to fix the problem, but the only way (for me) to work around this issue, was modifiying this line to:

tree_contents[k] = "%s %s\0%s" % [tmode, obj.name.force_encoding('ASCII-8BIT'), sha]

Which does not really look right to me.
Is there any way to fix this in gollum or even via config.rb?

Error Stacktrace:

localhost - - [28/Jul/2014:19:31:17 CEST] "GET /javascript/editor/langs/markdown.js?_=1406568677805 HTTP/1.1" 200 11837
http://localhost:4567/edit/intern/orga/my-page -> /javascript/editor/langs/markdown.js?_=1406568677805
Encoding::CompatibilityError - incompatible character encodings: UTF-8 and ASCII-8BIT:
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/gitlab-grit-2.6.7/lib/grit/index.rb:176:in `%'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/gitlab-grit-2.6.7/lib/grit/index.rb:176:in `block in write_tree'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/gitlab-grit-2.6.7/lib/grit/index.rb:171:in `each'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/gitlab-grit-2.6.7/lib/grit/index.rb:171:in `write_tree'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/gitlab-grit-2.6.7/lib/grit/index.rb:198:in `block in write_tree'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/gitlab-grit-2.6.7/lib/grit/index.rb:180:in `each'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/gitlab-grit-2.6.7/lib/grit/index.rb:180:in `write_tree'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/gitlab-grit-2.6.7/lib/grit/index.rb:198:in `block in write_tree'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/gitlab-grit-2.6.7/lib/grit/index.rb:180:in `each'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/gitlab-grit-2.6.7/lib/grit/index.rb:180:in `write_tree'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/gitlab-grit-2.6.7/lib/grit/index.rb:123:in `commit'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/gollum-lib-3.0.0/lib/gollum-lib/committer.rb:170:in `commit'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/gollum-3.0.0/lib/gollum/app.rb:248:in `block in <class:App>'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:1603:in `call'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:1603:in `block in compile!'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `[]'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `block (3 levels) in route!'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:985:in `route_eval'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `block (2 levels) in route!'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:1006:in `block in process_route'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in `catch'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in `process_route'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:964:in `block in route!'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in `each'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in `route!'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:1076:in `block in dispatch!'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:1073:in `dispatch!'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `block in call!'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `call!'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:886:in `call'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/rack-1.5.2/lib/rack/nulllogger.rb:9:in `call'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/rack-1.5.2/lib/rack/head.rb:11:in `call'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/show_exceptions.rb:21:in `call'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:180:in `call'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:2014:in `call'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:1478:in `block in call'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:1788:in `synchronize'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/sinatra-1.4.5/lib/sinatra/base.rb:1478:in `call'
    /opt/local/lib/ruby1.9/gems/1.9.1/gems/rack-1.5.2/lib/rack/handler/webrick.rb:60:in `service'
    /opt/local/lib/ruby1.9/1.9.1/webrick/httpserver.rb:138:in `service'
    /opt/local/lib/ruby1.9/1.9.1/webrick/httpserver.rb:94:in `run'
    /opt/local/lib/ruby1.9/1.9.1/webrick/server.rb:191:in `block in start_thread'

[thelinuxlich]

+1 for this!

[thelinuxlich]

Would this work? https://github.com/gitlabhq/grit_ext

Apparently is a extension to gitlab grit which adds utf8 support

[schmunk42]

Umm, I can't retest this at the moment, but could you try to wrap gollum into rack, like described here and see if the error goes away.

[dometto]

@thelinuxlich does the workaround proposed by @schmunk42 work? Otherwise, we would certainly appreciate a PR adding grit_ext to https://github.com/gollum/grit_adapter (our own priorities lie with moving away from grit alltogether).

[sssfff]

I can confirm that the work-around mentioned by @schmunk42 (edit line 176 in grit/index.rb) is working well. This solved my UTF-8 encoding problem.

Thanks!

[bartkamphorst]

@schmunk42 @sssfff @thelinuxlich Could any of you test whether this issue still persists with gollum & gollum-lib's rc branches and grit_adapter v.0.0.2 (which now uses gitlab-grit v.2.7.1 instead of 2.6.5)?

[dometto]

A regression test for this issue in gollum-lib would be useful anyway so we can make sure that alternative adapters don't break this in the future.

[bartkamphorst]

Yes, but judging from @schmunk42's original problem description, a regression test might be tricky to come up with:

I am having the following error, when editing pages via gollum, which were modified with an editor (PHPstorm or Mou) and pushed back into the wiki repo - and there as usually located two levels (folders) below root.

But I agree that it would be very helpful to have.

[datenzauberer]

This workaround saved my day. Many thanks @schmunk42

[CheneyWong]

+1 for this and thanks @schmunk42

[CheneyWong]

and a similar problem : #830

[bartkamphorst]

@ChanNew We are aware of the issues and their relatedness. What's unclear to me at the moment is whether this issue (or these issues) still persists (in the same form) with gollum & gollum-lib's rc branches (v. 4.0 and v. 3.1 respectively) and grit_adapter v.0.0.2 (which now uses gitlab-grit v.2.7.1 instead of 2.6.5). Would you be willing to test that? Also, having a good test case as @dometto suggested would be a helpful step towards a solution. Thanks.

[bennibu]

@bartkamphorst I can confirm this issue. We use gollum 3.1.0 (gollum-lib 4.0.0, gitlab-grit 2.7.2, ruby 2.0.0). The manuall patch for gitlab-grit from @schmunk42 works fine.

The test case is hard. We stumbled upon the problem when adding a page under the path /SSL_Zertifikate. So we are not using any non ascii characters here.

[schmunk42]

@bennibu We're experiencing a similar issue, we also have a lot of German Umlauts in our markdown docs, but they are not the problem. I can create new pages on the root level with contents from another page eg. /two/levels/deep.md which fails when updating.
So I thought I there may be also problem in the URLs with the /, but I can't reproduce it also.

Another thing which I haven't investigated too deeply are the locales (ENV vars) - do they affect Ruby or Rack?

PS: I tried to migrate our wiki to the latest version in a Docker container, here's the WIP Code - maybe we could create a reproduceable test case with it.