Windows RDoc encoding issue

[voxik]

Hello,

Running command:
C:\Users\vita.pik\rubies\Ruby-193-dev\lib\ruby\gems\1.9.1\gems\bundler-1.0.9>rdoc --op C:/Users/vita/.pik/rubies/Ruby-193-dev/lib/ruby/gems/1.9.1/doc/bundler-1.0.9/rdoc lib --title="bundler-1.0.9 Documentation" -v --debug 2> \temp\rdoc.log

to install Bundler documentation produces following codepage conversion error: http://pastie.org/1506876

C:\>rdoc --version
rdoc 3.1

C:\>ruby --version
ruby 1.9.3dev (2011-01-28) [i386-mingw32]

C:\>chcp
Aktivní znaková stránka: 852

Changing codepage to 65001 (utf-8) does not produce the error. However it is mystery to me why should documentation generation depend on current console codepage.

[drbrain]

Please try again with the latest RDoc (3.4) and reopen if it still occurs, this should be fixed as of RDoc 3.3.

RDoc assumes you want output in your native encoding which is determined by ruby using the locale. You can tell RDoc to always output UTF-8 by setting the RDOCOPT environment variable to "--encoding=UTF-8".

[voxik]

It is the same for 3.4:

C:\>gem rdoc bundler
Installing RDoc documentation for bundler-1.0.9...
ERROR:  While executing gem ... (Encoding::UndefinedConversionError)
    U+2019 from UTF-8 to CP852

C:\>rdoc --version

rdoc 3.4

[drbrain]

Ok, I can reproduce this, looking into it.

[drbrain]

I have fixed this on trunk for gem unpack bundler; cd bundler-1.0.9; rdoc --encoding CP852. Can you verify?

[voxik]

Yes, it works for me now:

c:\temp>chcp
Aktivní znaková stránka: 852

c:\temp>ruby -v
ruby 1.9.3dev (2011-01-28) [i386-mingw32]

c:\temp>rdoc --version
rdoc 3.5

c:\temp>gem unpack bundler && cd bundler-1.0.9 && rdoc --encoding CP852
Unpacked gem: 'c:/temp/bundler-1.0.9'
Parsing sources...
100% [128/128]  spec/update/source_spec.rb

Generating Darkfish format into c:/temp/bundler-1.0.9/doc...

Files:      128

Classes:     67 ( 56 undocumented)
Modules:     25 ( 23 undocumented)
Constants:   62 ( 22 undocumented)
Attributes:  62 ( 61 undocumented)
Methods:    530 (420 undocumented)

Total:      746 (582 undocumented)
 21.98% documented

Elapsed: 12.3s

c:\temp\bundler-1.0.9>cd ..

c:\temp>gem install bundler
Fetching: bundler-1.0.9.gem (100%)
Successfully installed bundler-1.0.9
1 gem installed
Installing ri documentation for bundler-1.0.9...
Installing RDoc documentation for bundler-1.0.9...

Thank you

[voxik]

Btw, for my curiosity, what is the reason for some encoding conversions? Is the RDoc output encoded by the console encoding or why is that?

[drbrain]

RDoc 3 transcodes to the console encoding by default. You can override the default by setting the RDOCOPT environment variable "--encoding UTF-8" or similar.

RDoc now transcodes to ensure the output documentation does not have corrupt output due to mixed encodings as ruby files may be in any encoding (indicated by the # coding: magic comment).

RDoc defaults to the console's encoding. This was chosen to make it easy to override.

(The bug I fixed in RDoc is due to a possible bug in String#gsub, http://redmine.ruby-lang.org/issues/show/4340)

[voxik]

So is it correct that

a) all files under the Ri and RDoc directories are stored as UTF-8 and converted into console codepage when displaying is requested

or

b) they are prepared in console codepage at that time of generations and later during displaying RDoc doesn't care about encoding?

Because the later would be "disastrous" IMO.

[drbrain]

For the darkfish generator the source .rb files are stored in the encoding indicated in the file from the # coding: comment. If no encoding is present ruby assumes US-ASCII encoding, so RDoc does the same. RDoc transcodes the file when read to the output encoding chosen then adds <meta content="text/html; charset=UTF-8" http-equiv="Content-Type" /> to the output document.

I think ri does the same, but I would have to double check. The bug you reported affects ri less because it doesn't do fancy stuff like attempt to insert © instead of ( c )

So I think b) is the answer.

Why would b) be disastrous? For HTML it should be no problem as the display encoding is indicated. For ri it should be no problem as the output encoding should be the same as the display encoding.

[voxik]

Ok, how RDoc treats source encoding is clear. However I hope you meant that tag with appropriate charset is inserted, otherwise it doesn't make sense to me.

And if b) is true, than it may happen that documentation with wrong encoding may appear on my computer. I don't like that idea. On one side I do not change console encoding that often, usually I don't care to be honest. But there are times during some experiments, bug triaging, whatever, when I switch console encoding. At that point there are 2 problems:

1. I cannot correctly display documentation I have already generated
2. If I generate documentation at that time, it will be probably incorrectly displayed later.

I even don't want to mention that you are possibly loosing information by transcoding of documentation from UTF-8 into local page ... Imagine you have arabic documentation source in UTF-8 and you will try to generate CP852 documentation for me, because I have such console encoding.

From that point of view is better to preserve the source encoding. Any conversion to current codepage is just loss of information.

[drbrain]

Yes, the meta tag is created with the proper charset.

Since CP852 does not have arabic characters you will receive an error similar to the one you reported:

$ ruby -e 'p "الجزيرة نت".encode Encoding::CP852'
-e:1:in `encode': U+0627 from UTF-8 to CP852 (Encoding::UndefinedConversionError)
    from -e:1:in `<main>'

Unfortunately it is difficult to preserve source encoding. A ruby project may have multiple encodings across its files such that a class may have some methods documented in UTF-8 and others in US-ASCII. For HTML mixing encodings is not allowed.

To ensure you always generate usable documentation, export RDOCOPT="--encoding UTF-8"

[voxik]

I would say that UTF-8 should be sensible default (except japanese may be?), I can't help myself. Although I can set environment variable for myself, what about the other users who will be affected?

And also since in theory you could know the documentation encoding, you could warn user that the documentation is not compatible with his console codepage.

[drbrain]

I think UTF-8 could be a sensible default too, but I am hesitant to set it at this point. If there are problems that defaulting to UTF-8 solves I will make it the default.

I will add the documentation encoding to the ri datastore and check it against the locale, that is a good suggestion, thank you!

[voxik]

Great! Thank you for this discussion and your support