Inconsistent line endings in /wiki/data/default/data/: \r\n vs. \n

[RogerHaase]

Per https://www.w3.org/TR/html401/interact/forms.html#h-17.13.4: "As with all MIME transmissions, "CR LF" (i.e., `%0D%0A') is used to separate lines of data." e.g. lines of data pulled from a form's textarea are separated by \r\n.

If a text item is edited and saved, the saved file in /wiki/data/default/data/ contains \r\n as line separators.

I believe the above is the desired state.

However, the current "moin get-item ..." incorrectly replaces \r\n with \n in the .data output. As get-item is used to extract the items in /contrib/sample/, these files have incorrect \n line endings. These incorrect line endings then persist when the sample data is loaded on a wiki.

After the py3 conversion, one adverse result is that the code near line 40 in /scripts/main/modify_item.py:

data = rev.data.read().replace('\r\n', '\n')

fails. A solution is to remove the .replace('\r\n', '\n'). The moin item-put command accepts the \r\n line endings as input and retains \r\n when storing the item.

As for the meta data, on windows the meta data is stored with \r\n line endings. *nix systems likely will save data with \n line endings (not yet tested). Both are acceptable as moin item-put stores meta data in Jason format; all line endings are removed.

TODO:

• fix bytes vs. string traceback on "moin item-get..." #862, correcting line endings on item-get, item-put
• recreate all /contrib/sample items with correct \r\n line endings
• retest /utils/diff3.py creates incorrect output after edit conflict

[ThomasWaldmann]

I guess there are 2 sane ways to deal with this:

A) consider data as binary, ever, even for text/* and not change it.

So it would just get / put item data "as is". For "put" this would have the problem that it would be possible to put \n-only text items. Maybe this could be dealt with on the validation layer by just rejecting such data.

B) for text/* items:

Writing to file:

• transform \r\n to \n (storage to internal)
• transform \n to platform dependent line endings when saving (internal to disk file) by using a file opened in text mode

Reading from file:

• open file in text mode -> \n internal representation
• \n internal -> \r\n in storage

The problem here is that there is some text stuff that is not text/* mimetype (but image or application).

[RogerHaase]

Will change item-put to output \r\n when input is \n

Will change item-get to output platform-dependent line endings, making it consistent with recent change to item-get for meta files.