Leading and internal spaces should be preserved in documentation

[kerol2r20]

Hi,
I found that .. code::robotframework directives cannot work well in resource file format when I set documen format as reStructuredText format.

*** Keywords ***
Simple keyword
    [Documentation]    
    ...    .. code:: robotframework
    ...
    ...        *** Test Cases ***
    ...        Simple test case
    ...            Simple keyword

docutils ouput <string>:1: (ERROR/3) Content block expected for the "code" directive; none found. error message.

I had tried to find out why it happen and found that libdoc visit each token and lost white space token.
After parse_resource_file function invoked, the doc of keyword remain like the following.
This is not a valid format for docutils.

.. code:: robotframework

*** Test Cases ***
Simple test case
Simple keyword

May it possible to reserve white space when using reStructuredText as document format?

[pekkaklarck]

The same problem occurs with all documentation formatting requiring leading spaces. This is due to the parser considering two or more spaces a separator between data tokens and thus

    ...            Simple keyword

is exactly the same as

    ...  Simple keyword

With documentation it would make sense to see what's the separator in the first row and use that with the remaining rows as well. The actual separator token, containing all those spaces, has already been thrown out when the documentation is handled otherwise, but the data tokens have column offsets that would allow adjusting rows in cases like in the example in the description. It would be harder to adjust lines in cases where the documentation starts already on the same row as the setting name, though:

*** Keywords ***
Simple keyword
    [Documentation]    The doc starts already here.
    ...
    ...    .. code:: robotframework
    ...
    ...        *** Test Cases ***
    ...        Simple test case
    ...            Simple keyword

I guess we could try handling the above example so that if the first documentation row has bigger column offset than the second, we use the column offset on the second row as the base offset. That starts to get a bit complicated but could possibly work. Are you @kerol2r20 interested to experiment with that?

[pekkaklarck]

The code for construction documentation from the underlying tokens is here. Incidentally, it was recently modified to fix #4670.

If this code is modified, it should be enhanced to handle also cases like

[Documentation]    First part            second part

where the spaces between documentation parts are collapsed into a single space. Also in this case the spaces between documentation parts are considered a separator and thrown away and Robot just joints these documentation parts together with a space as a separator. If we'd look at column offsets, we could add a correct amount of spaces there instead.

[pekkaklarck]

One challenge with looking at column offsets is that if a Documentation statement is generated programmatically using from_params or otherwise, the underlying tokens may not have line numbers or column offsets set. I guess we could simply ignore column offsets in that case and the results would be exactly the same as nowadays.

[pekkaklarck]

As a workaround for this issue you can escape spaces with a backslash like in the example below. It looks horrible but works:

*** Keywords ***
Simple keyword
    [Documentation]
    ...    .. code:: robotframework
    ...
    ...    \ \ *** Test Cases ***
    ...    \ \ Simple test case
    ...    \ \ \ \ \ \ Simple keyword

In documentation used during execution you could also use e.g. ${SPACE * 2} instead of \ \ , but Libdoc doesn't replace variables so it won't work with it. That possibly could be changed but it wouldn't look that good in the source file anyway.

[pekkaklarck]

I decided to prototype taking the base column offset into account and it wasn't too complicated. It broke some of our acceptance tests, but fixing them isn't too hard and it also makes sure the functionality works as expected.

[kerol2r20]

Excuse me. I didn't readlly understand what about "taking the base column offset into account ".
Is this mean that preserving all white space just after the first symbol parsed?

Like

*** Keywords ***
Simple keyword
    [Documentation]
    ...    .. code:: robotframework
#       ^^^ preserved all lead white space

[pekkaklarck]

The base column offset means the smallest offset a documentation token has. Documentation tokens with higher offset get spaces added to their value to match the base offset.. For example, with

*** Keywords ***
Simple keyword
    [Documentation]    
    ...    .. code:: robotframework    # This row defines base offset.
    ...
    ...        *** Test Cases ***
    ...        Simple test case
    ...            Simple keyword

the base offset is 11 because .. code starts from that positiion. *** Test Cases *** has offset 15 which means we'll add four (15-11) spaces to it. Same with Simple test case. Simple keyword has offset 19 so it needs eight leading spaces.

If there's a row that has smaller offset than the base offset, the base offset is reset.It solves this case:

*** Keywords ***
Simple keyword
    [Documentation]    Documentation starts.    # This row sets the initial base offset to 23
    ...                                         # Empty rows don't affect base offset.   
    ...    .. code:: robotframework             # This row resets base offset to 11.
    ...                                         # Remaining rows are handled like above.
    ...        *** Test Cases ***
    ...        Simple test case
    ...            Simple keyword

This may sound pretty complicated but it seems to work pretty well. Probably the biggest task related to this was updating documentation. Explaining this behavior wasn't too big a task, I didn't explain the "base offset algorithm" at all, but the related docs about test and suite documentation needed some cleanup and enhancements that took time. Fixing all tests that broke took some time too but now this ought to be ready.