Modifying lists and dictionaries used as dictionary variable values stopped working in RF 3.0.3

[agagalski]

Hi!

Robot Framework version: 3.0.3
Python interpreter type (Python, Jython or IronPython) and version: Python 2.7.12
Operating system name and version: Windows 10

SSCCE - see code example and output below
Code:

*** Settings ***
Documentation     Error reproduction
...
Library           Collections

*** Variables ***


*** Test Cases ***

Main
    ${list} =    Create List    aaaa    bbbb    cccc    dddd
    ${dict} =    Create Dictionary
    ...                      list    ${list}
    Log    ${dict}    WARN
    ${new_item} =    Set Variable    eeee
    Log    \${new_item}: ${new_item}    WARN
    Log    \&{dict}[list] before mod: &{dict}[list]    WARN
    Append To List    &{dict}[list]    ${new_item}
    Log    \&{dict}[list] after mod: &{dict}[list]    WARN

Output:

c:\Projects\Robot_sandbox>pip install --upgrade robotframework==3.0.2
Requirement already up-to-date: robotframework==3.0.2 in c:\python27\lib\site-packages
You are using pip version 9.0.1, however version 10.0.0 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

c:\Projects\Robot_sandbox>robot error_list_in_dict.robot
==============================================================================
Error List In Dict :: Error reproduction
==============================================================================
[ WARN ] {u'list': [u'aaaa', u'bbbb', u'cccc', u'dddd']}
[ WARN ] ${new_item}: eeee
[ WARN ] &{dict}[list] before mod: [u'aaaa', u'bbbb', u'cccc', u'dddd']
[ WARN ] &{dict}[list] after mod: [u'aaaa', u'bbbb', u'cccc', u'dddd', u'eeee']
Main                                                                  | PASS |
------------------------------------------------------------------------------
Error List In Dict :: Error reproduction                              | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  c:\Projects\Robot_sandbox\output.xml
Log:     c:\Projects\Robot_sandbox\log.html
Report:  c:\Projects\Robot_sandbox\report.html

c:\Projects\Robot_sandbox>pip install --upgrade robotframework==3.0.3
Collecting robotframework==3.0.3
Installing collected packages: robotframework
  Found existing installation: robotframework 3.0.2
    Uninstalling robotframework-3.0.2:
      Successfully uninstalled robotframework-3.0.2
Successfully installed robotframework-3.0.3
You are using pip version 9.0.1, however version 10.0.0 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

c:\Projects\Robot_sandbox>robot error_list_in_dict.robot
==============================================================================
Error List In Dict :: Error reproduction
==============================================================================
[ WARN ] {u'list': [u'aaaa', u'bbbb', u'cccc', u'dddd']}
[ WARN ] ${new_item}: eeee
[ WARN ] &{dict}[list] before mod: [u'aaaa', u'bbbb', u'cccc', u'dddd']
[ WARN ] &{dict}[list] after mod: [u'aaaa', u'bbbb', u'cccc', u'dddd']
Main                                                                  | PASS |
------------------------------------------------------------------------------
Error List In Dict :: Error reproduction                              | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  c:\Projects\Robot_sandbox\output.xml
Log:     c:\Projects\Robot_sandbox\log.html
Report:  c:\Projects\Robot_sandbox\report.html

Expected: new item is added to list embedded into dictionary (works in RF3.0.2)
Actual: after upgrade to RF3.0.3 new item is not added to list embedded into dictionary

[pekkaklarck]

Quick comments about this issue:

• This regression is caused by change to make the internal DotDict class recursively "dot-accessible"
  (Dictionary variables should support attribute access with nested dictionaries #2600). I will later explain the root cause in more detail.

• A workaround is using the embedded argument syntax like ${dict['list']} instead of accessing dictionary variable item like &{dict}[list].

• The syntax related to accessing dictionary and list items is going to change in the future (Accessing nested list and dictionary variable items like ${var}[key][0] #2601) and the &{dict}[list] syntax would need to be replaced with ${dict}[list].

The main question now is, is this severe enough problem that it would warrant RF 3.0.4 release or could the fix wait until RF 3.1 in June. This is a valid bug report, but because there's a workaround and the used syntax is going to change anyway, I'm not sure is 3.0.4 needed.

[pekkaklarck]

Issue #2814 has same root cause as this one and is even more severe. RF 3.0.4 where these problems are fixed is clearly needed.

[pekkaklarck]

This problem is due to DotDicts nowadays converting also internal dicts into DotDicts to allow nested "dotted access" like ${dict.child.key}. The code that goes through the values converts all dicts to DotDicts, but also processes all lists and list-like objects. As a result lists are recreated and all list-like objects are converted to lists. Lists being recreated caused this issue because added to them are not actually added to the original lists but only to their copies. All list-like objects being converted to lists is the root cause of #2814.

[pekkaklarck]

A fix to this problem has now been released as part of RF 3.0.4 rc 1. You can find more information about the release in the release notes and you can install the release simply by running this command:

pip install --pre --upgrade robotframework

Please test the release to ensure we get all possible problems fixed before the final release targeted for Tuesday!