Pandas method FutureWarning: frame.append

[jsheunis]

With:

• Python 3.9.7
• pybids 0.15.0
• pandas 0.14.0

I'm getting the following FutureWarning when running bids = BIDSLayout(self.dataset.path, derivatives=True):

...bids/variables/entities.py:245: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.

pointing to this line.

Should I submit a PR to replace that, or are there reasons why this should not be done?

[effigies]

My impression was that they didn't make all the .append() options available to pandas.concat (I'd need to go digging for the Pandas thread where people were complaining about this deprecation), so there wasn't an easy way to replicate the semantics.

That said, I'm no pandas expert. If you propose a change that doesn't break anything, I'm fine with it.

[jsheunis]

Based on a quick check, changing the line:

self.index = self.index.append(node_row, ignore_index=True)

to

self.index = pd.concat([self.index, node_row], ignore_index=True)

does indeed break stuff when running pytest locally:

pytest bids/variables/tests/test_entities.py
================================================================================ test session starts ================================================================================
platform darwin -- Python 3.9.11, pytest-7.1.1, pluggy-1.0.0
rootdir: /Users/jsheunis/Documents/psyinf/pybids
collected 5 items

bids/variables/tests/test_entities.py .FFFF                                                                                                                                   [100%]

===================================================================================== FAILURES ======================================================================================
______________________________________________________________________________ test_get_or_create_node ______________________________________________________________________________

layout1 = BIDS Layout: ...f/pybids/bids/tests/data/ds005 | Subjects: 16 | Sessions: 0 | Runs: 48

    def test_get_or_create_node(layout1):
        img = layout1.get(subject='01', task='mixedgamblestask', suffix='bold',
                          run=1, return_type='obj')[0]
        index = NodeIndex()

        entities = {'subject': '01', 'session': 1}
        sess = index.get_or_create_node('session', entities)
        assert sess.__class__ == Node

        sess2 = index.get_or_create_node('session', entities)
        assert sess2 == sess

        run = index.get_or_create_node('run', img.entities,
                                       image_file=img.filename, duration=480,
                                       repetition_time=2,
                                       n_vols=480/2)
>       assert run.__class__ == RunNode
E       AssertionError: assert <class 'bids.variables.entities.Node'> == RunNode
E        +  where <class 'bids.variables.entities.Node'> = <bids.variables.entities.Node object at 0x7fd1ca1aa040>.__class__

bids/variables/tests/test_entities.py:50: AssertionError
__________________________________________________________________________________ test_get_nodes ___________________________________________________________________________________

layout1 = BIDS Layout: ...f/pybids/bids/tests/data/ds005 | Subjects: 16 | Sessions: 0 | Runs: 48

    def test_get_nodes(layout1):
        index = load_variables(layout1, scan_length=480)
        nodes = index.get_nodes('session')
>       assert len(nodes) == 0
E       assert 1 == 0
E        +  where 1 = len([<bids.variables.entities.RunNode object at 0x7fd1ca0fca90>])

bids/variables/tests/test_entities.py:58: AssertionError
____________________________________________________________________________ test_get_collections_merged ____________________________________________________________________________

layout1 = BIDS Layout: ...f/pybids/bids/tests/data/ds005 | Subjects: 16 | Sessions: 0 | Runs: 48

    def test_get_collections_merged(layout1):
        dataset = load_variables(layout1, scan_length=480)
>       collection = dataset.get_collections('run', merge=True)

bids/variables/tests/test_entities.py:69:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
bids/variables/entities.py:149: in get_collections
    vs = clc.BIDSRunVariableCollection(vs, sampling_rate)
bids/variables/collections.py:334: in __init__
    super(BIDSRunVariableCollection, self).__init__(variables)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <bids.variables.collections.BIDSRunVariableCollection object at 0x7fd1ca14ad00>
variables = [<bids.variables.variables.SparseRunVariable object at 0x7fd1ca117f10>, <bids.variables.variables.SparseRunVariable ob...parseRunVariable object at 0x7fd1ca1cf880>, <bids.variables.variables.SparseRunVariable object at 0x7fd1ca18d6d0>, ...]
name = None

    def __init__(self, variables, name=None):

        self.name = name

        if not variables:
            raise ValueError("No variables were provided")
        SOURCE_TO_LEVEL = {
            "events": "run",
            "physio": "run",
            "stim": "run",
            "regressors": "run",
            "scans": "session",
            "sessions": "subject",
            "participants": "dataset",
        }
        var_levels = set(
            [
                SOURCE_TO_LEVEL[v.source] if v.source in SOURCE_TO_LEVEL else v.source
                for v in variables
            ]
        )

        # TODO: relax this requirement & allow implicit merging between levels
        if len(var_levels) > 1:
>           raise ValueError(
                "A Collection cannot be initialized from "
                "variables at more than one level of analysis. "
                "Levels found in input variables: %s" % var_levels
E               ValueError: A Collection cannot be initialized from variables at more than one level of analysis. Levels found in input variables: {'run', 'dataset'}

bids/variables/collections.py:65: ValueError
___________________________________________________________________________ test_get_collections_unmerged ___________________________________________________________________________

layout2 = BIDS Layout: .../pybids/bids/tests/data/7t_trt | Subjects: 10 | Sessions: 20 | Runs: 20

    def test_get_collections_unmerged(layout2):
        dataset = load_variables(layout2, types=['sessions'], scan_length=480)
        colls = dataset.get_collections('subject', merge=False)
>       assert len(colls) == 10
E       assert 1 == 10
E        +  where 1 = len([<bids.variables.collections.BIDSVariableCollection object at 0x7fd1cacc5a60>])

bids/variables/tests/test_entities.py:81: AssertionError
============================================================================== short test summary info ==============================================================================
FAILED bids/variables/tests/test_entities.py::test_get_or_create_node - AssertionError: assert <class 'bids.variables.entities.Node'> == RunNode
FAILED bids/variables/tests/test_entities.py::test_get_nodes - assert 1 == 0
FAILED bids/variables/tests/test_entities.py::test_get_collections_merged - ValueError: A Collection cannot be initialized from variables at more than one level of analysis. Leve...
FAILED bids/variables/tests/test_entities.py::test_get_collections_unmerged - assert 1 == 10
=========================================================================== 4 failed, 1 passed in 16.46s ============================================================================

[effigies]

Got curious to fiddle with this again. It requires an ugly hack (DataFrame(Series(entities)).T), but to be fair, it seems we were depending on some ugly implicit coercions.