sqlalchemy crash

[bpinsard]

I have a weird crash when running this simple query, but just in the code I have written, not when running line by line the same in ipython:

query_entities={'datatype': 'anat', 'extension': '.json', 'flip': '1', 'mt': 'on', 'session': '001', 'suffix': 'MTS'}
all_subjects_jsons = bids_layout.get(**query_entities)

traceback

Traceback (most recent call last):
  File "/home/basile/.virtualenvs/my_env/bin/my_env", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/basile/data/src/my_env/src/my_env/cli/run.py", line 30, in main
    initialize(layout, session_uniform=args.uniform_session)
  File "/home/basile/data/src/my_env/src/my_env/cli/init.py", line 46, in initialize
    all_subjects_jsons = bids_layout.get(**query_entities)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/basile/.virtualenvs/my_env/lib/python3.12/site-packages/bids/layout/layout.py", line 653, in get
    entities = self.get_entities()
               ^^^^^^^^^^^^^^^^^^^
  File "/home/basile/.virtualenvs/my_env/lib/python3.12/site-packages/bids/layout/layout.py", line 403, in get_entities
    results = query.all()
              ^^^^^^^^^^^
  File "/home/basile/.virtualenvs/my_env/lib/python3.12/site-packages/sqlalchemy/orm/query.py", line 2673, in all
    return self._iter().all()  # type: ignore
           ^^^^^^^^^^^^
  File "/home/basile/.virtualenvs/my_env/lib/python3.12/site-packages/sqlalchemy/orm/query.py", line 2827, in _iter
    result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
                                                  ^^^^^^^^^^^^^^^^^^^^^
  File "/home/basile/.virtualenvs/my_env/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2306, in execute
    return self._execute_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/basile/.virtualenvs/my_env/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2170, in _execute_internal
    ) = compile_state_cls.orm_pre_session_exec(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/basile/.virtualenvs/my_env/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 549, in orm_pre_session_exec
    session._autoflush()
  File "/home/basile/.virtualenvs/my_env/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2994, in _autoflush
    self.flush()
  File "/home/basile/.virtualenvs/my_env/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4296, in flush
    self._flush(objects)
  File "/home/basile/.virtualenvs/my_env/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4431, in _flush
    with util.safe_reraise():
  File "/home/basile/.virtualenvs/my_env/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/basile/.virtualenvs/my_env/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4392, in _flush
    flush_context.execute()
  File "/home/basile/.virtualenvs/my_env/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
    rec.execute(self)
  File "/home/basile/.virtualenvs/my_env/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 591, in execute
    self.dependency_processor.process_saves(uow, states)
  File "/home/basile/.virtualenvs/my_env/lib/python3.12/site-packages/sqlalchemy/orm/dependency.py", line 604, in process_saves
    self._synchronize(
  File "/home/basile/.virtualenvs/my_env/lib/python3.12/site-packages/sqlalchemy/orm/dependency.py", line 625, in _synchronize
    sync.clear(dest, self.mapper, self.prop.synchronize_pairs)
  File "/home/basile/.virtualenvs/my_env/lib/python3.12/site-packages/sqlalchemy/orm/sync.py", line 88, in clear
    raise AssertionError(
AssertionError: Dependency rule on column 'files.path' tried to blank-out primary key column 'tags.file_path' on instance '<Tag at 0x7bd5db505100>'

[edit]
oups, went out too fast.
So I have no idea what is causing this only in the code. Any idea? Thanks!

[bpinsard]

When trying to run with pdb, you cannot do much because sqlachemy is left in a state that only gets you sqlalchemy.exc.PendingRollbackError if you try to investigate.

[effigies]

Does changing versions of sqlalchemy affect whether this reproduces? I note that you're using Python 3.12, which we are not yet testing on (I'm working on that now).

[bpinsard]

Oh that might be it, however the same few lines in ipython on the same version works. Unless the previous query modify the BIDSLayout that put it in an unstable state. I will give a try on older python.

[bpinsard]

I have the same issue in python3.10, I was using 3.12 for some new typing syntax and will post if I trigger any issue with pybids , so far none.

Figured out that I was modifying some BIDSFile entities in-place at some point, it seems to be what caused the issue.
Thanks for your help.

BTW: I am trying to draft a small tools for continuous BIDS protocol compliance (see: https://neurostars.org/t/continous-bids-multi-site-protocol-compliance/29204/2 ), any feedback is welcome. :)

[effigies]

One thing that might be worth looking into if you're just using the layout is https://github.com/cmi-dair/bids2table. It's extremely fast and not SQL-based.