Circular import while using pydicom with multiprocessing

[romainhild-ge]

Describe the bug
When importing pydicom with a multi process program, a circular import happens.
The stack trace is the following:

multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 243, in serve_client
    request = recv()
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/home/romain/Documents/.env/lib/python3.8/site-packages/pydicom/__init__.py", line 35, in <module>
    from pydicom.filewriter import dcmwrite, write_file
  File "/home/romain/Documents/.env/lib/python3.8/site-packages/pydicom/filewriter.py", line 23, in <module>
    from pydicom.values import convert_numbers
  File "/home/romain/Documents/.env/lib/python3.8/site-packages/pydicom/values.py", line 621, in <module>
    ) -> Union[pydicom.uid.UID, SequenceType[pydicom.uid.UID]]:
AttributeError: partially initialized module 'pydicom' has no attribute 'uid' (most likely due to a circular import)
---------------------------------------------------------------------------

The error occurs in multiple process, sometimes with the following:

  File "/home/romain/Documents/.env/lib/python3.8/site-packages/pydicom/dataelem.py", line 31, in <module>
    from pydicom.valuerep import PersonName
ImportError: cannot import name 'PersonName' from partially initialized module 'pydicom.valuerep' (most likely due to a circular import)

Expected behavior
The import should take place without error.

Steps To Reproduce
The bug is hard to reproduce, I'm using a library that I can't share, but we only do a import pydicom.
I have not been able to reproduce the bug in a simple way for it to happen every time, it can be random or occur every time when changing an unrelated file.
If anyone has an idea to reproduce it, I'm willing to try.

Your environment

module	version
platform	Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.29
Python	3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0]
pydicom	2.1.2
gdcm	module not found
jpeg_ls	module not found
numpy	1.22.3
PIL	9.0.1

[darcymason]

This is difficult to address without a reproducible bit of code. It seems that it only happens with multiprocessing, because we have no reports of circular imports otherwise, and the test suite has very high coverage. Howeer, pydicom does have a couple/few imports that are buried inside functions to avoid circular import; perhaps those don't work under multiprocessing for some reason.

Your environment says pydicom 2.1. Have you tried with pydicom 2.3? I'm not aware of any specific changes that might affect this, but just in case.

[romainhild-ge]

We went back to pydicom 2.1 because the circular import happened way more frequently with 2.3.
I agree that reproducing the bug would be great but it is really weird. With a version of my code, the bug was happening every time I launched my app, and then I just added one blank line in a file, save it and rerun my app, the bug was gone...

I'll keep trying to reproduce the bug in a simple way.

[romainhild-ge]

It seems the issue came from the serialization when passing pydicom objects in multiprocessing queues.
We rearrange our code so that we cast every object to built-in types and we no longer have the issue.

[mrbean-bremen]

Do you still think that this is something that pydicom shall address?

[romainhild-ge]

It would be great if pydicom could be used in multiprocessing context and passing values into queues.
At least, maybe put a warning somewhere in the doc.

[mrbean-bremen]

Do you now have a reproducible example so we can investigate this?

[romainhild-ge]

No I could not find a stable way to reproduce this bug, one engineer had the issue at each run of our app, so we were able to confirm that it improved things, but we could not find a simple reproductible example.