Problems with python in new version from 2021-05-04

[rikardn]

Description

This workflow started to fail after the latest update of the virtual environment

https://github.com/pharmpy/pharmpy/actions/runs/817030678

The issue is with python and this error shows up multiple times:

Python 3.8.10 /home/runner/work/pharmpy/pharmpy/.tox/py38/bin/python
  /opt/hostedtoolcache/Python/3.8.10/x64/lib/python3.8/pkgutil.py:415: in get_importer
      importer = sys.path_importer_cache[path_item]
  E   KeyError: PosixPath('/home/runner/work/pharmpy/pharmpy/.tox/py38/lib/python3.8/site-packages/pharmpy/plugins')
  
  During handling of the above exception, another exception occurred:
  tests/nonmem/conftest.py:43: in pheno
      model = pharmpy.Model(pheno_path)
  .tox/py38/lib/python3.8/site-packages/pharmpy/model_factory.py:27: in Model
      model_class = detect_model(src)
  .tox/py38/lib/python3.8/site-packages/pharmpy/plugins/utils.py:20: in detect_model
      plugins = load_plugins()
  .tox/py38/lib/python3.8/site-packages/pharmpy/plugins/utils.py:41: in load_plugins
      for _, modname, ispkg in pkgutil.iter_modules([plugin_path], 'pharmpy.plugins.'):
  /opt/hostedtoolcache/Python/3.8.10/x64/lib/python3.8/pkgutil.py:129: in iter_modules
      for i in importers:
  /opt/hostedtoolcache/Python/3.8.10/x64/lib/python3.8/pkgutil.py:419: in get_importer
      importer = path_hook(path_item)
  E   AttributeError: 'PosixPath' object has no attribute 'startswith'

Nothing was changed in the call to pkgutil and the error message is between internal calls of the standard library.

Here is the last run without these issues: https://github.com/pharmpy/pharmpy/actions/runs/816357539

Area for Triage:
Python

Question, Bug, or Feature?:
Bug

Virtual environments affected

• Ubuntu 16.04
• Ubuntu 18.04
• Ubuntu 20.04
• macOS 10.15
• macOS 11.0
• Windows Server 2016 R2
• Windows Server 2019

Image version
20210504.1

Expected behavior
All tests should pass and the exception should not appear. Running locally works fine.

Actual behavior
A python exception is raised.

Repro steps
Failing workflow: https://github.com/pharmpy/pharmpy/actions/runs/817030678

[miketimofeev]

Hi @rikardn!
It can be the issue not with the image but with the updated Python version 3.8.10, which was released a few days ago. Could you please try to specify the exact Python version 3.8.9 in your workflow and check if the issue persists?

[miketimofeev]

@rikardn and same for 3.9 — it was updated from 3.9.4 to 3.9.5 the same day as 3.8

[rikardn]

Thanks @miketimofeev . This is indeed the problem. I changed from python "3.8" to "3.8.9" and all tests pass again. Still failing with python 3.9 so I will test also 3.9.4.

[rikardn]

Python 3.9.4 also works.

So going back to python 3.9.4 and python 3.8.9 resolved the issues.

The passing workflow:
https://github.com/pharmpy/pharmpy/actions/runs/817415441

[miketimofeev]

I suppose the identical changes, which cause the issue, were introduced both in 3.8.10 and 3.9.5. Could you try to run the same tests locally in either 3.8.10 or 3.9.5 to confirm the theory?

[rikardn]

I built python 3.8.10 from source and ran the tests with that locally. Now I get the same errors as with Github Actions. The problem indeed seems to be connected to the new python versions. I'll do some debugging to see if I can pinpoint the exact cause.

[rikardn]

Ok, I can confirm that it is a regression in python 3.8.10

This snippet:

import pkgutil
from pathlib import Path


for _, modname, ispkg in pkgutil.iter_modules([Path("/home/mypath")], 'somepackage.somesubpackage'):
        print(modname, ispkg)

runs fine using python 3.8.9, but not with python 3.8.10. Changing from Path to a string makes it work.

I'll check if it was reported on the python issue tracker.

[rikardn]

I filed this:

https://bugs.python.org/issue44061

Since this issue is not directly related to the github actions virtual environment you can close it if you like.

[miketimofeev]

@rikardn thanks for the confirmation! I'm going to close the issue as it's not related to the images.
Please feel free to contact us if you have any concerns.