Unable to use neptune logger after a version upgrade of pytorch lightning

[joshua840]

🐛 Bug

Unable to initialize Neptune logger after version upgrade of pytorch lightning from 1.6.5 to 1.7.0.

 File "/PATH/test2.py", line 14, in <module>
    cli_main()
  File "/PATH/test2.py", line 11, in cli_main
    logger = NeptuneLogger(run=run, log_model_checkpoints=False)
  File "/MYENV/envs/torch_1.12/lib/python3.10/site-packages/pytorch_lightning/loggers/neptune.py", line 278, in __init__
    self._verify_input_arguments(api_key, project, name, run, neptune_run_kwargs)
  File "/MYENV/envs/torch_1.12/lib/python3.10/site-packages/pytorch_lightning/loggers/neptune.py", line 377, in _verify_input_arguments
    if run is not None and not isinstance(run, Run):
TypeError: isinstance() arg 2 must be a type, a tuple of types, or a union

In my case, the value of Run is None, which is determined in the following code.

https://github.com/Lightning-AI/lightning/blob/14b36f8109db63f4c9ed2480734ce5a0dbbc71b9/src/pytorch_lightning/loggers/neptune.py#L43

If I downgrade lightning to 1.6.5, the error does not occur.

To Reproduce

Code

import pytorch_lightning as pl
from pytorch_lightning.loggers.neptune import NeptuneLogger
import neptune.new as neptune

# project_name should be "WORKSPACE_NAME/PROJECT_NAME"
PROJECT_NAME = 'YOUR_NEPTUNE_PROJECT'
API_KEY = 'YOUR_NEPTUNE_API_KEY'

def cli_main():
    run = neptune.init(api_token=API_KEY, project=PROJECT_NAME, capture_stdout=False)
    logger = NeptuneLogger(run=run, log_model_checkpoints=False)

if __name__ == "__main__":
    cli_main()

Expected behavior

Test code should works regardless of the PyTorch lightning version.

Environment

Please save the following environment information and create env by this command.

conda env create --file ENV.yml

Conda environment

name: torch_1.12
channels:
  - pytorch
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - absl-py=1.2.0=pyhd8ed1ab_0
  - aiohttp=3.8.1=py310h5764c6d_1
  - aiosignal=1.2.0=pyhd8ed1ab_0
  - async-timeout=4.0.2=pyhd8ed1ab_0
  - attrs=21.4.0=pyhd3eb1b0_0
  - blas=1.0=mkl
  - blinker=1.4=py310h06a4308_0
  - boto3=1.24.28=py310h06a4308_0
  - botocore=1.27.28=py310h06a4308_0
  - bottleneck=1.3.5=py310ha9d4c09_0
  - bravado=11.0.3=pyhd8ed1ab_0
  - bravado-core=5.17.0=pyh9f0ad1d_0
  - brotlipy=0.7.0=py310h7f8727e_1002
  - bzip2=1.0.8=h7b6447c_0
  - c-ares=1.18.1=h7f98852_0
  - ca-certificates=2022.6.15=ha878542_0
  - cachetools=5.0.0=pyhd8ed1ab_0
  - certifi=2022.6.15=py310hff52083_0
  - cffi=1.15.1=py310h74dc2b5_0
  - charset-normalizer=2.0.4=pyhd3eb1b0_0
  - click=8.0.4=py310h06a4308_0
  - colorama=0.4.5=pyhd8ed1ab_0
  - cryptography=37.0.1=py310h9ce1e76_0
  - cudatoolkit=11.3.1=h2bc3f7f_2
  - dataclasses=0.8=pyh6d0b6a4_7
  - ffmpeg=4.2.2=h20bf706_0
  - freetype=2.11.0=h70c0345_0
  - frozenlist=1.2.0=py310h7f8727e_1
  - fsspec=2022.7.1=pyhd8ed1ab_0
  - future=0.18.2=py310h06a4308_1
  - giflib=5.2.1=h7b6447c_0
  - gitdb=4.0.7=pyhd3eb1b0_0
  - gitpython=3.1.18=pyhd3eb1b0_1
  - gmp=6.2.1=h295c915_3
  - gnutls=3.6.15=he1e5248_0
  - google-auth=2.10.0=pyh6c4a22f_0
  - google-auth-oauthlib=0.4.6=pyhd8ed1ab_0
  - grpcio=1.42.0=py310hce63b2e_0
  - idna=3.3=pyhd3eb1b0_0
  - importlib-metadata=4.11.4=py310hff52083_0
  - intel-openmp=2021.4.0=h06a4308_3561
  - jmespath=0.10.0=pyhd3eb1b0_0
  - jpeg=9e=h7f8727e_0
  - jsonref=0.2=py_0
  - jsonschema=4.4.0=py310h06a4308_0
  - lame=3.100=h7b6447c_0
  - lcms2=2.12=h3be6417_0
  - ld_impl_linux-64=2.38=h1181459_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=11.2.0=h1234567_1
  - libgomp=11.2.0=h1234567_1
  - libidn2=2.3.2=h7f8727e_0
  - libopus=1.3.1=h7b6447c_0
  - libpng=1.6.37=hbc83047_0
  - libprotobuf=3.19.1=h4ff587b_0
  - libstdcxx-ng=11.2.0=h1234567_1
  - libtasn1=4.16.0=h27cfd23_0
  - libtiff=4.2.0=h2818925_1
  - libunistring=0.9.10=h27cfd23_0
  - libuuid=1.0.3=h7f8727e_2
  - libvpx=1.7.0=h439df22_0
  - libwebp=1.2.2=h55f646e_0
  - libwebp-base=1.2.2=h7f8727e_0
  - lz4-c=1.9.3=h295c915_1
  - markdown=3.4.1=pyhd8ed1ab_0
  - markupsafe=2.1.1=py310h5764c6d_1
  - mkl=2021.4.0=h06a4308_640
  - mkl-service=2.4.0=py310h7f8727e_0
  - mkl_fft=1.3.1=py310hd6ae3a3_0
  - mkl_random=1.2.2=py310h00e6091_0
  - monai=0.9.1=pyhd8ed1ab_0
  - monotonic=1.5=py_0
  - msgpack-python=1.0.3=py310hd09550d_0
  - multidict=6.0.2=py310h5764c6d_1
  - ncurses=6.3=h5eee18b_3
  - neptune-client=0.16.4=pyhd8ed1ab_0
  - nettle=3.7.3=hbbd107a_1
  - numexpr=2.8.3=py310hcea2de6_0
  - numpy=1.23.1=py310h1794996_0
  - numpy-base=1.23.1=py310hcba007f_0
  - oauthlib=3.2.0=pyhd3eb1b0_1
  - openh264=2.1.1=h4ff587b_0
  - openssl=1.1.1q=h7f8727e_0
  - packaging=21.3=pyhd3eb1b0_0
  - pandas=1.4.3=py310h6a678d5_0
  - pillow=9.2.0=py310hace64e9_1
  - pip=22.1.2=py310h06a4308_0
  - protobuf=3.19.1=py310h295c915_0
  - psutil=5.9.0=py310h5eee18b_0
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.7=py_0
  - pycparser=2.21=pyhd3eb1b0_0
  - pydeprecate=0.3.2=pyhd8ed1ab_0
  - pyjwt=2.4.0=py310h06a4308_0
  - pyopenssl=22.0.0=pyhd3eb1b0_0
  - pyparsing=3.0.4=pyhd3eb1b0_0
  - pyrsistent=0.18.0=py310h7f8727e_0
  - pysocks=1.7.1=py310h06a4308_0
  - python=3.10.4=h12debd9_0
  - python-dateutil=2.8.2=pyhd3eb1b0_0
  - python_abi=3.10=2_cp310
  - pytorch=1.12.1=py3.10_cuda11.3_cudnn8.3.2_0
  - pytorch-lightning=1.7.0=pyhd8ed1ab_0
  - pytorch-mutex=1.0=cuda
  - pytz=2022.1=py310h06a4308_0
  - pyu2f=0.1.5=pyhd8ed1ab_0
  - readline=8.1.2=h7f8727e_1
  - requests=2.28.1=py310h06a4308_0
  - requests-oauthlib=1.3.0=py_0
  - rsa=4.9=pyhd8ed1ab_0
  - s3transfer=0.6.0=py310h06a4308_0
  - setuptools=59.5.0=py310hff52083_0
  - simplejson=3.17.6=py310h7f8727e_0
  - six=1.16.0=pyhd3eb1b0_1
  - smmap=4.0.0=pyhd3eb1b0_0
  - sqlite=3.39.2=h5082296_0
  - swagger-spec-validator=2.7.4=pyhd8ed1ab_0
  - tensorboard=2.9.1=pyhd8ed1ab_0
  - tensorboard-data-server=0.6.0=py310h597c629_2
  - tensorboard-plugin-wit=1.8.1=pyhd8ed1ab_0
  - tk=8.6.12=h1ccaba5_0
  - torchaudio=0.12.1=py310_cu113
  - torchmetrics=0.9.3=pyhd8ed1ab_0
  - torchvision=0.13.1=py310_cu113
  - tqdm=4.64.0=pyhd8ed1ab_0
  - typing-extensions=4.3.0=py310h06a4308_0
  - typing_extensions=4.3.0=py310h06a4308_0
  - tzdata=2022a=hda174b7_0
  - urllib3=1.26.11=py310h06a4308_0
  - websocket-client=0.58.0=py310h06a4308_4
  - werkzeug=2.2.1=pyhd8ed1ab_0
  - wheel=0.37.1=pyhd3eb1b0_0
  - x264=1!157.20191217=h7b6447c_0
  - xz=5.2.5=h7f8727e_1
  - yaml=0.2.5=h7b6447c_0
  - yarl=1.7.2=py310h5764c6d_2
  - zipp=3.8.1=pyhd8ed1ab_0
  - zlib=1.2.12=h7f8727e_2
  - zstd=1.5.2=ha4553b6_0
  - pip:
    - einops==0.4.1
    - pyyaml==6.0
prefix: YOUR_CONDA_PATH

cc @Blaizzy

[awaelchli]

Thanks @joshua840
I see you are using PL 1.7.0. The import issue was fixed recently, can you upgrade to the latest version and try again?

[Blaizzy]

Hey @joshua840

Thanks for reporting,

This is a known issue:
#14114 (comment)

Solution

As @awaelchli suggested, the solution is to upgrade to the latest version or any version bigger or smaller than v1.7.0.

Please let us know if this helps you

[joshua840]

Thank you for your reply. I got the same results after creating a new environment and re-installing every latest package version.

Install commands for new env

conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch
conda install pytorch-lightning -c conda-forge
conda install -c conda-forge neptune-client

conda env export output

name: torch_1.12.1
channels:
  - pytorch
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - absl-py=1.2.0=pyhd8ed1ab_0
  - aiohttp=3.8.1=py39hb9d737c_1
  - aiosignal=1.2.0=pyhd8ed1ab_0
  - async-timeout=4.0.2=pyhd8ed1ab_0
  - attrs=22.1.0=pyh71513ae_1
  - blas=1.0=mkl
  - blinker=1.4=py_1
  - boto3=1.24.72=pyhd8ed1ab_0
  - botocore=1.27.73=pyhd8ed1ab_0
  - bottleneck=1.3.5=py39h7deecbd_0
  - bravado=11.0.3=pyhd8ed1ab_0
  - bravado-core=5.17.1=pyhd8ed1ab_0
  - brotlipy=0.7.0=py39h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - c-ares=1.18.1=h7f98852_0
  - ca-certificates=2022.6.15.2=ha878542_0
  - cachetools=5.2.0=pyhd8ed1ab_0
  - certifi=2022.9.14=pyhd8ed1ab_0
  - cffi=1.15.1=py39h74dc2b5_0
  - charset-normalizer=2.0.4=pyhd3eb1b0_0
  - click=8.1.3=py39hf3d152e_0
  - colorama=0.4.5=pyhd8ed1ab_0
  - cryptography=37.0.1=py39h9ce1e76_0
  - cudatoolkit=10.2.89=hfd86e86_1
  - dataclasses=0.8=pyhc8e2a94_3
  - ffmpeg=4.3=hf484d3e_0
  - freetype=2.11.0=h70c0345_0
  - frozenlist=1.2.0=py39h7f8727e_0
  - fsspec=2022.8.2=pyhd8ed1ab_0
  - future=0.18.2=py39hf3d152e_5
  - giflib=5.2.1=h7b6447c_0
  - gitdb=4.0.9=pyhd8ed1ab_0
  - gitpython=3.1.27=pyhd8ed1ab_0
  - gmp=6.2.1=h295c915_3
  - gnutls=3.6.15=he1e5248_0
  - google-auth=2.11.0=pyh6c4a22f_0
  - google-auth-oauthlib=0.4.6=pyhd8ed1ab_0
  - grpcio=1.42.0=py39hce63b2e_0
  - idna=3.3=pyhd3eb1b0_0
  - importlib-metadata=4.11.4=py39hf3d152e_0
  - intel-openmp=2021.4.0=h06a4308_3561
  - jmespath=1.0.1=pyhd8ed1ab_0
  - jpeg=9e=h7f8727e_0
  - jsonref=0.2=py_0
  - jsonschema=3.2.0=pyhd8ed1ab_3
  - lame=3.100=h7b6447c_0
  - lcms2=2.12=h3be6417_0
  - ld_impl_linux-64=2.38=h1181459_1
  - lerc=3.0=h295c915_0
  - libdeflate=1.8=h7f8727e_5
  - libffi=3.3=he6710b0_2
  - libgcc-ng=11.2.0=h1234567_1
  - libgomp=11.2.0=h1234567_1
  - libiconv=1.16=h7f8727e_2
  - libidn2=2.3.2=h7f8727e_0
  - libpng=1.6.37=hbc83047_0
  - libprotobuf=3.15.8=h780b84a_1
  - libstdcxx-ng=11.2.0=h1234567_1
  - libtasn1=4.16.0=h27cfd23_0
  - libtiff=4.4.0=hecacb30_0
  - libunistring=0.9.10=h27cfd23_0
  - libwebp=1.2.2=h55f646e_0
  - libwebp-base=1.2.2=h7f8727e_0
  - lz4-c=1.9.3=h295c915_1
  - markdown=3.4.1=pyhd8ed1ab_0
  - markupsafe=2.1.1=py39hb9d737c_1
  - mkl=2021.4.0=h06a4308_640
  - mkl-service=2.4.0=py39h7f8727e_0
  - mkl_fft=1.3.1=py39hd3c417c_0
  - mkl_random=1.2.2=py39h51133e4_0
  - monotonic=1.5=py_0
  - msgpack-python=1.0.3=py39hd09550d_0
  - multidict=6.0.2=py39hb9d737c_1
  - ncurses=6.3=h5eee18b_3
  - neptune-client=0.16.7=pyhd8ed1ab_0
  - nettle=3.7.3=hbbd107a_1
  - numexpr=2.8.3=py39h807cd23_0
  - numpy=1.23.1=py39h6c91a56_0
  - numpy-base=1.23.1=py39ha15fc14_0
  - oauthlib=3.2.1=pyhd8ed1ab_0
  - openh264=2.1.1=h4ff587b_0
  - openssl=1.1.1q=h7f8727e_0
  - packaging=21.3=pyhd8ed1ab_0
  - pandas=1.4.3=py39h6a678d5_0
  - pillow=9.2.0=py39hace64e9_1
  - pip=22.1.2=py39h06a4308_0
  - protobuf=3.15.8=py39he80948d_0
  - psutil=5.9.0=py39h5eee18b_0
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.7=py_0
  - pycparser=2.21=pyhd3eb1b0_0
  - pydeprecate=0.3.2=pyhd8ed1ab_0
  - pyjwt=2.4.0=pyhd8ed1ab_0
  - pyopenssl=22.0.0=pyhd3eb1b0_0
  - pyparsing=3.0.9=pyhd8ed1ab_0
  - pyrsistent=0.18.1=py39hb9d737c_1
  - pysocks=1.7.1=py39h06a4308_0
  - python=3.9.13=haa1d7c7_1
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python_abi=3.9=2_cp39
  - pytorch=1.12.1=py3.9_cuda10.2_cudnn7.6.5_0
  - pytorch-lightning=1.7.6=pyhd8ed1ab_0
  - pytorch-mutex=1.0=cuda
  - pytz=2022.2.1=pyhd8ed1ab_0
  - pyu2f=0.1.5=pyhd8ed1ab_0
  - pyyaml=6.0=py39hb9d737c_4
  - readline=8.1.2=h7f8727e_1
  - requests=2.28.1=py39h06a4308_0
  - requests-oauthlib=1.3.1=pyhd8ed1ab_0
  - rsa=4.9=pyhd8ed1ab_0
  - s3transfer=0.6.0=pyhd8ed1ab_0
  - setuptools=63.4.1=py39h06a4308_0
  - simplejson=3.17.6=py39hb9d737c_1
  - six=1.16.0=pyhd3eb1b0_1
  - smmap=3.0.5=pyh44b312d_0
  - sqlite=3.39.2=h5082296_0
  - swagger-spec-validator=2.7.6=pyhd8ed1ab_0
  - tensorboard=2.10.0=pyhd8ed1ab_2
  - tensorboard-data-server=0.6.0=py39hd97740a_2
  - tensorboard-plugin-wit=1.8.1=pyhd8ed1ab_0
  - tk=8.6.12=h1ccaba5_0
  - torchaudio=0.12.1=py39_cu102
  - torchmetrics=0.9.3=pyhd8ed1ab_0
  - torchvision=0.13.1=py39_cu102
  - tqdm=4.64.1=pyhd8ed1ab_0
  - typing=3.10.0.0=pyhd8ed1ab_0
  - typing-extensions=4.3.0=py39h06a4308_0
  - typing_extensions=4.3.0=py39h06a4308_0
  - tzdata=2022c=h04d1e81_0
  - urllib3=1.26.11=py39h06a4308_0
  - websocket-client=1.4.1=pyhd8ed1ab_0
  - werkzeug=2.2.2=pyhd8ed1ab_0
  - wheel=0.37.1=pyhd3eb1b0_0
  - xz=5.2.5=h7f8727e_1
  - yaml=0.2.5=h7f98852_2
  - yarl=1.7.2=py39hb9d737c_2
  - zipp=3.8.1=pyhd8ed1ab_0
  - zlib=1.2.12=h5eee18b_3
  - zstd=1.5.2=ha4553b6_0
prefix: YourCondaPath

[awaelchli]

I think that's a bug of some sort in our RequirementCache which evaluates whether a package is available or not.
@carmocca has the implementation of RequirementAvailable/cache changed or fixed when moved to the lightning-utilities?

I find it strange that on 1.7.6 I get this (in the environment as posted above):

>>> from pytorch_lightning.utilities.imports import _RequirementAvailable
>>> _NEPTUNE_AVAILABLE = _RequirementAvailable("neptune-client")

>>> _NEPTUNE_AVAILABLE
Requirement 'neptune-client' not met, DistributionNotFound: The 'webcolors; extra == "format"' distribution was not found and is required by jsonschema

>>> bool(_NEPTUNE_AVAILABLE)
False

>>> _NEPTUNE_AVAILABLE = _RequirementAvailable("neptune")
>>> bool(_NEPTUNE_AVAILABLE)
False

The message

Requirement 'neptune-client' not met, DistributionNotFound: The 'webcolors; extra == "format"' distribution was not found and is required by jsonschema

confuses me a bit because neptune is still importable without problem.

On master I get:

>>> from lightning_utilities.core.imports import RequirementCache
>>> bool(RequirementCache("neptune-client"))
True

[carmocca]

No. If you diff the implementations, they are equal other than the class name. It seems to work fine on my machine when I checkout 1.7.6:

from pytorch_lightning.utilities.imports import _RequirementAvailable

neptune_client = _RequirementAvailable("neptune-client")
print(str(neptune_client))
print(bool(neptune_client))

neptune = _RequirementAvailable("neptune")
print(str(neptune))
print(bool(neptune))

from lightning_utilities.core.imports import RequirementCache
neptune_client2 = RequirementCache("neptune-client")
print(str(neptune_client2))
print(bool(neptune_client2))

Without neptune-client:

Requirement 'neptune-client' not met, DistributionNotFound: The 'neptune-client' distribution was not found and is required by the application
False
Requirement 'neptune' not met, DistributionNotFound: The 'neptune' distribution was not found and is required by the application
False
Requirement 'neptune-client' not met, DistributionNotFound: The 'neptune-client' distribution was not found and is required by the application
False

With neptune-client:

Requirement 'neptune-client' met
True
Requirement 'neptune' not met, DistributionNotFound: The 'neptune' distribution was not found and is required by the application
False
Requirement 'neptune-client' met
True

[awaelchli]

I believe #14714 fixes this problem, at least for me.
@carmocca The pkg_resources.require(self.requirement) method we use there seems to fail because partially some requirements are not met, but the package is still importable. So I switched to module_available for neptune. This fixes the issue for me.

[carmocca]

If I manually uninstall webcolors, I get the same messaging as you.

I don't think this is a bug in pkg_resources, but an issue with the requirements either in neptune or in jsonschema.

(note that neptune pins jsonschema<4.0, so 3.2 will be installed)

import pkg_resources

>>> pkg_resources.require("neptune-client")
Traceback (most recent call last):
  File "/usr/lib/python3.10/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "/home/carmocca/git/py310/lib/python3.10/site-packages/pkg_resources/__init__.py", line 909, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/carmocca/git/py310/lib/python3.10/site-packages/pkg_resources/__init__.py", line 795, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'webcolors; extra == "format"' distribution was not found and is required by jsonschema

>>> pkg_resources.require("jsonschema")
[jsonschema 3.2.0 (/home/carmocca/git/py310/lib/python3.10/site-packages), six 1.16.0 (/home/carmocca/git/py310/lib/python3.10/site-packages), setuptools 63.2.0 (/home/carmocca/git/py310/lib/python3.10/site-packages), pyrsistent 0.18.1 (/home/carmocca/git/py310/lib/python3.10/site-packages), attrs 22.1.0 (/home/carmocca/git/py310/lib/python3.10/site-packages)]

>>> pkg_resources.require("jsonschema[format]")
Traceback (most recent call last):
  File "/usr/lib/python3.10/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "/home/carmocca/git/py310/lib/python3.10/site-packages/pkg_resources/__init__.py", line 909, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/carmocca/git/py310/lib/python3.10/site-packages/pkg_resources/__init__.py", line 795, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'webcolors; extra == "format"' distribution was not found and is required by jsonschema

If I had to guess, neptune should install jsonschema[format] as that will include webcolors

Didn't go deeper into the rabbit hole, but neptune devs should explore it: cc @Blaizzy @Raalsky

[carmocca]

@joshua840 In the meantime, you can check if this is your problem by doing pip install webcolors. If the above is the problem, it should fix it for you

[awaelchli]

@carmocca Thank you for helping investigate
We could continue with the quick fix and at the same time ask neptune if it is appropriate to add jsonschema[format].
I still think that our implementation is strange, since the package is perfectly importable and thus "available", and that's what we want to check in our logic. The current silent behavior is a bit worrying imo.