False Positive import-error for tensorflow

[petrux]

Steps to reproduce

• create a virtual env with python3.5
• run it
• install the latest TensorFlow and pylint via pip
• create a file bug.py write the following code:

    import tensorflow.estimator

    if __name__ == '__main__':
        print(dir(tensorflow.estimator))

• run pylint bug.py

Current behavior

The linter raises an error Unable to import 'tensorflow.estimator, but running the script everything works fine.

Expected behavior

No error should have been raised.

pylint --version output

pylint 2.1.1
astroid 2.0.4
Python 3.5.3 (default, Jan 19 2017, 14:11:04) 
[GCC 6.3.0 20170118]

[PCManticore]

Can you try with --extension-pkg-whitelist=tensorflow.estimator,tensorflow?

[petrux]

@PCManticore I tried but nothing changed.

[AndyHuang0000]

I have the same problem too, placing some .py file under the same directory,
importing them from other .py file,
pylint keep showing "Unable to import" E0401

I am using VSCode, where clicking into the module works well.
Another user reported this problem earlier
microsoft/vscode-python#1185

[Zhiqiang-Ye]

--extension-pkg-whitelist=tensorflow also failed with some false positive E1101 of protobuf

[datakristy]

@AndyHuang0000 I had the same problem, did you ever find a fix?

[elephantum]

I get the same behaviour with tensorflow version 1.12.

Pylint works fine with tensorflow 1.11 though.

[danchyy]

I updated to Tensorflow 1.13.0rc2 and this issue started occurring.

Python version: 3.7.1

Pylint version: 2.2.2

There were no issues at Tensorflow version 1.12.0. and Python version 3.6.5

I tried to whitelist tensorflow.estimator and tensorflow but no help.

EDIT:

Found the workaround by adding the following line to pylintrc:

generated-members=tensorflow.estimator.*,tf.estimator.*

[rkeithlynn]

Same issue with the following versions using vscode 1.31.1 w/virtualenv
python 3.7.2
pylint 2.3.0
astroid 2.2.0
tensorflow 1.31.1

When I installed tensorflow globally (not just inside the virtualenv) it resolved the issue
ie. c:\pip install tensorflow

[PCManticore]

Thanks for the examples folks. It might be that tensorflow does something funky with those attributes, in which case we'll need special support in our inference engine library, astroid, to figure that out. In the meantime, the solution proposed by @danchyy here seems alright to me.

[bersbersbers]

import tensorflow.keras is another example of this, independent of VS Code, even on the command line. The workaround by @rkeithlynn does not help (I only have one global installation of Python and don't use any environments.)

bug.py:1:0: E0401: Unable to import 'tensorflow.keras' (import-error)

[PCManticore]

@bersbersbers You can try with --ignored-modules=tensorflow.keras.

[byarbrough]

--ignored-modules=tensorflow.keras works for Python 3.7.4, pylint 2.3.1, and Tensorflow 1.14.0

[Flamefire]

The problem is that tensorflow is a proxy package with lazily loads tensorflow_core, an internal package. Using the latter does work with pylint but will (and does) cause errors on runtime as you are not supposed to use that.

See this file which will be installed as __init__.py: https://github.com/tensorflow/tensorflow/blob/191f36cb17bd87bc8adf008e5a1e3f66c3afdcc2/tensorflow/virtual_root_template_v2.__init__.py

A solution would be a pylint plugin which dynamically adds everything from tensorflow_core to tensorflow or to make pylint understand this redirection.

[PCManticore]

@Flamefire thanks for the comment. It sounds to me like we could add an astroid transform for tensorflow similar to the likes of numpy: https://github.com/PyCQA/astroid/tree/master/astroid/brain
I don't think pylint is going to understand what tensorflow is doing any time soon.

[Flamefire]

Sure. Something like generating a (faked) list of from tensorflow_core import foo or even from tensorflow_core import * could already work. The tensorflow __init__.py template can be used to tell which modules will be required.

Could you create such a "brain" (or what it is called)?

[Flamefire]

I created a plugin which seems to work:
Save in (e.g.) ~/.local/lib/python3.6/site-packages/pylint_tensorflow/__init__py and pass --load-plugins pylint_tensorflow to pylint:

from astroid import MANAGER, register_module_extender
from astroid.builder import AstroidBuilder
from astroid.exceptions import AstroidBuildingError


def register(_):
    pass


def _getParsedCode():
    from tensorflow import _top_level_modules
    code = 'import tensorflow_core\n' + '\n'.join(
        'from tensorflow_core import {name} as {name}'.format(
            name=name.split('.')[1]) for name in _top_level_modules)
    return AstroidBuilder(MANAGER).string_build(code)


def tensorflowExtender():
    return _getParsedCode()


def _tensorflow_fail_hook(modname):
    parts = modname.split('.', 1)
    if parts[0] == 'tensorflow':
        # Redirect to tensorflow_core
        parts[0] = 'tensorflow_core'
        return MANAGER.ast_from_module_name('.'.join(parts))
    raise AstroidBuildingError(modname=modname)


register_module_extender(MANAGER, 'tensorflow', tensorflowExtender)
MANAGER.register_failed_import_hook(_tensorflow_fail_hook)

@PCManticore If this code looks correct, it can be added as a brain.

[avncalst]

@Flamefire I used your plugin and added in visual studio's settings.json, "python.linting.pylintArgs": ["--load-plugins,pylint_tensorflow", ]. Unfortunately the modules are not recognized in visual studio, such as from tensorflow.keras.optimizers import Adam. I am using Tensorflow 2.0. However my code runs fine.

[Flamefire]

I tested it from command line where it seemed to work. Maybe you are running into an intellisense issue and not the linter? See tensorflow/tensorflow#32982 (comment)

Also: It should be: "python.linting.pylintArgs": ["--load-plugins=pylint_tensorflow", ]

[PanicButtonPressed]

@Flamefire Thanks for providing the pylint plugin for tensorflow. I tested it and recognized, that the unable to import ... errors were completely resolved. However, any module not found errors are not detected, at least for the top_level_modules. So it is still possible to import modules, which don't exist.

[PCManticore]

It sounds to me that @Flamefire 's PR works well in most cases, at least it gets rid of the import-error issues. We can investigate the other ones mentioned by @PanicButtonPressed in a separate effort. @Flamefire when you have some time, would you submit that patch to astroid?

[Flamefire]

Related TF issue: tensorflow/tensorflow#32982

--> Will probably be fixed in next 2.x release

[seehuhn]

The plugin shown @Flamefire does not work for me. I installed the plugin and made sure that it is actually run by pylint (by temporarily adding a print statement). But when I run pylint, I still get the error.

My code (runs with no errors):

#! /usr/bin/env python3

"""Experiment with the Keras interfact to tensorflow."""

import tensorflow.keras as tfk

model = tfk.Sequential()
model.add(tfk.layers.Dense(64, activation='relu'))
model.add(tfk.layers.Dense(64, activation='relu'))
model.add(tfk.layers.Dense(2))

model.compile(optimizer=tfk.optimizers.Adam(0.01),
              loss=tfk.losses.CategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

Pylint call:

> pylint --load-plugins=pylint_tensorflow tfexp.py
************* Module tfexp
tfexp.py:5:0: E0401: Unable to import 'tensorflow.keras' (import-error)

------------------------------------------------------------------
Your code has been rated at 1.67/10 (previous run: 1.67/10, +0.00)

Software used:

• Python 3.7.7 on MacOSX
• pylint 2.3.1
• astroid 2.2.5
• tensorflow 2.1.0

[caolele]

For me, using from tensorflow import estimator instead of import tensorflow.estimator makes the error gone.

[xdhmoore]

I am having a similar issue with the line:

import tensorflow.compat.v1 as tf

The following does seem to work for me, though I guess it's an internal namespace, so not ideal...

import tensorflow_core._api.v1.compat.v1

Because I think this namespace is probably a re-exposing of another namespace, it wouldn't surprise me if there is extra trickery going on for this one specifically.

However, here is a way to easily reproduce any of the problems above with docker:

docker run tensorflow/tensorflow:1.15.2-py3 sh -c 'python --version && python -m pip install pylint && echo "import tensorflow.compat.v1 as tf" > main.py && python -c "import tensorflow.compat.v1 as tf" && python -m pylint -d all -e E main.py'

[bhack]

Is there any PR about this submitted to astroid?

[mihaimaruseac]

TF no longer uses a virtual pip package. Is this still an issue on latest TF?

[pks]

I can still reproduce this w/ tensorflow==2.5.0, pylint==2.8.3, and astroid==2.5.6:

> cat bla.py
import tensorflow as tf
tf.cast(42, dtype=tf.float32)
> pylint --ignored-classes=tensorflow,tf --ignored-modules=tensorflow,tf bla.py
************* Module bla
bla.py:1:0: C0114: Missing module docstring (missing-module-docstring)
bla.py:2:0: E1123: Unexpected keyword argument 'dtype' in function call (unexpected-keyword-arg)
bla.py:2:0: E1120: No value for argument 'DstT' in function call (no-value-for-parameter)

----------------------------------------------------------------------
Your code has been rated at -45.00/10 (previous run: -45.00/10, +0.00)

Edit: Maybe that's more related to #3613. I can verify that this occurs w/ many/most(?) tensorflow calls.

[Pierre-Sassoulas]

Seems like the brain from @Flamefire (#2603 (comment)) is not integrated yet in astroid but it could be.

[MuhammedBuyukkinaci]

As of October 2023, --ignored-modules=tensorflow.keras solution worked like a charm. My environment is as follows:

• Mac M1 Pro(having Arm64 architecture)
• IDE: PyCharm 2022.2.3 (Professional Edition)
• Python 3.11
• Tensorflow Version == 2.14.0
• Linter = Pylint == 3.0.1

[Dobiasd]

Can confirm, the problem still exists.

The following minimal Dockerfile reproduces the problem:

FROM python:3.11.6
RUN pip install tensorflow==2.15.0 keras==2.15.0
RUN pip install pylint==3.0.2
RUN echo "from tensorflow.keras import Model" > main.py
RUN pylint main.py

...
/main.py:1:0: E0401: Unable to import 'tensorflow.keras' (import-error)
/main.py:1:0: E0611: No name 'keras' in module 'tensorflow' (no-name-in-module)
...

And replacing RUN pylint main.py with RUN pylint --ignored-modules=tensorflow.keras main.py successfully works around it.