Add namespace redirect hook for qiskit-aer (#5089)

* Remove namespace packaging and hardcode elements (attempt 2)

Namespace packages are constant source of problems for users. The python
packaging ecosystem around splitting packages across namespaces is
fragile at the best of times and can often leave a you with an
environment that isn't recoverable (especially when mixing install
methods). There is also a performance hit whenever there is a piece of
the namespace we allow external packages to extend since it requires
doing a full python path search which can be slow depending on the
backing I/O and the number of paths in sys.path for an environment. This
commit starts the process of addressing this by removing the arbitrary
namespace hook points and hard coding the element namespace maps via a
custom import loader at the root of the namespace. This has 2 advantages
it removes the use of namespace packages so the fragility and
performance impact are fixed since every element will be renamed to use
'qiskit_' instead of 'qiskit.', but it also makes it explicit where we
extend the namespace. The previous method allowed any package to extend
qiskit.* and qiskit.providers.* with whatever they wanted.

We'll need to coordinate updating the elements with this merging,
because it is a breaking change for each element (although not for end
users).

A potential follow on is to add a plugin interface for 3rd party
providers like what was proposed in #1465 so that we can make external
providers externally discoverable without needing to add manual hook
points moving forward (this was done for backwards compat with the aqt
and honeywell provider).

This is a second attempt at removing namespace packaging. The first
attempt in PR #4767 was merged and had to be reverted because there were
some circular import error issues that needed to be resolved. Since
having this in terra blocks CI for all the qiskit elements a revert was
necessary to unblock developement for the entire project while those
were resolved.

* Try using new aer path for qiskit.Aer alias

* Try moving Aer and IBMQ alias to the end

* Fix typo

* Fix circular import issue

* Fix typos

* Run black

* Remove file encoding

* Adjust init with import re-direct

* Make qiskit_aer a opportunistic load

To ensure that someone with an old version of Aer installed can still
access it via the old namespace this changes the meta finder logic to
first try qiskit_aer, if it's importable then we build the redirect path
first and use that for the legacy qiskit.providers.aer path. If it's not
then we just return None and fall back to the other finders in
sys.meta_path.

To support this the pkgutil hook is added back to qiskit.providers to
add the namespace hook for old version of aer using namespace packagingm
although not strictly necessary because the implicit support for
namespace packages will still likely work we can remove it at a later
date.

* Deprecate qiskit.Aer entrypoint

* Fix lint

* Remove unnecessary noqa comments

* Revert version.py change and use old aer import for now

* Fix typo

* Add back pkgutil extension to root init

* Make redirect hook more generic

* Add comments explaining the various hooks

* Make qiskit.Aer a pending deprecation instead of a deprecation

* Restrict namespace redirect error to ModuleNotFoundError

* Apply suggestions from code review

Co-authored-by: Jake Lishman <jake@binhbar.com>

Co-authored-by: Jake Lishman <jake@binhbar.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

qiskit/__init__.py

@@ -31,6 +31,15 @@
 sys.modules["qiskit._accelerate.results"] = qiskit._accelerate.results
 
 
+# Extend namespace for backwards compat
+from qiskit import namespace
+
+# Add hook to redirect imports from qiskit.providers.aer* to qiskit_aer*
+# this is necessary for backwards compatibility for users when qiskit-aer
+# and qiskit-terra shared the qiskit namespace
+new_meta_path_finder = namespace.QiskitElementImport("qiskit.providers.aer", "qiskit_aer")
+sys.meta_path = [new_meta_path_finder] + sys.meta_path
+
 # qiskit errors operator
 from qiskit.exceptions import QiskitError, MissingOptionalLibraryError
 
@@ -52,6 +61,9 @@
 # Allow extending this namespace. Please note that currently this line needs
 # to be placed *before* the wrapper imports or any non-import code AND *before*
 # importing the package you want to allow extensions for (in this case `backends`).
+
+# TODO: Remove when we drop support for importing qiskit-aer < 0.11.0 and the
+# qiskit-ibmq-provider package is retired/archived.
 __path__ = pkgutil.extend_path(__path__, __name__)
 
 # Please note these are global instances, not modules.
@@ -61,11 +73,11 @@
 
 # Moved to after IBMQ and Aer imports due to import issues
 # with other modules that check for IBMQ (tools)
-from qiskit.execute_function import execute  # noqa
-from qiskit.compiler import transpile, assemble, schedule, sequence  # noqa
+from qiskit.execute_function import execute
+from qiskit.compiler import transpile, assemble, schedule, sequence
 
-from .version import __version__  # noqa
-from .version import QiskitVersion  # noqa
+from .version import __version__
+from .version import QiskitVersion
 
 
 __qiskit_version__ = QiskitVersion()
@@ -83,6 +95,13 @@ def __bool__(self):
                 from qiskit.providers import aer
 
                 self.aer = aer.Aer
+                warnings.warn(
+                    "The qiskit.Aer entry point will be deprecated in a future release and "
+                    "subsequently removed. Instead you should use this "
+                    "directly from the root of the qiskit-aer package.",
+                    PendingDeprecationWarning,
+                    stacklevel=2,
+                )
             except ImportError:
                 return False
         return True
@@ -93,6 +112,13 @@ def __getattr__(self, attr):
                 from qiskit.providers import aer
 
                 self.aer = aer.Aer
+                warnings.warn(
+                    "The qiskit.Aer entry point will be deprecated in a future release and "
+                    "subsequently removed. Instead you should use this "
+                    "directly from the root of the qiskit-aer package.",
+                    PendingDeprecationWarning,
+                    stacklevel=2,
+                )
             except ImportError as ex:
                 raise MissingOptionalLibraryError(
                     "qiskit-aer", "Aer provider", "pip install qiskit-aer"

qiskit/namespace.py

@@ -0,0 +1,69 @@
+# This code is part of Qiskit.
+#
+# (C) Copyright IBM 2020.
+#
+# This code is licensed under the Apache License, Version 2.0. You may
+# obtain a copy of this license in the LICENSE.txt file in the root directory
+# of this source tree or at http://www.apache.org/licenses/LICENSE-2.0.
+#
+# Any modifications or derivative works of this code must retain this
+# copyright notice, and modified files need to carry a notice indicating
+# that they have been altered from the originals.
+
+# pylint: disable=unused-argument
+
+"""Module for utilities to manually construct qiskit namespace"""
+
+import sys
+from importlib.abc import MetaPathFinder, Loader
+import importlib
+
+
+def _new_namespace(fullname, old_namespace, new_package):
+    names = fullname.split(".")
+    new_namespace_names = new_package.split(".")
+    old_namespace_names = old_namespace.split(".")
+    fullname = ".".join(new_namespace_names + names[len(old_namespace_names) :])
+    return fullname
+
+
+class QiskitLoader(Loader):
+    """Load qiskit element as a namespace package."""
+
+    def __init__(self, new_package, old_namespace):
+        super().__init__()
+        self.new_package = new_package
+        self.old_namespace = old_namespace
+
+    def module_repr(self, module):
+        return repr(module)
+
+    def load_module(self, fullname):
+        old_name = fullname
+        fullname = _new_namespace(fullname, self.old_namespace, self.new_package)
+        module = importlib.import_module(fullname)
+        sys.modules[old_name] = module
+        return module
+
+
+class QiskitElementImport(MetaPathFinder):
+    """Meta importer to enable unified qiskit namespace."""
+
+    def __init__(self, old_namespace, new_package):
+        super().__init__()
+        self.old_namespace = old_namespace
+        self.new_package = new_package
+
+    def find_spec(self, fullname, path=None, target=None):
+        """Return the ModuleSpec for Qiskit element."""
+        if fullname.startswith(self.old_namespace):
+            try:
+                importlib.import_module(
+                    _new_namespace(fullname, self.old_namespace, self.new_package)
+                )
+                return importlib.util.spec_from_loader(
+                    fullname, QiskitLoader(self.new_package, self.old_namespace), origin="qiskit"
+                )
+            except ModuleNotFoundError:
+                return None
+        return None

qiskit/providers/__init__.py

@@ -692,4 +692,6 @@ def status(self):
 
 
 # Allow extending this namespace.
+# TODO: Remove when we drop support for importing qiskit-aer < 0.11.0 and the
+# qiskit-ibmq-provider package is retired/archived.
 __path__ = pkgutil.extend_path(__path__, __name__)

qiskit/version.py

@@ -104,6 +104,8 @@ def _load_versions(self):
         import pkg_resources
 
         try:
+            # TODO: Update to use qiskit_aer instead when we remove the
+            # namespace redirect
             from qiskit.providers import aer
 
             self._version_dict["qiskit-aer"] = aer.__version__