Rework namespace extension

[FlorianWilhelm]

Right now we historically use pkg-resource-style namespace packages. According to the official packaging guide this seems to be still valid but an outdated option. With version 4.0 we should break with this behavior and should start using the new native namespace packages as define in PEP 420 that requires Python 3.3 and above.

[FlorianWilhelm]

@abravalheri, I am not sure anymore if we should really change this. I see no advantages except of the fact that they call it recommended in the docs. But there are risks of course as always. Do you see any advantages at all? Ah, and the fact that find_packages doesn't work anymore seems to indicate that we would have to list all packages explicitly in setup.cfg again. I find our current solution with

[options.packages.find]
where = src
exclude =
    tests

much nicer.

[abravalheri]

Yeah, I was reading the links you sent this morning and wondering the exactly same thing...

Is pkgutil-style namespace packages better? Does it allow us to use the same current approach?

[FlorianWilhelm]

@abravalheri Okay, then I think that we agree on the fact that the new native namespace packages seems to give us no tangible advantages but a lot of risks and they need more configuration in setup.cfg. Consequently, we shouldn't introduce them in PyScaffold 4.0 just for the sake of having a new feature. Also it says in the packaging guide that one should stay with pkg-resources-style namespace packages and not migrate to something new. Also the pkgutil-style namespace packages seem to provide no advantages. So, let's stick to what we have :-)

[abravalheri]

So it seems that setuptools now provides a find_namespace option which implements PEP 420 and can be used from setup.cfg. Moreover, it is backward compatible (this is the wording used in the docs and the implementation is done via inheritance), which means it could be used as more powerful a drop in replacement for find (it might be misleading though when no namespace package is used).

The main disadvantage in using it instead of find is that it would normally end up classifying all the folders in a repository as namespace packages, but since we use a src-layout we probably are "imune" to that... (the best is to run some experiments).

I will re-open this issue, for investigation (specially because obviating pkg_resources is a milestone in setuptools, see #309)

[smartsammler]

Since setuptools version 40.1 there is the find_namespace: directive with which it's probably possible to create native namespaces (PEP420).
We tested a namespace package setup with PyScaffold, deleted the src/$namespace/__init__.py and changed packages = find: in the setup.cfg to packages = find_namespace:, increase the required setuptools version to 0.40.1 and it worked.
We did so, because we wanted to build Debian packages with py2dsc-deb and the installation failed/needed extra arguments, because the __init__.py was not replaced automatically.

[abravalheri]

Thank you very much @smartsammler! Nice to know that this works, it will probably be part of v4.

Now the only thing I am not exactly sure is if we should always use find_namespace instead of find (since it does not seem to be dangerous for src-based layouts) or if we do it conditionally, just in case the user activates the --namespace extension.

[smartsammler]

Actually Sphinx did not work anymore correctly. To fix this one also has to provide the --implicit-namespaces flag to sphinx-docs.
docs/conf.py:

-    cmd_line_template = "sphinx-apidoc -f -o {outputdir} {moduledir}"
+    cmd_line_template = "sphinx-apidoc --implicit-namespaces -f -o {outputdir} {moduledir}"

(Maybe this information is helpful in case someone also want to use putup and then change to native namespaces).

Unfortunately I don't know if find_namespace does no harm to non-native namespace packges, nor do I know it for the Sphinx configuration.

[abravalheri]

Thank you very much @smartsammler, I think the best is to do the change you propose, and see if we have any feedback, but I suppose as long as we are using the src-layout we should be good to go (everything inside the src folder should be part of the package).