Move to setup.py based installer

[MasterOdin]

Based on looking into using a setup.py as result of discussion in #44, at this time, I don't believe it's possible without doing a much larger restructuring of asciidoc. While #71 puts down the basics of what's necessary, it still leaves the very unfortunate issue that asciidoc relies on knowing its actual installation location so that it can then load the various conf files and other things. There's no real good way to fix this while also using setup.py that doesn't then require some amount of weird hacking in the path as it gets installed (which is bad). The whole idea of installing it to "any directory" is from an old time and there's no reason to continue to supporting it, especially as moving to a full "pythonic" setup would make it possible to install asciidoc from pypi as well as make it easier for maintainers going forward to build this into package maintainers and the such.

Ideally, I'd say that the best way forward would be to fully embrace setup.py and turn asciidoc into a more "proper" python module such that you'd have an asciidoc folder that contains all python files (as well as __init__.py) which would remove the need for the asciidocapi.py file (as asciidoc would now be normally importable), move the conf/image files inside that folder (and use pkg_resources to load them), and then one could also start to break apart asciidoc.py into smaller module files, add unit testing, etc.

However, this would be a fairly large change and I'd leave it up to @elextr to determine when (if ever) would be an appropriate time that this might be done.

[elextr]

@MasterOdin Pythonification (tm) definitely should be done, couple of questions below.

Does having an application use "standard" Python install methods still work for distro packagers making a system package? ping @aerostitch

I'm not an expert on Python packaging, I understand module installation, but how does that install the user asciidoc command in their path?

The asciidocapi probably needs to exist for quite a while in parallel with any new API to support people who have already used it, its use can be deprecated, but probably needs a long changeover period.

Since it sounds like its a fairly significant change, making it in master directly is probably risky, I suggest we make a branch in the main repo for it, say "repackage" (better names welcome) where old build scripts can be deleted and other breaking changes made. After its settled and a deprecation period for the packaging in master served, it can be merged back into master, and given that other changes to master are (currently) fairly light it should not be much effort to keep repackage up to date with master in the meantime. This way it can be started immediately and progress as resources are available.

[aerostitch]

Hi guys,

Happy to see some traction on this subject! :)
From a distro packaging point of view we have an entire tooling for python packages that integrate fully the installation using the standard python (setup.py) mechanism, so that's perfectly fine with me.

Thanks for checking! ;)
Joseph

[MasterOdin]

I'm not an expert on Python packaging, I understand module installation, but how does that install the user asciidoc command in their path?

The easiest and recommended way (to me) is to specify an entrypoint within the setup.py that would look something like:

    entry_points = {
        'console_scripts': [
            'asciidoc=asciidoc:main',
            'a2x=asciidoc.a2x:main'
        ],
    }

(the actual paths to functions and things would probably change). Python would then handle installing the asciidoc and a2x executables for the enduser.

The asciidocapi probably needs to exist for quite a while in parallel with any new API to support people who have already used it, its use can be deprecated, but probably needs a long changeover period.

Sure. However, as it's now going to be a "proper" python module, the usage of it does become largely simplified as we no longer need to do a ton of work on trying to figure out what the path of import for asciidoc (it basically just becomes always "import asciidoc" and no need to manually looking for files.

Since it sounds like its a fairly significant change, making it in master directly is probably risky, I suggest we make a branch in the main repo for it, say "repackage" (better names welcome) where old build scripts can be deleted and other breaking changes made. After its settled and a deprecation period for the packaging in master served, it can be merged back into master, and given that other changes to master are (currently) fairly light it should not be much effort to keep repackage up to date with master in the meantime. This way it can be started immediately and progress as resources are available.

I'm open to this. Feel free to create that branch on Github and I'll open some PRs against it.

[elextr]

No better name having been suggested, branch "repackage" created.

[elextr]

What else needs to be done to test on test.pypi.org ?

[MasterOdin]

Nothing beyond I guess agreeing who would own the package on test.pypi.org. I believe one could setup a package as having multiple owners, or an owner and a maintainer, etc.

It looks like @florentx does own https://pypi.org/project/asciidoc though.

[elextr]

@MasterOdin IIUC it probably doesn't matter who owns the test.pypi.org since thats liable to go away at any time with no notification, so creating one , testing it out and deleting it should be no trouble.

Probably the project should be asciidoc-py3 anyway, but since @florentx has not used the asciidoc one maybe he will be willing to point it at the asciidoc-py3 one (if pypi can do that).

[MasterOdin]

I don't believe you can point PyPI projects at other projects, but you could just upload it to both asciidoc and asciidoc-py3 (my vote would be to focus on the former though given that's what it's going to be named in the various linux distro releases).

Anyway, a test.pypi release page for it: https://test.pypi.org/project/asciidoc/ I'm going to keep using the ".dev#" release style here just so I can increment it upwards as I make various fixes for the official 9.0.0 release.

Tested by doing:

mpeveler@~/$ cp -R asciidoc-py3 asciidoc-py3-tmp
mpeveler@~/$ cd asciidoc-py3-tmp
mpeveler@~/asciidoc-py3-tmp$ rm -rf asciidoc
mpeveler@~/asciidoc-py3-tmp$ # docker build . -t asciidoc3:latest
mpeveler@~/asciidoc-py3-tmp$ docker run -it -v $(pwd):/srv/asciidoc asciidoc3:latest /bin/bash
root@6b80c4a1c4fb:/srv/asciidoc# pip install -i https://test.pypi.org/simple/ asciidoc
root@6b80c4a1c4fb:/srv/asciidoc# ./tests/testasciidoc.py run

and all tests pass using sources generated from master. (note, the build from PyPI relies on the changes in #71)

[MasterOdin]

Any movement here on what you'd like to see happen next?

[aerostitch]

I'd rather we keep the asciidoc name for the package and the binary to avoid confusing people and make a smoother transition.
Python2 is only supported for a few months anyway so I'm going to have to replace the python2 implementation by the python3 in the same package for Debian in the days to come and there will be no reason for both implementations to coexist.
My advise would be to ship it and build fixes based on the feedback afterwards.
@elextr would you be onboard with that?

[elextr]

@MasterOdin sorry I missed your post of the pypi.

for me pip install -i https://test.pypi.org/simple/ asciidoc gives

Collecting asciidoc
  Could not find a version that satisfies the requirement asciidoc (from versions: )
No matching distribution found for asciidoc

Has the test pypi expired or something?

@aerostitch yeah, the point of this version is to continue after Python 2 EOL, so moving the distro to this is the way to go. Keeping the name in Debian makes sense, but the repo will probably keep its current name and the old python 2 one will be renamed to asciidoc-py2 to be kept for hysterical (erm that should be historical) reasons. The actual asciidoc repo name is likely to get used for non-implementation specific uses.

Since there is not much happening I think we can tag a version in master at any point, @MasterOdin are there any PRs that have to be applied before that?

[MasterOdin]

@elextr Are you sure you're using pip3 there?

$ pip install -i https://test.pypi.org/simple/ asciidoc
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Looking in indexes: https://test.pypi.org/simple/
Collecting asciidoc
  ERROR: Could not find a version that satisfies the requirement asciidoc (from versions: none)
ERROR: No matching distribution found for asciidoc

$ pip3 install -i https://test.pypi.org/simple/ asciidoc
Looking in indexes: https://test.pypi.org/simple/
Collecting asciidoc
Installing collected packages: asciidoc
Successfully installed asciidoc-9.0.dev1

For open PRs, I'd say that #72 is the only "need" to be merged for python3. The others are not directly related to this effort, but I'd really like to see the testing/Travis ones merged sooner rather than later as it helps ensure compliance on an on-going basis.

[elextr]

@MasterOdin oh, ok, as you can see it didn't give me the deprecation message, this is a LTS Linux not bleeding edge.

Its a bit dumb that the default pip can't install an app irrespective of the version, thats the command pypi shows, but I suppose that will go away in time as distros move up to python 3 by default.

I also had to install setuptools, but still got an error compiling a wheel (why is it trying?) although it said it finished installing, but what? There is no asciidoc or similar command available.

So far its not a very good end user experience, see my concerns in my first comment on this thread. That appears to be mostly pypi's problems but can we work around it for end users?

I expect most users will be for making docs from older sources that need the Python version of asciidoc due to customisations, not really experts in Python.

BTW, yes I know its in .local/bin but thats not in PATH by default (at least on Ubuntu and derivatives). However it does appear to work, so well done.

Is #72 applicable to master (its to the repackage branch ATM).

[MasterOdin]

@MasterOdin oh, ok, as you can see it didn't give me the deprecation message, this is a LTS Linux not bleeding edge.

Its a bit dumb that the default pip can't install an app irrespective of the version, thats the command pypi shows, but I suppose that will go away in time as distros move up to python 3 by default.

I also had to install setuptools, but still got an error compiling a wheel (why is it trying?) although it said it finished installing, but what? There is no asciidoc or similar command available.

So far its not a very good end user experience, see my concerns in my first comment on this thread. That appears to be mostly pypi's problems but can we work around it for end users?

I expect most users will be for making docs from older sources that need the Python version of asciidoc due to customisations, not really experts in Python.

BTW, yes I know its in .local/bin but thats not in PATH by default (at least on Ubuntu and derivatives). However it does appear to work, so well done.

For linux, you'd probably need to do sudo pip3 install -i https://test.pypi.org/simple/ asciidoc to have it get it installed to /usr/local/bin, but I can't say for sure what's going on as I'm not sure how you've configured things on your end. I can assure you though that this is how any other CLI based python program specifies its setup though.

With regards to the pip installation line, I'd assume non-technical minded folks would land here (or asciidoc.org) first, which can have pip3 install asciidoc (or whatever we want). There's unfortunately nothing to be done for people who land on pypi who aren't well-versed on how python works on their system, but are adamant to not read the README and not use what comes in their package manager.

Is #72 applicable to master (its to the repackage branch ATM).
I think it depends on what it is you're looking to cut a release of. Are you cutting a release of 9.0.0 for the old-style method of how installation/usage has worked? Or are you cutting a release of 9.0.0 for the new setup.py style of installation/usage (where then repackage would be merged into master)? #72 is only required for the latter, which is what I assumed you were looking for given the context of this issue. I'd like to hear from you (@elextr) what your expectations on what should be in the 9.0.0 release and a roadmap of where this issue would land in a release.

---

For earlier discussion, my comment about asciidoc vs asciidoc-py3 was only for the package name on https://pypi.org, not the name of this repo, where I would argue to use the former (if possible) to match the name of the binary of debian, fedora, etc) and I think would be less confusing.

I agree the repo name should remain as asciidoc-py3 to separate itself from the asciidoc-py2 implementation and the larger (forthcoming?) asciidoc standard repo.

[elextr]

For earlier discussion, my comment about asciidoc vs asciidoc-py3 was only for the package name on https://pypi.org, not the name of this repo, where I would argue to use the former (if possible) to match the name of the binary of debian, fedora, etc) and I think would be less confusing.

Yes, but the problem is that we don't have access to the asciidoc pypi. As there doesn't seem to be a way on pypi to contact a project maintainer I did try to ping the github user I think is the same person, but never got a reply.

[MasterOdin]

If @florentx does not respond (ping), one could always post an issue on https://github.com/pypa/warehouse to request ownership transfer (see pypi/warehouse#1506 for discussion and at the bottom examples of people posting said issues). I'm not sure how much trouble it would be then to get it transferred except that it would take time waiting for a response.

However, could just go with asciidoc-py3 for the time being.

[florentx]

I see my name here, sorry for the annoyance of squatting the name.
I had project to do this packaging few years ago 😁 .

Thank you for doing the job.
Please tell me who, PyPI user(s), would take ownership of the package and I'll assign him as owner.

[MasterOdin]

Well, it is better that you squatted the name and in-turn passed it back to the org instead of some random person squatting it with something unrelated!

Thanks for making me owner so that I can help with the transition as necessary, and I am happy to remove myself after @elextr is added if so desired.

[elextr]

@florentx thank you for the offer.

I suggest that both of us (if thats possible) since having a single person controlling a unique asset is risky. If they become unavailable I assume Pypi has a process for transferring control, but I'm sure its a pain.

[MasterOdin]

You can have as many owners on a package as you'd like as well as just general maintainers who are allowed to make releases, but are not allowed to modify the metadata about the package.

@elextr what is your username on PyPI and I can add you to the package.

[elextr]

Its so long since I used pypi I'm not sure of the username I used, and certainly can't remember the password, and trying likely usernames didn't get me a password reset email, but it may be pointing to a long defunct email anyway.

So I just set up "elextr" since it luckily was free.

[MasterOdin]

I've added you as an owner to it. However, I would like to propose turning on Travis (#9) and then can add a deploy to pypi section in the .travis.yml to handle it whenever a new tag gets cut in the repo. I've personally set up a bot user (Master_Odin_Bot) to handle this in my own projects, which is useful as if this user ever gets hacked due to credential leak or whatever, they only have contributor access over my projects instead of owner access.

[MasterOdin]

master is now using pypi for distribution. It's currently an alpha release (10.0.0a1) as testing goes on, but it's available via pypi: https://pypi.org/project/asciidoc/10.0.0a1/