Release Process

[elextr]

Moved from closed #41 #41 (comment) #41 (comment) #41 (comment)

[elextr]

@aerostitch agree 9.0.0. We can certainly tag versions

@MasterOdin I don't suppose you have tested that release process actually works right with this repo?

Come to that, does it actually work at all, since it still has website stuff in it by my quick perusal.

If there are significant fixes needed to the old AAP system that (AFAIK) nobody knows, rather than work it out, maybe its better to change to a modern build/release system, like meson.

[aerostitch]

Maybe the 1st step would be to define what we expect to happen when a release is built so that we can define what is needed.

To me, the website stuff should be managed in a separate repo so a release here should not affect the doc (maybe it can create a PR in the doc repo to update the latest release version and point to the changelog, but I don't think it should be more).

So basically the only things left I see in the release process would be:

• update common.aap and configure.ac (but I doubt we should keep those)
• update asciidoc.py and a2x.py
• update the changelog
• cut the tag and populate the release description
• verify that the automatic build in Docker hub went through
• advertise the release (this includes updating the website I'd say)

I see in the Makefile.in that the man pages are regenerated but maybe that could be just a build done and attached as a tarball to the release? But I honnestly doubt that anybody still installs asciidoc manually unless we decide to port it to pypi but that's not ready yet.

We could also make the example/website part go away maybe?

Do you see anything else?

[elextr]

The website is a whole 'nother problem.

Because its a gh-pages website it is a branch, so its actually inside the repository at the moment. Updating it is manual I believe, run build website using one of the systems and then make a pull request on the gh-pages branch with the result. Although there may be more steps needed, I havn't actually done it.

That won't update THE Asciidoc website though, since the asciidoc.org redirect points to asciidoc/asciidoc gh-pages branch.

No the docs (and therefore the website source) should not be in another repo, they should be with the code so that one pull request updates both in sync. Asciidoctor currently has them separate and is planning to combine them IIUC. The website product should only be in the gh-pages branch.

Whichever build system is used, it should make the website pages, but not automatically as part of the build. It doesn't seem sensible to have a separate build process for the website.

update common.aap and configure.ac (but I doubt we should keep those)

Ahh, I missed configure.ac, wondered why we had a makefile.in. So there are two build/release tools "supported", AAP and Autofools.

But I honnestly doubt that anybody still installs asciidoc manually unless we decide to port it to pypi but that's not ready yet.

Agree, I see the build as being simply to update embedded version stuff and make the distribution tarball. Asciidoc itself runs quite happily without installation so no "build" process is needed for that. But don't you need the ROFF manpages made for the tarball?

update asciidoc.py and a2x.py

Should be done by the build vers-update target as its called in AAP, not sure it the Autotools build has the same target but I think it does it as part of the process.

update the changelog

Which is sadly a manual step that "somebody" has to do.

cut the tag and populate the release description

Well, the updated stuff from the build above should be made into a PR which is then merged. Then the changelog PR (which is a single commit) is merged and that is what should be tagged.

Not sure what you mean by "populate the release description".

advertise the release (this includes updating the website I'd say)

Which brings us circularly back to the website 😁

[MasterOdin]

The website product could be in another repo and it wouldn't affect the release process too much, as I'd like to make the whole thing just run through Travis anyway.

If there are significant fixes needed to the old AAP system that (AFAIK) nobody knows, rather than work it out, maybe its better to change to a modern build/release system, like meson.

I've looked through the AAP sources and they're all relatively simple that it shouldn't be too hard to convert them to straight makefiles (or a higher level build tool like meson, but we certainly don't need to use AutoTools for this, just make), or even just simple bash scripts depending on what a "release" constitutes or if anyone actually needs the build tools for certain tasks (especially if more automation through Travis is done for releases). AAP should 100% be dropped though.

We could also make the example/website part go away maybe?

That's actually the source for the the gh-pages, even though a large number of the pages are "symlinks" to other pages within the repo. It's also used for a couple of the tests. However, we can setup the build of the website and subsequent pushes to gh-pages to be hooked up to pushes to master or tagged releases within Travis, which I'd advocate for.

@MasterOdin I don't suppose you have tested that release process actually works right with this repo?

I've tested parts of the build as I've played around with both make and a setup.py for the repo, and would be confident I could certainly produce a full setup for the build as part of Travis-CI. Just need to know what you'd constitute as a "release".

Should be done by the build vers-update target as its called in AAP, not sure it the Autotools build has the same target but I think it does it as part of the process.

It doesn't, but easy enough to fix.

[aerostitch]

Agree, I see the build as being simply to update embedded version stuff and make the distribution tarball. Asciidoc itself runs quite happily without installation so no "build" process is needed for that. But don't you need the ROFF manpages made for the tarball?

Nope we regenerate the man pages from the asciidoc format. using the Makefile. :)

update the changelog

Which is sadly a manual step that "somebody" has to do.

There are plenty of tools out there to take care of that automatically! :)

I agree with @MasterOdin and I don't think the release process for the actual tool should be more than:

• update version number
• update changelog
• so the release (with a nice description in the github interface maybe?)

To me the tasks about updating the website and advertising on whatever communication channels/social medias you have available for asciidoc are satellites tasks that comes post-release.

The website could be re-generated with each commit via travis, this way we wouldn't have to think about it.
And as all the changes go through PR we could easily use tools like github-changelog-generator for example. I think the current changelog could be renamed changelog.8 and have a changelog.9 for the current changes. This way it would just be another step in the release target of the Makefile.

[MasterOdin]

At this current stage, I don't think an automated step is necessary for the changelog. I'll make a PR later today to start the changelog for 9.0.0.

I'll also go through the various *.aap and makefiles within the project and give a description of all of the different tasks that they do (in a separate issue) so that we can make steps towards unifying it all under something that isn't aap.

[elextr]

@aerostitch

Nope we regenerate the man pages from the asciidoc format. using the Makefile. :)

Ok.

There are plenty of tools out there to take care of that automatically! :)

Never found any that were useful, a changelog isn't just a list of commits in Git for developers, its a human list of changes for users. And its usually ordered by something like "New Features", "Changes", "Fixes", "Incompatibilities", not commit ordering. Still AI is improving every day :)

To me the tasks about updating the website and advertising on whatever communication channels/social medias you have available for asciidoc are satellites tasks that comes post-release.

Yes, from a packagers point of view.

But the "release" process from a project point of view isn't just tagging and making a tarball. Non-technical things are also all part of the release process, and for the project just as important, that includes documentation webpages, and dissemination ... including pinging packagers. 😁

BTW thanks for maintaining Asciidoc packages for ages and helping get this set up to be packaged.

@MasterOdin

It doesn't, but easy enough to fix.

Ok, maybe the version number should be in a separate file independent of build system, not in the AAP files.

I'll make a PR later today to start the changelog for 9.0.0.

Thanks.

something that isn't aap.

It seems a nice system, its just that nobody uses it AFAIK so there isn't any expertise about, and its developer is busy with his other minor project, called "vim" :)

Autotools isn't really designed for a Python project, but if it works at least its supported by most of the distros build farms.

[MasterOdin]

It seems a nice system, its just that nobody uses it AFAIK so there isn't any expertise about, and its developer is busy with his other minor project, called "vim" :)

Yeah, it looks kind of neat, but outside of this project and in some brief looking around some vim stuff, I've never seen A-A-P used which I think makes for a good reason to switch to something that more people are likely familiar with and also will probably already have installed (make).

I'd like to point out that I'm proposing switching to just make, and not requiring anything else (I'd use CMake over autotools, which I think has aged really poorly) as there's no compilation step happening here, nor any sort of linkage of dynamic libraries or the like which is really when those higher level tools over make shine.

[elextr]

I'd like to point out that I'm proposing switching to just make

I think thats fine, so long as it produces the relevant products, both *x tarball and windows zip file and runs on any platform (this means be careful of GNU make extensions). I guess targets default (ie update versions), "dist", and "check" are the main ones, for a Python program that runs from its source directory I don't see "install" and "uninstall" being very useful make targets.

[apjanke]

I don't see "install" and "uninstall" being very useful make targets.

If you want to play nice with package managers (like Homebrew), then a make install (or a make install that runs from the result of make dist) is probably a useful target, since that basically defines the artifacts that are made available to the system and included in packaged distributions. The "runs from source directory" result of plain make is probably not sufficient for most package managers.

[elextr]

Ok, so install I guess, although really what goes where is up to the distribution, asciidoc doesn't care. And given that distributions don't agree what goes where it is making the makefile more complex if $PREFIX and $BINDIR and $DOCDIR and whatever else needs to be respected.

Note that because Asciidoc is Python, the "runs from source dir" needs no make or anything, even in a raw git clone asciidoc.py and a2x.py should run correctly.

[apjanke]

Usually it's only Java projects that take such a "some assembly required" approach to installation. :)

Note that because Asciidoc is Python, the "runs from source dir" needs no make or anything, even in a raw git clone asciidoc.py and a2x.py should run correctly.

IMHO, from the user's perspective, the fact that Asciidoc is Python is an irrelevant internal implementation detail. What the user cares about is that asciidoc, a2x, and other commands are made available to them. And for most users, that means that they're in "the system" and on their default $PATH. Running from the source directory requires $PATH manipulation and doesn't play well with package managers or the "standard" Unix filesystem hierarchy.

what goes where is up to the distribution, asciidoc doesn't care. And given that distributions don't agree what goes where it is making the makefile more complex if $PREFIX and $BINDIR and $DOCDIR and whatever else needs to be respected.

If you can create a Makefile that follows the GNU Makefile conventions, then you will have reasonable defaults, and standard customization hooks for all of the distribution-specific paths that all common distributions care about. The autotools toolchain supports this. I can help with PRs to do this if you're interested.

Speaking as a package maintainer, if you don't have a make install or equivalent that puts commands into some "system" prefix, then you have an incomplete product.

[elextr]

IMHO, from the user's perspective, the fact that Asciidoc is Python is an irrelevant internal implementation detail. What the user cares about is that asciidoc, a2x, and other commands are made available to them.

Agreed.

Running from the source directory requires $PATH manipulation and doesn't play well with package managers or the "standard" Unix filesystem hierarchy.

It was my understanding that you thought that make had to be run before Asciidoc would run, so I was simply pointing out that was not correct, I wasn't suggesting that for most users running from a source directory was a preferred installation.

Ummm, overall I think @apjanke @MasterOdin and I are mixing up which "makefile" we are talking about. Certainly I was confused at least once.

I am proposing a project tool to patch the version number and create a distribution tarball and windows zipfile and website and so on to replace the AAP script. It doesn't have install but does have the GNU make dist and make test functionality and it should run on multiple platforms as AAP did, so releases and tests can be made on any platform.

This is a tool for Asciidoc team to help with release, it could just as easily be a Python script as a makefile, but a makefile has the advantage that it can ensure patches, webfiles etc are remade consistently if any file changes, without rebuilding everything. So its an improvement on AAP in that way. My comment about avoiding GNU make extensions was aimed at ensuring it met the multi-platform requirements noted above.

The tarball still has the traditional configure; make install support, which I think is what @apjanke is talking about. This is generated by the makefile that replaces AAP, using autotools, cmake, meson, any tool. But (I hope) @MasterOdin wasn't suggesting that configure or this makefile be manually maintained. Since they apply only to Linux they indeed should meet the somewhat complex GNU standard, even if thats overkill for a Python project. As @apjanke points out distros depend on that standard but as also pointed out, almost no users will.

On the other hand the Windows zipfile does not have a configure or makefile since it just unzips, which I guess is why the test is in the AAP script where it applies to all platforms.

then you have an incomplete product.

We don't have a "product", in one of the rare cases where I agree with RMS, let me paraphrase him, "we make the software because we want to, and we make it available for others to use, we are happy if they find it useful, but we do not gain from their use, we are sad if they do not find it useful, but we do not lose if they stop using it", something to remember about volunteer projects in this age of corporate sponsored open source with "products" whose success is judged by "user numbers". [sorry, end rant]

[MasterOdin]

One thing I'm curious about is how does distros handle distribution/installation of python projects like numpy, scipy, or some other example that has a mix of python and data files that have a setup.py and don't have a makefile and can be installed via pip, but can also be installed via apt, yum, etc.? Just so we have a point of view of "what would happen if we forwent AAP and make altogether for the more standard python approach" for distribution.

[aerostitch]

@elextr:

Never found any that were useful, a changelog isn't just a list of commits in Git for developers, its a human list of changes for users. And its usually ordered by something like "New Features", "Changes", "Fixes", "Incompatibilities", not commit ordering. Still AI is improving every day :)

The ones I like are the ones that rely on PR instead of commits. The idea is to add relevant labels on the PR so that they are categorized properly in the changelog with the tool.

Ummm, overall I think @apjanke @MasterOdin and I are mixing up which "makefile" we are talking about. Certainly I was confused at least once.

I think the simplest thing to do would just use setuptools and build a setup.py like every other python projects and then push it via pip. This way we don't reinvent the wheel and if someone wants to install it on any platform without package manager, then one uses pip.

@MasterOdin

Just so we have a point of view of "what would happen if we forwent AAP and make altogether for the more standard python approach" for distribution.

The distro have complete tooling for that to make it almost totally transparent. I think we should just go ahead and do that. Looks like the cleanest way to go here and a lot of python people are already familiar with it, so easier to maintain in the long run.

[elextr]

@aerostitch

The idea is to add relevant labels on the PR so that they are categorized properly in the changelog with the tool.

Ok, that makes more sense, I had only found projects that essentially used the git history as the changelog, something to look at later.

The distro have complete tooling for that to make it almost totally transparent.

Ok, thats great, I didn't want to only have a Python literate install method (pip) since many of the Asciidoc users are likely to be writers, not Pythonistas :)

I totally agree that using the standard Python mechanisms as the only (well, except windows) install mechanism within the project is the way to go, and if it makes distros work easier, even better.

@apjanke does homebrew work like that for Python apps too?

Now all we need is to have a script that does the version patch (or even better, do away with it somehow) and a script that tests, makes the windows zipfile, and (future) website. It could even be Python, which has portable zip built in so no need for a separate tool.

[MasterOdin]

Now all we need is to have a script that does the version patch (or even better, do away with it somehow) and a script that tests,

If this repo is moved towards more "proper" python setup, then this would be become easier to do, but for now, a minimal makefile that has a "make version" path or something is probably best for now. Same with a "make test" that just executes "test/testasciidoc.py" is also easy, but the explicit here is that the makefile is totally a developer tool and not anything a normal end user would have to really worry about using. They would be pushed to use setup.py (either through pip, their package manager, or via a git clone and then python3 setup.py install instead of the configure, make, make install process).

makes the windows zipfile, and (future) website.

These tasks are best suited I think towards Travis (for the generating of a zip and building the website) as these can be hooked up to run on pushes to master, tagged releases, etc. (with the ability to append files to tagged releases as well).

[apjanke]

It was my understanding that you thought that make had to be run before Asciidoc would run, so I was simply pointing out that was not correct, I wasn't suggesting that for most users running from a source directory was a preferred installation.

Oh, thanks. You're right. Sorry for any confusion. I'm just coming at this primarily from a package maintainer's viewpoint, so I've got biases.

Ummm, overall I think @apjanke @MasterOdin and I are mixing up which "makefile" we are talking about. Certainly I was confused at least once.

The tarball still has the traditional configure; make install support, which I think is what @apjanke is talking about.

You are right; I think I've gotten confused here too. As long as the distribution tarball has the traditional configure; make install, then that's all that Homebrew core cares about, and I would think that's all that most package managers care about.

After reading @elextr's comments, I think you can ignore all my comments, because I didn't understand the staged makefile you're talking about here.

[apjanke]

We don't have a "product", in one of the rare cases where I agree with RMS, let me paraphrase him, "we make the software because we want to, and we make it available for others to use, we are happy if they find it useful, but we do not gain from their use, we are sad if they do not find it useful, but we do not lose if they stop using it"

/rant mode engaged

True. True. But don't you want it to take over the world? As the developer, you're uniquely positioned to make this software more portable and easily installable than any packager is.

Not saying you should. Not saying you gotta. But don't you wanna?

/rant mode disengaged

And if you don't, no worries. This software is still useful, and package managers can adapt.

[apjanke]

@apjanke does homebrew work like that for Python apps too?

As I understand it, Homebrew handles Python-implemented applications by:

• Taking a snapshot of their dependencies in the formula file using Homebrew's "resource" mechanism, and
• Installing all those dependencies in a dedicated virtualenv directory under the application installation directory

This is for stability and isolation purposes, so that
a) If you install Python-dependent app Foo of a given version, you get the exact same installation with the same dependent Python libraries, and
b) Any two Python-dependent apps Foo and Bar are isolated from each other, so they can't break or otherwise influence each other's behaviors

To make this work, (I think) Homebrew uses standard Python installation mechanisms, but wraps them in an application-installation-specific root directory.

The documentation for Homebrew is at:

• Homebrew and Python
• Python for Formula Authors

The main thing to note here is that (IMHO) AsciiDoc counts as a Python "application", not a Python "library", in Homebrew terms.

[MasterOdin]

I think it would probably help splitting this issue up into a couple of actionable items of what it is that we want done here.

I finally had time to sit down and look at what the project has and is currently doing with regards to both make (via the included configure.ac and Makefile.in) and A-A-P. I'll break down the various A-A-P scripts and their endpoints below:

common.aap:

• set global variable to be used in other *.aap scripts for version and release date ($VERS and $DATE)

main.aap

• vers: prints out the version and release date from common.aap
• vers_update: writes the version variable from common.aap into asciidoc.py, a2x.py, and configure.ac
• tags: delete the tags file and then run ctags over asciidoc.py, asciidocapi.py, tests/testasciidoc.py
• docs: run ./doc/main.aap
• website: run ./examples/website/main.aap
• distribution: runs vers_update, docs, website and then does:
  • Runs autoconf
  • symlinks current directory into asciidoc-$VERS
  • creates a tar (named asciidoc-$VERS.tar.gz) from all files listed in MANIFEST file
  • checks if zip command exists, and if so creates a zip (named asciidoc-$VERS.tar.gz) using the previously created symlink directory
  • Removes the symlink directory
• test: Runs python ./asciidoc.py --doctest, python ./asciidocapi.py, executes ./doc/main.aap test, and then python ./tests/testasciidoc.py run if there are *.html files in ./tests/data

./doc/main.aap:

• all: converts all of the .txt files to the appropriate format (html, man, pdf, epub, etc.) as well as generating out pdf files and the like for distribution
• clean: removes all generated files
• spell: checks the spelling in the text files through aspell using ./doc/asciidoc.dict as a dictionary for it
• clean_testfiles: Removes files generated for tests (uses a subset of what clean does, but does include *.png files)
• test: Runs clean_testfiles and then generates the output from running asciidoc on the files in ./doc

./examples/website/main.aap:

• all: runs asciidoc over the files in the directory, getting html generated files
• copy: deletes and then creates directory ~/webs/asciidoc and then copies the files necessary for the website to that directory. Does execute ./main.aap distribution as part of this to ensure all necessary files have been generated.
• clean: Deletes the generated web pages
• spell: runs aspell over files in directory

The actual installation steps of asciidoc have always been handled by the Makefile.in file.

Of the files in ./doc/ and ./examples/website/, what are needed for package maintainers? All of the *.html and *.pdf files or just the manpages? Could probably just focus on the endpoints in the top-level main.aap and ./doc/main.aap and leave the ./examples/website/main.aap for a separate issue/task?

At this point, I think trying to move to a more "pythonic" setup.py is not feasible without making much larger changes to the asciidoc source as it does a lot of wonky things around the concept of trying to locate asciidoc and its resources on the filesystem.

[MasterOdin]

So my plan here is to translate ./doc/main.aap and ./examples/website/main.aap into list of commands in the toplevel makefile with a prefix of doc_ and website_ on the command to show where it affects. Then will tackle the ./main.aap targets.

@elextr please let me know if you're alright with the target names, want any changes, etc. I've opened #81 as a sort of test of what this translation process might look like, though as I go along, will probably be going back and refactoring old targets, variables used, etc.

[MasterOdin]

With 9.0.0 out, the release process I followed was:

1. Make sure changelog is up-to-date
2. update configure.ac to have right version + date
3. Run vers_update and commit change python files
4. Create release on GH under https://github.com/asciidoc/asciidoc-py3/releases (this will automatically kick off a GH action that generates a tar.gz and zip file and attach it to the release)
5. Post to asciidoc mailing list
6. Create PR on homebrew updating asciidoc

Are there steps that people think are missing here, or is this good? I think making a Developing section in the README with a Creating a Release subsection is probably as good a place as any to enshrine these steps.

[apjanke]

Maybe:
3b. git tag v-<version>; git push --tags
?

And how about updating the version to <nextversion>-SNAPSHOT after doing all this, to differentiate in-progress versions of the software from the final tagged releases?

[MasterOdin]

GH will automatically create a new tag as part of a release so it's not really necessary to manually do that. Going the route of creating and pushing a tag would be to then I guess generate a GH release automatically (via GH action), but the end result is the same.

[apjanke]

Gotcha. Nevermind then.

[MasterOdin]

Documented the process at https://github.com/asciidoc/asciidoc-py3/wiki/Release-Process. Agreed on the -SNAPSHOT bit. I think we've got the rest of the discussion in this issue either done or broken out into other specific issues.