Unify version numbers in various packaging scenarios

[paulfantom]

Bug report summary

• makeself produces its own versioning based on date and other factors
• make dist produces artifacts with future versions. Currently it bumps patch part in semantic versioning. This behavior is miss leading as it produces a tarball with a non-existent version.
• .travis/create_artifacts.sh script tries to modify packages
• netdata.spec.in is using non-existent version number
• netdata binary is also using different versioning to store its version number

The easiest way would be to require git tree when building packages and using git describe instead of inventing our own solution to an already solved problem.

As a bonus using git describe gives us a way to easier reproduce bugs when new issues are coming since we can see exactly which revision is used by an end user.

It will break third-party packaging scripts which depend on tarballs without .git directory.

OS / Environment

All

Component Name

Packaging

Steps To Reproduce

1. Run:

./.travis/create_artifacts.sh

2. ls

Expected behavior

Versions are using git describe format, which is the following:
<LAST_TAG>-<NUMBER_OF_COMMITS_SINCE_TAG>-<SCM_SYSTEM_LETTER>-<SHORT_GIT_HASH>
for example commit 2511353 would be looking like this:

v1.12.0-rc0-65-g1c74e65d

git describe is robust enough not to include -<NUMBER_OF_COMMITS_SINCE_TAG>-<SCM_SYSTEM_LETTER>-<SHORT_GIT_HASH> part when current commit is a tagged commit.

[paulfantom]

@cakrit @gmosx @ktsaou please comment on this.

[paulfantom]

For me it makes sense to use a versioning system (git) to set a version of a package 😄

[paulfantom]

Testable proof of concept is in #5051

Also @Ferroin please check this.

[paulfantom]

@FedericoCeratto maybe more input why you think this approach is wrong?

[Ferroin]

Just a couple of thoughts on this:

• We shouldn't need this for actual releases, we should be tagging them (it looks on my end like 1.11 and 1.11.1 never got tagged BTW), and just using regular version numbers. I'd argue that should probably extend to release candidates too, but I'm a bit less certain on that one.
• We can't realistically unify versioning perfectly, every distro has it's own rules for how they indicate any local patches, so we have to expect that we will continue to see odd version numbers, even once we have our own repos (distros that already have Netdata in their repos and keep it up to date are likely to just keep doing so).

[paulfantom]

We are tagging releases and this is not going to change.

it looks on my end like 1.11 and 1.11.1 never got tagged BTW

They are tagged:

• https://github.com/netdata/netdata/tree/v1.11.0
• https://github.com/netdata/netdata/tree/v1.11.1

Since we are using GitHub for releases and we would like to keep it, then releases are connected to tags (each release has a tag). This part is already automated and is already using git underneath. More problematic is how we are dealing right now with automatic updates and "nightly" distribution channel. This obviously cannot be using git tag, but we should have some mechanism to differentiate versions between today and yesterday package.

We can't realistically unify versioning perfectly, every distro has it's own rules for how they indicate any local patches, so we have to expect that we will continue to see odd version numbers,

The whole plan is to distribute it ourselves, so there is no need for any local patches. And in such scenario unification is entirely possible. Right now other distros are packaging only our stable distribution channel and here nothing changes apart from requiring .git directory to find what are you building (which they already should be doing either way).

---

There are TWO main changes:

1. requiring git for determining package version in build system
2. versioning packages for nightly distribution channel (only we are distributing netdata this way)

First is obvious to me, we are using git which is versioning system so we should use it to determine package version. This can be done in a couple of ways, ex. using git tag or git describe and both will give the same result when repo HEAD is on a tagged commit, but the difference lays with nightly (between stable releases) channel. In this case we actually need a commit number for maintenance and debugging purposes, so the best way would be to embed a part of commit hash into a version number and git describe gives us a good and robust way to do this.

As I see it, using git describe gives us more robust and less error prone build and release process as we wouldn't be relying on custom scripts and various magic commands. Removing this magic can actually be seen in #5051. That PR is all what is needed to move us to using git describe.

[ktsaou]

So, the only problem with this, is the lack the git describe when downloading the file without git. Right?

If yes, can we add a file packaging/version.txt with the content of git describe from travis, immediately after merging a PR? If yes, we could use that file instead of git describe during build.

[paulfantom]

If you add a file, then you create a commit and this means that this approach (version in a file) is immediately outdated for builds from master branch. This way you are building package based on a commit N but version string is referencing commit N-1.

In short version.txt file would be ok for building packages from releases, but it fails when building from master.

I am still waiting for messages on why we cannot always rely on git for dynamically determining package version. And I would like to have "production" examples when someone is actually using non-git workflow to build packages with a reasoning why this cannot be done with git. Otherwise I consider non-git workflow as not really needed.

[Ferroin]

I am still waiting for messages on why we cannot always rely on git for dynamically determining package version. And I would like to have "production" examples when someone is actually using non-git workflow to build packages with a reasoning why this cannot be done with git. Otherwise I consider non-git workflow as not really needed.

Gentoo. With very limited exceptions, stuff in the portage tree has to have a source tarball that can be downloaded without needing anything other than wget. Without meeting that requirement, you add extra dependencies on whatever tools you use to fetch the sources and you need special handling in the ebuild. They also prefer to host copies of the source archives themselves so that people can still use the package even if the upstream source just vanishes. Also, it's very unlikely that you will manage to get Netdata removed from the portage tree (not that it being there is a bad thing, they stay up to date and provide a live ebuild (one which builds from the tip of the upstream default VCS branch), which means they are one distro we don't have to worry about covering).

Even ignoring Gentoo and other source-based distros, I personally see exactly zero reason we can't have a check in the version identifying stuff that looks to see if there isn't a .git directory, and just lists the commit as 'local'. That will cover 95% of our potential issues, and if we add an extra check to see if the working directory is clean when the build starts, we can cover the other 5% (and that's easy enough too, just add a + to the commit hash in the version if there are changes from the last commit in the working directory).

[ktsaou]

On my experience:

1. IoT cross-compilation tool chains - Check the forums of various IoT, people are trying to get netdata into routers, NAS boxes, etc. and in some cases having the whole tool chain is a real pain...

2. Old companies tend to have old boxes too. Not everyone in the world uses a supported distro. I have seen many times production systems running obsolete distros (and many cases this is not a problem because the system are not internet connected).

So, yes I agree we could stick to git, but I think there should be a way to compile netdata without git being available. For example, having a N-1 version is still better than no compilation at all. After all this +1 will be just the version increase. Nothing important.

[paulfantom]

ok, so let's use git for majority of stuff and a version file as a fallback. This version file could store a number in a format close to git describe one, but instead of git hash part it could have dev or some other keyword. This way we can also solve a problem of N-1 version number stored. So instead of storing last commit version like v1.12.0-rc1-34-g8dcd801f we can have v1.12.0-rc1-35-dev which is pointing to current commit but doesn't store a commit hash, just how many commits passed since previous tag and such version can be predetermined before creating a commit.

[paulfantom]

Full implementation done in #5051

[wcallgair]

Sorry to necro this but rpm cannot support this naming scheme.

rpmbuild -ts netdata-v1.12.0-rc2-53-ga67afbe.tar.gz error: line 84: Illegal char '-' in: Version: v1.12.0-rc2-53-ga67afbe

[paulfantom]

@wcallgair there is a WIP PR where I try to implement a workaround for RPMs: #5135