Keeping F-Droid builds up-to-date

[Iey4iej3]

This post is an effort to facilitate the process of keeping F-Droid builds up-to-date as described.

Refer to https://gitlab.com/fdroid/fdroiddata/issues/1248#note_223417909 to make a cooperation possible.

[micolous]

Thanks, I'm following the thread you mention.

[vilhelmgray]

I updated the Fdroid au.id.micolous.farebot metadata to autoupdate the Fdroid package whenever a new Metrodroid release is tagged. However, we discovered a minor issue in the Metrodroid build.gradle file: the versionCode and versionName lines are not detected properly.

After investigating the cause, I realized that fdroidserver is expecting the versionCode and versionName in a certain format: it uses regex to parse the lines literally.

Unfortunately, the Metrodroid build.gradle lines are dynamically formed:

metrodroid/build.gradle

Lines 529 to 530 in 7a6b5de

	versionCode "git -C ${projectDir} rev-list --count HEAD".execute().text.trim().toInteger()
	versionName "git -C ${projectDir} describe --always --tags --dirty".execute().text.trim().replaceAll("^v", "")

Since, the fdroidserver code is parsing them literally, it doesn't execute the git commands -- instead the entire line is evaluated as-is and thus rejected as improperly formed.

[vilhelmgray]

Since versionCode is just a git rev-list count from HEAD, it can be reimplemented as a pre-commit git hook. Just put the following bash script in .git/hooks/pre-commit:

#!/bin/sh

versionCode=$(($(git rev-list --count HEAD) + 1))

sed -i 's/\(\s*\)versionCode.*/\1versionCode = '$versionCode'/' build.gradle

exit 0

This will use the local system's sed utility to dynamically replace the versionCode line before a commit.

I'm not sure yet the best way to address the versionName line; a pre-commit hook wouldn't work since it occurs before a git tag is issued. Perhaps versionName should be hardcoded for tagged releases.

[vilhelmgray]

After giving it some more thought, I think the best setup is to hardcode both the versionCode and versionName lines, updating with each tagged release. My rationale is that the current approach is susceptible to collisions and errors.

For example, the current source code release on Github fails to produce a sensible versionCode and versionName since the tarball is not a git repository -- the git commands simply return an error when executed in the project directory.

However, even if someone clones the Metrodroid git repository, the git commands to determine versionCode and versionName are not particularly robust. For example, a shallow clone (--depth=1) will result in a versionCode of 1, while versionName is left without a git tag prefix.

Similar issues arise if executed from a branch, where versionCode can collide with a versionCode in the master branch. The pre-commit git hook wouldn't help in the case of pull requests either since multiple pull requests may share the same versionCode update.

[vilhelmgray]

Metrodroid references a Maven repo that is untrusted by Fdroid:

metrodroid/build.gradle

Line 7 in 3868662

maven { url "https://kotlin.bintray.com/kotlinx" }

metrodroid/jvmcli/build.gradle

Line 7 in 3868662

maven { url "https://kotlin.bintray.com/kotlinx" }

metrodroid/proto/build.gradle

Line 5 in 3868662

maven { url "https://kotlin.bintray.com/kotlinx" }

Is it possible to change the Maven repo to one of the ones allowed by Fdroid?

[micolous]

Thanks for following up on this. Let me address each of the issues here:

I think the best setup is to hardcode both the versionCode and versionName lines, updating with each tagged release. My rationale is that the current approach is susceptible to collisions and errors.

I disagree with this -- the reason that I've got this automation in place is to make it so that I only need to update one thing; the git tag info. :)

This feels entirely like an F-Droid build system issue -- Gradle is a DSL and effectively you can put in arbitrary anything (so parsing with sed is bad).

The way to solve this properly is a Gradle plugin that can export metadata from the android Gradle plugin (or, this might be possible with standard tasks).

it can be reimplemented as a pre-commit git hook.

These are more error-prone, and you have the trouble of needing to tag a version and capture the version "atomically". This way I can make several "tagged releases" and easily change that tag before I push it out publicly in case there is some major problem with it.

For example, the current source code release on Github fails to produce a sensible versionCode and versionName since the tarball is not a git repository -- the git commands simply return an error when executed in the project directory.

This is not the only problem with GitHub's tarballs, they also fail to enumerate submodules correctly (see #32). So hard coding to "also" fix that doesn't solve all of the problems with those tarballs. Ultimately, I consider them totally broken and explicitly advise against using them.

Metrodroid references a Maven repo that is untrusted by Fdroid (kotlinx)

I think we may be blocked on some iOS specific issues that are keeping us from using a newer version of kotlinx.serialization (that is available on jcenter and/or Maven Central) -- @phcoder would have more context on this.

This also impacts kotlinio -- the release we're using is missing in Maven Central

[vilhelmgray]

This feels entirely like an F-Droid build system issue -- Gradle is a DSL and effectively you can put in arbitrary anything (so parsing with sed is bad).

The way to solve this properly is a Gradle plugin that can export metadata from the android Gradle plugin (or, this might be possible with standard tasks).

That's a fair point. I think you're right, this should probably be something fixed upstream on the fdroidserver side. Luckily this isn't much of a show-stopper since we can manually bump versions easily until they add support.

Metrodroid references a Maven repo that is untrusted by Fdroid (kotlinx)

I think we may be blocked on some iOS specific issues that are keeping us from using a newer version of kotlinx.serialization (that is available on jcenter and/or Maven Central) -- @phcoder would have more context on this.

This also impacts kotlinio -- the release we're using is missing in Maven Central

Is this kotlinx git repo on Github the same one you need: https://github.com/Kotlin/kotlinx-io

I don't know much about maven repos, but if so it may be possible to use JitPack to build and host the release you need.

Finally, the third_party/protobuf submodule causes a build error since that git repo has a binary file in it. That's not your fault since it's the upstream protobuf project setup, but maybe there's a way to provide protobuf via a Maven repo instead or something similar so avoid that issue.

[micolous]

It may be possible to use JitPack to build and host the release you need.

That looks neat – that may be able to help us remove other submodule requirements too. :)

Do you happen to know how this interacts with F-Droid's checks?

The third_party/protobuf submodule causes a build error since that git repo has a binary file in it. That's not your fault since it's the upstream protobuf project setup, but maybe there's a way to provide protobuf via a Maven repo instead or something similar so avoid that issue.

Unfortunately, we need to a copy of the Objective-C sources for Protobuf library for the Protobuf runtime.

For Android and JRE builds we already depend on the protobuf-javalite artefacts from Maven. :)

[vilhelmgray]

It may be possible to use JitPack to build and host the release you need.

That looks neat – that may be able to help us remove other submodule requirements too. :)

Do you happen to know how this interacts with F-Droid's checks?

JitPack is one of the whitelisted repos for FDroid, so if the libraries you need are available on Github (and JitPack builds them successfully), you can use the JitPack maven repo in your build.gradle to pull in those libraries:

allprojects {
	repositories {
		...
		maven { url 'https://jitpack.io' }
	}
}
...
dependencies {
	implementation 'com.github.User:Repo:Tag'
}

No need to change anything for FDroid in that case, so it should be an easy switch to make.

For Android and JRE builds we already depend on the protobuf-javalite artefacts from Maven. :)

So for FDroid the third_party/protobuf submodule is not needed since protobuf-javalite is already provided? In that case this won't be a show-stopper either since we can easily tell the fdroidserver to ignore this submodule.

That's good, that means the only thing we need to get the FDroid build working is to find an alternative for that kotlinx maven repo -- possibly via JitPack, or something else.