Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set up own F-Droid repo to push urgent updates to users #1981

Closed
AltNico opened this issue Jan 1, 2019 · 20 comments
Closed

Set up own F-Droid repo to push urgent updates to users #1981

AltNico opened this issue Jan 1, 2019 · 20 comments

Comments

@AltNico
Copy link

@AltNico AltNico commented Jan 1, 2019

At 35C3 there was a F-Droid meetup where @theScrabi showed up and explained your need at NewPipe to sometimes quickly push updates to F-Droid. This is totally understandable and we at F-Droid would really like to offer quicker updates through our infrastructure, but at the moment we can't improve publication time without reducing F-Droid's security standards.

To give some insights: as you all know, at F-Droid everything gets built from source. Unlike Google Play, this means that the building process isn't decentralized at developers' infrastructure, but we rather have a central place where everything is built. This is a great target for people to infiltrate F-Droid's infrastructure and get onto hundreds of thousands of people's devices. Therefore, we have quite high security standards here and run builds only on fully trusted hardware, maintained by fully trusted F-Droid members.

There are plans to decentralize F-Droid's build infrastructure to let builds run on multiple semi-trusted servers run by semi-trusted F-Droid members, but those plans require a lot of work and aren't reachable within the next months. Anyway, what's needed for them are apps to be built reproducibly. So most of the work proposed in this issue needs to be done anyway when F-Droid starts to offer quicker updates.

What I propose to NewPipe (and any other app that wants to distribute quicker updates for apps installed through F-Droid) mainly consists of two parts:

  • Set up a F-Droid repo where you publish official builds of NewPipe that can be reproducibly built by F-Droid
  • Include F-Droid's update channels library into NewPipe to inform users about urgent updates

Let me explain this a little bit further.

People around F-Droid worked hard to make reproducible builds possible. What this means is that F-Droid tries to build an app and then compares this build with the app provided by upstream. If those builds match, F-Droid publishes the upstream .apk. As of now, there are two reproducibly built apps in F-Droid: Öffi thanks to @schildbach's and Briar thanks to @grote's work.

This alone doesn't solve F-Droid's slow publication time yet. It still needs the same amount of time to publish apps, in some cases it could even take more time if there are problems with apps' reproducibility.

However, one important aspect changed: the apps are now signed by upstream, meaning upstream can publish updates which can be installed seamlessly by users.

Here, F-Droid's update channels come into play: if F-Droid for whatever reason needs too long to publish an update or upstream wants to publish an urgent update almost instantly, that update channel library can be used to notify users about an available update that's so far not yet in F-Droid's official repo, but which can be installed from upstream's repo.

The implementation and UX would be like this:

  • On the app's first run, users can opt-in to get notified by upstream about urgent updates.
  • The app then regularly checks if there's an urgent update available by upstream. This can be a simple JSON file inside NewPipe's source code repo which contains the version of the app that every user should at least have.
  • If that version is newer than the version installed on the user's device, F-Droid's update channel runs and notifies the user about an update. The user can then click on that notification which results in the update being downloaded. It can then be installed with another click, opening Android's package manager. The UX is the same like inside F-Droid's app, only that it's inside NewPipe and doesn't touch F-Droid at all.

Be aware that implementing F-Droid's update channel could result in the app being kicked out of Google Play. This isn't a problem for NewPipe but I state it here for everyone else reading this. Even if the mechanism is deactived, an app that includes update channel's code violates Google Play's guidelines. For example, Öffi got kicked out of Google Play for including a deactived code which would show a donation button to its users.

By using F-Droid's update channels, you also get another feature for free: when people install .apks from F-Droid without installing the F-Droid app, they normally don't get updates for it. With that library, they would get notified about updates and could install it conveniently with some taps on the popping up notification.

Of course you can also use this F-Droid repo of NewPipe for more cool stuff. People can add it to F-Droid like any other repo and you could push beta and nightly builds to that repo, with those builds automatically checking for updates and proposing them to users all the time.

Some more interesting links:

  • At verification.f-droid.org you can see if your app is already reproducible and what's missing in case it's not. Just search for NewPipe and watch out for the build's date.
  • The build scripts of Briar and Öffi. Note the Binaries entry and the use of disorderfs at Briar's side. This is an essential hack because as of now, it's not possible to build Android apps reproducibly with standard Android SDK tools! (This was reported to Google several months ago, but so far they didn't react.)
  • On GitLab F-Droid's nightly repo is hosted.

The F-Droid documentation also contains a lot of useful information:

I'm CCing @eighthave and @Bubu as they were also at the meetup and possibly know more about my proposed stuff. Also, so far nobody has done this. I'm proposing it to NewPipe because it doesn't have that problem with being on Google Play and could serve as an example app for other apps. This is of course also interesting for Conversations, where @iNPUTmice attended the F-Droid meetup, but they have the problem of being on Google Play and so far don't publish official builds outside of Google Play.


@schildbach
Copy link

@schildbach schildbach commented Jan 1, 2019

Thanks for this extensive writeup! I'd like to add that Bitcoin Wallet is reproducable as well and it's interesting for the fact that it uses the backwards-compatible way with dual signatures. Existing users will get updates signed by F-Droid; new installs will get the developer signed version.

The configuration metadata can be found in this directory: https://gitlab.com/fdroid/fdroiddata/tree/master/metadata/de.schildbach.wallet/signatures

@eighthave
Copy link
Contributor

@eighthave eighthave commented Jan 2, 2019

@uniqx and I did a lot of work to get the multi-signature support working, happy to hear that it is appreciated

@licaon-kter
Copy link
Contributor

@licaon-kter licaon-kter commented Jan 19, 2019

This is the right time...

@TheAssassin
Copy link
Member

@TheAssassin TheAssassin commented Jan 24, 2019

We've set up a demo repository a couple of hours ago, based on the binaries from the GitHub release page. Of course, those binaries don't have the same quality as the original F-Droid ones, therefore I plan to import the existing F-Droid builds and cross-sign them as suggested by @schildbach. Eventually, we might even set up a continuous integration system on infrastructure and build proper binaries from the metadata using the F-Droid tools.

Infos will follow soon!

@eighthave
Copy link
Contributor

@eighthave eighthave commented Jan 24, 2019

@theScrabi
Copy link
Member

@theScrabi theScrabi commented Jan 24, 2019

@TobiGr you where interested in this topic. Her would be the thread to go :)

@TheAssassin
Copy link
Member

@TheAssassin TheAssassin commented Jan 24, 2019

The repo URL I mentioned last night is https://archive.newpipe.net/fdroid/repo. We've tested it a bit, and it seems to work. Right now, it only provides the latest GitHub binaries, and you'll have to remove the old version to see the new one. We'll try to make sure that from then on, we also provide F-Droid binaries and ensure smooth upgrades by cross-signing those packages and including them in our repository. I guess I'll bother @eighthave once I find some time to set this up.

The repository is somewhat more secure than a completely automated one, as I update it manually, and the builds I want to include from F-Droid will be signature-checked before inclusion. Also, old GitHub builds will be dropped as soon as "better" versions from F-Droid are available. The repository is running on private infrastructure (i.e., we control it).

@TheAssassin
Copy link
Member

@TheAssassin TheAssassin commented Jan 24, 2019

@theScrabi can you please investigate whether we can make use of https://gitlab.com/fdroid/update-channels to allow users to update more quickly next time?

@eighthave
Copy link
Contributor

@eighthave eighthave commented Jan 24, 2019

@theScrabi
Copy link
Member

@theScrabi theScrabi commented Jan 24, 2019

We just merged the app updated notification PR. This one has the purpose to notify the user about new releases of newpipe at the release page of github. So I think with this we already have quite a need solution. While Update channels will definitely be an improvement, I think our main goal atm should be to get reproducable builds running.

@TheAssassin
Copy link
Member

@TheAssassin TheAssassin commented Jan 24, 2019

Reproducible as in "exactly compatible to F-Droid binary-wise"?

@TheAssassin
Copy link
Member

@TheAssassin TheAssassin commented Jan 24, 2019

@theScrabi I think the update channels lib is worth implementing even if we will provide "only" one channel for now. The notifications are more reliable, as they are based on the F-Droid repository. As @eighthave said, it could use our own repository.

@yausername
Copy link
Member

@yausername yausername commented Dec 19, 2019

@eighthave
I can't find recent newpipe releases on https://verification.f-droid.org
Are these verification builds not getting triggered for newpipe?

@eighthave
Copy link
Contributor

@eighthave eighthave commented Dec 19, 2019

@yausername
Copy link
Member

@yausername yausername commented Dec 19, 2019

I still can't find builds after versionCode 69. Is https://verification.f-droid.org updated via some jenkins job?

@TobiGr
Copy link
Member

@TobiGr TobiGr commented Jan 6, 2020

It looks like the latest build was verified. This means, we can proceed to set up our own repo. We need to upload the latest binaries and update the indices.

@eighthave
Copy link
Contributor

@eighthave eighthave commented Jan 7, 2020

@opusforlife2
Copy link
Collaborator

@opusforlife2 opusforlife2 commented Jan 23, 2020

@TobiGr What work is left to do here apart from what eighthave mentioned?

@opusforlife2
Copy link
Collaborator

@opusforlife2 opusforlife2 commented Aug 4, 2020

Done, finally. :)

@TobiGr TobiGr closed this Aug 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
11 participants