Requesting clarification on Kiwi build process

[ubergeek77]

Hey there, not filing this issue to start an argument, as I know source code has been a point of contention for a while, I'm just trying to understand how the builds work.

It looks like this repository hasn't been updated in 3 months, but builds continue to be made for the latest versions of Chromium. There was even a build today.

If we look at the Actions for this repository, it seems a bit odd.

https://github.com/kiwibrowser/src.next/actions/runs/6859307693/job/18651412292

• The source code is cloned, but no files are actually used for the build.
• The only file present in the source code that is referenced directly is args.gn, but the Actions step that refers to it is actually incorrect and can't find the file, then ends up uploading a blank file instead.
• Pre-built Chromium APKs are downloaded via rclone from a URL that is censored from the build
• These builds are uploaded as-is and not modified by the build at all except for signing them
• The step titled Calling buildbot has a hardcoded false condition preventing it from ever running, and would have curl'd ipinfo.io for some reason?

src.next/.github/workflows/build_and_sign_release_apk.yml

Lines 125 to 127 in 64f705f

	- name: Calling buildbot
	if: false
	run: curl ipinfo.io

It's not clear what exactly what the function of this repository's source code is for these builds, nor is it clear how to actually build the latest version of Kiwi from this repository. But I think some users may find it alarming that a "CI/CD" is pulling pre-built binaries from source inside of an Actions workflow that clones code and then never uses it 😬

I am not finger pointing, but at face value, without any context for how this build infrastructure works, this CI/CD appears to be largely performative, and not really the open source CI/CD build process a casual user might think it is. Not saying it IS trying to be performative, but it currently looks that way. I am trusting that there is a rationale for these decisions.

Has this project moved to a closed-source development process? If yes, can this be clearly stated in the Readme?

If not, is there any information on how to build the latest version of Kiwi from source using this repository?

Thanks for your efforts on this browser!

[kiwibrowser]

Hey,

TL;DR: Time is precious and I have limited amount of it but I agree that your vision is better in a pure world.
Kiwi is staying open-source, but expect the GitHub updates to be delayed (but will eventually come), the same way Telegram had such for some time, probably some big clean-up in 2024.

Keep in mind that Kiwi is provided for free and relies on volunteers work.

The long long version (sharing open thoughts):
You are actually quite kind in your approach.
It's a mix of many things.

The buildbot step with ipinfo.io is some left over of a test, likely to see from which datacenter in Azure connections were coming from.

This was to be able to place to binary objects cache as closest to the buildbots as possible.

Regarding the question of the updates of the repo, there is one part technical, one non-tech, and this is a conscious choice.

I just want to share the reasoning behind; I do not say this is the right thing to do, but just a perception of how I see it and the thought process behind.

From a tech-side, building Chromium is super slow and taking lot of space + CPU.
I've tried at some point to go the route you mention, and actually, you can see the steps that have been used for very long time:
https://github.com/kiwibrowser/src (check the workflows)

They perfectly work there, but it has some drawbacks.

My initial vision was to have everything running in GitHub Actions, and I actually had succeeded at some point in the past.
In the past, using that specific workflow, Kiwi had fully transparent build (and was one of the only browser to have end-to-end build), but it had practically no impact (nor positive, nor negative), and at the end, it was a bit sad because it didn't matter so much and nobody actually cared of the efforts (maybe one helpful person noticed in 2 years).

So many many workarounds had to be found (rclone as a cache for .o files) but also some cool magic (like for example, using MacOS workers), and it was cheap and convenient, but it didn't last.

The current setup right now is that I have one big machine, and then I do all the work / experiments then, and then I do a final build (still taking about an hour, because I have to do 4x dev builds + 4x standard builds)

For that reason, it's faster to work on the machine and then eventually sync with the public repo.

At the end I have to take decisions on how I organize my day, and Kiwi still has the place of a fun side-project (though it's a popular one I agree).

I do not have infinite time to work on Kiwi (I need to work on other projects to pay for my everyday life), and I do not have infinite ressources allocated to Kiwi either (and this is a side-consequence of... being ethical, being ethical means no money, because you don't sell data of the users, you don't push crypto, etc.);

I could potentially hire, but I have somehow little hope on the open-source community to spontaneously engage and come to work for free.

It's already very difficult to find skilled people for Chromium projects, so now imagine volunteers that would give hours and hours for free are even a more rare animal (and I am one of such, I gave away literally 1000+ hours in /src repo).

On Minecraft maybe there is a ratio of 1 dev on about 1'000 players (this is imaginary) who can script LUA or create scenarios, but on a Chromium project it gets probably more around 1 dev for 100'000 users.

I'm still amazed by some of the contributors in Kiwi though, some folks (d3ward, Delphox and others, and translators too!) really made Kiwi a much much much better product, and these people are really, really amazing (d3ward saved the project many times).

At the end, maintaining the repo is more of a net cost, than benefit; so from a pure cold reasoning, it's not a priority in terms of time / reward ratio, at least right now (it's not 5 minutes thing but more like few days of work).

In practice, nearly all the people interested into /src.next forks are almost exclusively the same type of pattern "commercial company from China looking to add crypto to the browser", or something similar, and it's very annoying.

Fixing the build pipeline and maintaining the repo has a cost, we don't talk about money, but about the most precious thing in the world: time.

Hopefully a better 2024, I kinda liked the old system as well.

Arnaud.

[ubergeek77]

A very comprehensive response, thank you very much!

As for getting everything running in Actions, I have had success relying on on-demand Spot runners in AWS triggered via GitHub Actions, and I'm happy to share my Action for that if it's of any use:

https://github.com/ubergeek77/aws-ec2-spot-runner/

The default GitHub Actions VMs are next to useless, and I can't imagine any kind of Chromium build working well on those. The Action I linked would let you launch on-demand AWS instances to to a build which are inexpensive and ephemeral. The VMs destroy themselves after the build is done, or after a failsafe amount of time, to prevent costs from being accrued. It's pretty simple and relies entirely on scripting, so everything that Action does is fully readable. You're more than welcome to take inspiration from it :)

With this as the build system, I imagine it would cost about ~$0.25 to $0.40 for a full build. This would add up, but given your current estimate, I think the worst case scenario would be ~$7 per month, which would allow for 17 builds per month, plenty to maintain Chrome updates with the estimates you provided.

Regarding these comments:

Regarding the question of the updates of the repo, there is one part technical, one non-tech, and this is a conscious choice.
TL;DR: Time is precious and I have limited amount of it but I agree that your vision is better in a pure world.
Kiwi is staying open-source, but expect the GitHub updates to be delayed (but will eventually come), the same way Telegram had such for some time, probably some big clean-up in 2024.

From my perspective, I personally do not believe there is much time commitment required to push the current copy of the code you have to this repo. You are maintaining a separate repo which is now out of band from this one, but that code exists somewhere. The time commitment you mention to keeping this repo up to date is stemmed solely from the decision to do staggered source releases. If you only had one repo, then keeping the source updated and keeping the app updated would be one and the same.

However, I have seen your other discussions about this on Reddit, and I am getting the picture that this staggered source release is done in an attempt to combat cloned apps.

And regarding:

I could potentially hire, but I have somehow little hope on the open-source community to spontaneously engage and come to work for free.

That's true, but also keep in mind that many people are probably being turned away by the current state of the repo, and they're also not able to contribute to the project currently, since the last 3 months worth of releases have sadly been closed source.

For instance, I have a lot of experience with CI/CD management, and I would like to help, but this repository currently gives me almost no information on how Kiwi's current builds work. I can't contribute to your project because the relevant information is behind closed doors. If this repo was fully open source, I would be happy to help you achieve fully hands-off reproducible builds where your only responsibility would be to keep the code updated, but as it stands, your project direction now requires you to manually handle all of this.

I do understand your position though, and I thank you for your response. TL;DR is, these decisions were made to combat cloned Kiwi apps, so Kiwi is only "open source" a few times a year. Admittedly, I do wish that was stated more explicitly in the Readme, but I will admit that your answers to this question when asked have always been consistent, so I don't think you are really hiding anything. But for what it's worth, I think there will be less headache-inducing discussions about Kiwi's source if this was explicitly stated somewhere.

I think that ultimately, a decision will need to be reached on open source vs combating Kiwi forks. Chromium forks are a dime a dozen on the Play Store, and I think that is just the unfortunate reality. You can't really have both.

If you ever change project direction and decide to build fully against this repo, I would be happy to help contribute to the project, and I'm sure many others would too :)

In any case, thanks again for your response!

[foxjaw]

Why not just say "it's closed source for now". There's no shame in admitting so.

[Sunxiaowang1994]

Can't understand why you can't provide complete code that can be compiled locally, it's not very friendly to secondary developers.

[foxjaw]

@Sunxiaowang1994 Don't ask too much. Else these issues will be limited to the collaborators. Like how all dictatorial closed source devs do.
As a secondary dev / primary user, our goal is to choose software which feels trustworthy to us. If we don't feel that, we probably should avoid it & start searching for alternatives.