Non-deterministic builds between two machines #1548
Labels
Comments
|
Apparently sqldelight relies on file order as per underlying file system. I can locally reproduce the issue: produces this output: |
|
It's quite possible, yes! I suspect a lot of Gradle plugins fall into this trap. |
|
@JakeWharton does your reaction mean that it's an acknowledged issue that should get fixed in an upcoming release? Or is more info needed? Should I bake a minimal example instead of using our project? I am glad we can work around this but we can't expect external auditors to work around this, given I have not yet figured out how to hide the disorderfs voodoo inside a Dockerized build script so I would very much appreciate a fix from sqldelight and awareness for the issue, so it doesn't return in a later update. |
|
I certainly acknowledge it and that it needs fixed, yes. We probably just
need to issue a sort() somewhere.
…On Sat, Jan 11, 2020 at 12:13 AM Leo Wandersleb ***@***.***> wrote:
@JakeWharton <https://github.com/JakeWharton> does your reaction mean
that it's an acknowledged issue that should get fixed in an upcoming
release? Or is more info needed? Should I bake a minimal example instead of
using our project?
I am glad we can work around this but we can't expect external auditors to
work around this, given I have not yet figured out how to hide the
disorderfs voodoo inside a Dockerized build script so I would very much
appreciate a fix from sqldelight and awareness for the issue, so it doesn't
return in a later update.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1548?email_source=notifications&email_token=AAAQIELSBHYKXLVEBMUWMZ3Q5FIQLA5CNFSM4KEYENNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIVZTZY#issuecomment-573282791>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAQIEIAFVBEGSMWTY56MSTQ5FIQLANCNFSM4KEYENNA>
.
|
|
How is it going? This is still a huge issue for me and others. |
For me that means reproducibility and thus auditability is not getting enough attention. There are many applications that could do a lot of harm if the binary was infected with a backdoor, yet projects appear to not care. Any release manager of any app could under distress or unknowingly inject bad stuff in a build. My focus is Bitcoin wallets and even there the consciousness is very low. |
|
I really care about this issue and although I thought cashapp has the resources to get this done quickly, my project Walletscrutiny is now also offering a $500 bug bounty. |
|
if there's a diff in the smali i assume theres a diff in the generated kotlin code, are you able to upload that? I assume its the query identifiers but want to confirm that is the case other things that would help push this forward is having a failing test in a PR that we can then fix, it sounds like you're able to reproduce by changing the filesystem ordering, is this something that could be done in one of our gradle integration tests? |
Disorderfs is designed to do such tests. For reproducible test results you would mount the project to an FS sorted alphabetically and to an FS with the sort order reversed. If the result is the same, you should be good. I'm no expert on disorderfs but for reproducing Mycelium Wallet builds I run these lines. Essentially If I remember correctly, |
|
a minimal example/test-case would be helpful to reproduce the bug in a way that easier to debug, maybe running the tests in https://github.com/cashapp/sqldelight/tree/master/sqldelight-gradle-plugin/src/test twice with disorderfs , the first invocation with |
|
The issue is fixed in V2 which is currently in RC version "Version 2.0.0-rc02". it was fixed in version 2.0.0-alpha01: https://github.com/cashapp/sqldelight/releases/tag/2.0.0-alpha01 From changelog:
This is the commit that fix it in V2: for V1 Muun project has a fork of this repo at: with a fix for this issue, this is the commit that add the sorting operation on V1 branch: |
|
@AlecStrong V1 is still in use and does not contain this fix. Please merge it to V1 for those affected. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
#1328 sounds as if sqldelight was supposed to deterministically build but we get diffs in every(?) sqldelight generated file between two devs, both running linux and equal Dockers. In fact, using or not the Docker container makes no difference on the respective machines but somehow we have a huge diff between the two machines.
The code is this commit and the diff is
this.
The text was updated successfully, but these errors were encountered: