WIP Implement ahc-ar

[gkaracha]

Closes #649.

[gkaracha]

@TerrorJack This is still a rough first draft but maybe it's time to have a look, tell me if I am going the right way about it? It seems to be working just fine on my Ubuntu, but I have no idea if it would really work on darwin (e.g. fix issue #345).

[gkaracha]

Nah, scrap that. I missed an error from getGNUArchEntries due to name truncation. I'll fix that first and we can discuss options later.

[ProofOfKeags]

I can test on Darwin if you want to fish for what issues you might run into.

[gkaracha]

I can test on Darwin if you want to fish for what issues you might run into.

Ah, that'd be great! I will ping when the patch is ready for testing.

[gkaracha]

OK, @TerrorJack you can have a look now. We still have to decide how much we want to deviate from what GNU ar does. The patch currently

• just ignores all arguments but the first .a file and the list of .o files. Shall we warn or do something with the rest?
• does not parse quoted arguments correctly (e.g. "this is a filename with spaces in it.o"), it uses words to space-separate the arguments. Do we need that?
• parses @file arguments recursively (as specified by man ar). Do we need that behavior?

[TerrorJack]

@gkaracha

• No need to warn, since ahc-ar is only called by Cabal and we know exactly how it is called
• Use lines instead of words, see https://haskell-code-explorer.mfix.io/package/Cabal-2.4.1.0/show/Distribution/Simple/Program/ResponseFile.hs#L34 for how the response file is generated.
• As discussed earlier, the recursive parsing behavior should be removed for simplicity. There's no recursion in Cabal-generated response files.

[TerrorJack]

One place we forgot to modify: the boot.sh boot script compiles libraries in two ways: either calling ahc-cabal, or calling a Setup program and completing the configure/build/install steps manually. This PR only affects ahc-cabal so far; we also need to modify Asterius.Boot and add --with-ar=<path to ahc-ar> to the default configure options.

[gkaracha]

@TerrorJack

see https://haskell-code-explorer.mfix.io/package/Cabal-2.4.1.0/show/Distribution/Simple/Program/ResponseFile.hs#L34 for how the response file is generated.

Nice, thanks for the source. This means that we have to undo the effects of escapeResponseFileArg too, right?

[TerrorJack]

I don't think there's need to unescape object file paths for now. The bizzare object file names have never occured in the field. If later we encounter a crash in third party libraries we can fix this up.

[TerrorJack]

@gkaracha Please cherry pick 8bd5b82 on this branch and re-test. For unknown reasons, --with-ar= must be hard-wired in boot.sh. If you add -v3 to configure options you can show that the system ar is still used previously.

[gkaracha]

@gkaracha Please cherry pick 8bd5b82 on this branch and re-test.

Done: https://buildkite.com/tweag-1/asterius/builds/609. Fingers crossed.

[TerrorJack]

@gkaracha And also 0085dcf. The patch just gives up the reparsing/combining logic of object files.

The reason for the patch is a segmentation fault in macOS booting as shown in CI. I suspect there might have been an OOM condition, and giving up object recombining seems to make ahc-ar work on macOS. (Although it's failing for something else now)

[TerrorJack]

I apologize for the "good first issue" tag in #649 btw. I just removed it. 🤦

[gkaracha]

@gkaracha And also 0085dcf. The patch just gives up the reparsing/combining logic of object files.

Yes, I think that's a good idea in general, it is a change in logic we best include in a separate PR.

The reason for the patch is a segmentation fault in macOS booting as shown in CI. I suspect there might have been an OOM condition, and giving up object recombining seems to make ahc-ar work on macOS. (Although it's failing for something else now)

Hmmmm, something similar is blocking #622 (it also OOM for 2 or more cores). If/When we find the source of this I will revisit #622 to see if it's unblocked.

I apologize for the "good first issue" tag in #649 btw. I just removed it.

No worries about that! (but still, thanks for removing 😛)

[TerrorJack]

@ProofOfKeags This PR solves the ar problem on macOS, but not sufficient to fully fix macOS support; there's another error as in https://app.circleci.com/pipelines/github/tweag/asterius/1697/workflows/97316b3c-006c-4e78-9e40-c8e14560563f/jobs/13829.

[ProofOfKeags]

@TerrorJack Do you know the cause of the new error? or does that need to be investigated?

[TerrorJack]

@ProofOfKeags See #674 (comment). It'll take some time to figure out the solution; right now a possible workaround is adding gnu gcc to replace the xcode stuff in PATH in the shell which you call asterius, but I don't have time to test whether it'll actually work.