updated buildkite pipeline generation

[metaflow]

• fixed build for linux (clang was missing)

• removed /monolithic-.. from build directory - it does not add anything and makes path longer for windows which is not great;

• added env-based configuration to control cache and agent targeting;

• print (s)ccache stats to file not to pullute normal log.

[aeubanks]

the commit title doesn't make sense, can you make it something like "Various fixes to buildkite pipeline generation"

[aeubanks]

how can you set this?

[metaflow]

you can do that when creating a build in buildkite UI or triggering through API with "environment" varaibles. It's not available for "normal" users

[aeubanks]

make alphabetical just to be consistent and look nice

[metaflow]

Done.

[aeubanks]

quoting is inconsistent, I believe all of these don't need quotes
(other files as well)

[metaflow]

Done.

[metaflow]

Test build https://buildkite.com/llvm-project/experimental-premerge-checks/builds/60

[joker-eph]

This seems broken: https://buildkite.com/llvm-project/github-pull-requests/builds/1350#018a8848-f70e-47f4-8aec-0bb48c49a7ca

$ trap 'kill -- $' INT TERM QUIT; .ci/generate-buildkite-pipeline-premerge \| buildkite-agent pipeline upload
  | 2023-09-12 07:26:50 INFO   Reading pipeline config from STDIN
  | 2023-09-12 07:26:50 INFO   Updating BUILDKITE_COMMIT to "6de56a7dc4ff114eb36f42f1dbe8c63e557f061b"
  | Files modified:
  | mlir/include/mlir/Dialect/Affine/IR/AffineOps.h
  | mlir/include/mlir/Dialect/Affine/IR/AffineOps.td
  | mlir/include/mlir/IR/SymbolInterfaces.td
  | mlir/lib/Dialect/Affine/IR/AffineOps.cpp
  | mlir/test/Dialect/Affine/invalid.mlir
  | mlir/test/Dialect/Affine/load-store-invalid.mlir
  | Directories modified:
  | mlir
  | 2023-09-12 07:26:50 FATAL  Pipeline parsing of "(stdin)" failed: line 5: did not find expected alphabetic or numeric character
  | 🚨 Error: The command exited with status 1

This seems to match the - '*_result.json' replaced with - *_result.json

[metaflow]

thanks for the heads up @joker-eph ! That should be fixed now but you need to rebase your PR to see the change. I guess noop commit update should do

[metaflow]

ah you've merged it already 😄

[ldionne]

@metaflow The reason for not adding clang to the monolithic pipeline is that clang now has its own pipeline like libc++. After your patch we will be building and testing clang twice for no reason. Can you remove it again?

[metaflow]

@ldionne w/o clang flang will not compile. Do you also check flang and all dependencies of clang in clang pipeline?

[ldionne]

I think it's fine to add clang to the set of projects in LLVM_ENABLE_PROJECTS for dependency purposes, but we could avoid calling check-clang. I guess it's moot cause we're using check-all right now.

We don't check flang in the Clang pipeline because the Clang pipeline's primary responsibility should be to test Clang, not other downstream projects.

[joker-eph]

I think it's fine to add clang to the set of projects in LLVM_ENABLE_PROJECTS for dependency purposes, but we could avoid calling check-clang. I guess it's moot cause we're using check-all right now.

FYI, the script for premerge generates invocations like:

'./.ci/monolithic-linux.sh "clang;flang;llvm;mlir" "check-flang check-mlir"'

So even though clang is enabled, it shouldn't be tested I think? (we don't want to test LLVM either when it isn't touched)

[ldionne]

The scheduled build script does call check-all, but you're right about the script that generates pre-commit CI pipelines.