Skip to content

Set up CI to run buck2 samples #357

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 29 commits into from
Dec 4, 2024
Merged

Set up CI to run buck2 samples #357

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
722674e
Set up CI to run buck2 samples
nlopezgi Nov 22, 2024
d99837b
Set up CI to run buck2 samples
nlopezgi Nov 22, 2024
ffd2b87
setting permissions
nlopezgi Nov 22, 2024
d77d44f
fix find command
nlopezgi Nov 22, 2024
1043d33
fix find command
nlopezgi Nov 22, 2024
b241a7b
verbose shell
nlopezgi Nov 22, 2024
21db2a3
fix runner id
nlopezgi Nov 22, 2024
494f458
fixing building zstd
nlopezgi Nov 22, 2024
252a68d
fix downloading buck2 binary
nlopezgi Nov 22, 2024
4d50341
fix buck2 version
nlopezgi Nov 22, 2024
c7ff911
nit
nlopezgi Nov 22, 2024
99ed06f
use debian 12
nlopezgi Nov 22, 2024
b70cd31
fix
nlopezgi Nov 22, 2024
2860a92
use statically linked buck2 binary
nlopezgi Nov 22, 2024
1cce485
fix auth header
nlopezgi Nov 22, 2024
6169024
add env for github token
nlopezgi Nov 25, 2024
c3df3f9
remove unnecessary env
nlopezgi Nov 25, 2024
870dce1
test using github auth for EF CI
nlopezgi Nov 27, 2024
b05fc96
test using github auth for EF CI
nlopezgi Nov 27, 2024
9ec249e
test using github auth for EF CI
nlopezgi Nov 27, 2024
97253f5
test using github auth for EF CI
nlopezgi Nov 27, 2024
e8bf516
test acces to container
nlopezgi Nov 27, 2024
80f3e95
testing github auth
nlopezgi Nov 28, 2024
1fa3578
cleaning up now that github auth is working
nlopezgi Nov 28, 2024
a7b06ac
fail on error on buck2 ci setup
nlopezgi Nov 28, 2024
3d19f29
use installed zstd
nlopezgi Dec 2, 2024
fcec9c6
review comments
nlopezgi Dec 4, 2024
9ad4056
move alias declaration to test-buck2.sh
nlopezgi Dec 4, 2024
ec88553
enabling aliases in shell script
nlopezgi Dec 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,3 +102,29 @@ jobs:
OS: linux
REMOTE_EXECUTION: ${{ matrix.remote_execution }}
run: python3 infra/test-all.py

buck2-test:
runs-on:
- self-hosted
- "os=linux"
- "arch=x64"
- "engflow-container-image=docker://645088952840.dkr.ecr.eu-west-1.amazonaws.com/engflow-ci/debian11-dind-x64@sha256:a0f386162b6498b9220a33f33f158fd175d2873ddd6ed30ca67aa33915850e1e"
- "engflow-pool=ci_sysbox_x64"
- "engflow-runtime=sysbox-runc"
- "engflow-runner-id=${{ github.repository_id }}_buck2-test_${{ github.run_id }}_${{ github.run_number }}_${{ github.run_attempt }}"
timeout-minutes: 10

steps:
- uses: actions/checkout@v4

- name: Set up Buck2 binary
shell: bash
run: infra/setup-buck2.sh

- name: Set up authentication
shell: bash
run: infra/auth-buck2.sh

- name: Build and test
shell: bash
run: infra/test-buck2.sh
2 changes: 1 addition & 1 deletion buck2/cpp/.buckconfig
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ digest_algorithms = SHA256
engine_address = <CLUSTER_NAME>.cluster.engflow.com
action_cache_address = <CLUSTER_NAME>.cluster.engflow.com
cas_address = <CLUSTER_NAME>.cluster.engflow.com
http_headers = x-engflow-auth-method:jwt-v0,x-engflow-auth-token:LONG_JWT_STRING
http_headers = <AUTH_HTTP_HEADERS>

[build]
execution_platforms = root//platforms:remote_platform
Expand Down
6 changes: 4 additions & 2 deletions buck2/cpp/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,12 @@ following:
engine_address = <CLUSTER_NAME>.cluster.engflow.com
action_cache_address = <CLUSTER_NAME>.cluster.engflow.com
cas_address = <CLUSTER_NAME>.cluster.engflow.com
http_headers = x-engflow-auth-method:jwt-v0,x-engflow-auth-token:LONG_JWT_STRING
http_headers = <AUTH_HTTP_HEADERS>
```

To obtain the value of `LONG_JWT_STRING`, log into https://<CLUSTER_NAME>.cluster.engflow.com/gettingstarted and use the value of `x-engflow-auth-token` in section `Method 2: JWT`.
To obtain the value of `<AUTH_HTTP_HEADERS>`, log into https://<CLUSTER_NAME>.cluster.engflow.com/gettingstarted and obtain the value of `x-engflow-auth-token` in section `Method 2: JWT`, take note of this value. Then set `AUTH_HTTP_HEADERS` with the value `x-engflow-auth-method:jwt-v0,x-engflow-auth-token:<JWT_TOKEN_FROM_GETTINGSTARTED_PAGE>.

Note for CI runs, the auth method used is [Github Tokens](https://docs.engflow.com/re/config/authentication.html#github-tokens).

### Usage instructions

Expand Down
2 changes: 1 addition & 1 deletion buck2/golang/.buckconfig
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ digest_algorithms = SHA256
engine_address = <CLUSTER_NAME>.cluster.engflow.com
action_cache_address = <CLUSTER_NAME>.cluster.engflow.com
cas_address = <CLUSTER_NAME>.cluster.engflow.com
http_headers = x-engflow-auth-method:jwt-v0,x-engflow-auth-token:LONG_JWT_STRING
http_headers = <AUTH_HTTP_HEADERS>

[build]
execution_platforms = root//platforms:remote_platform
Expand Down
6 changes: 4 additions & 2 deletions buck2/golang/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,9 @@ following:
engine_address = <CLUSTER_NAME>.cluster.engflow.com
action_cache_address = <CLUSTER_NAME>.cluster.engflow.com
cas_address = <CLUSTER_NAME>.cluster.engflow.com
http_headers = x-engflow-auth-method:jwt-v0,x-engflow-auth-token:LONG_JWT_STRING
http_headers = <AUTH_HTTP_HEADERS>
```

To obtain the value of `LONG_JWT_STRING`, log into https://<CLUSTER_NAME>.cluster.engflow.com/gettingstarted and use the value of `x-engflow-auth-token` in section `Method 2: JWT`.
To obtain the value of `<AUTH_HTTP_HEADERS>`, log into https://<CLUSTER_NAME>.cluster.engflow.com/gettingstarted and obtain the value of `x-engflow-auth-token` in section `Method 2: JWT`, take note of this value. Then set `AUTH_HTTP_HEADERS` with the value `x-engflow-auth-method:jwt-v0,x-engflow-auth-token:<JWT_TOKEN_FROM_GETTINGSTARTED_PAGE>.

Note for CI runs, the auth method used is [Github Tokens](https://docs.engflow.com/re/config/authentication.html#github-tokens).
2 changes: 1 addition & 1 deletion buck2/python/.buckconfig
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ digest_algorithms = SHA256
engine_address = <CLUSTER_NAME>.cluster.engflow.com
action_cache_address = <CLUSTER_NAME>.cluster.engflow.com
cas_address = <CLUSTER_NAME>.cluster.engflow.com
http_headers = x-engflow-auth-method:jwt-v0,x-engflow-auth-token:LONG_JWT_STRING
http_headers = <AUTH_HTTP_HEADERS>

[build]
execution_platforms = root//platforms:remote_platform
Expand Down
6 changes: 4 additions & 2 deletions buck2/python/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,9 @@ following:
engine_address = <CLUSTER_NAME>.cluster.engflow.com
action_cache_address = <CLUSTER_NAME>.cluster.engflow.com
cas_address = <CLUSTER_NAME>.cluster.engflow.com
http_headers = x-engflow-auth-method:jwt-v0,x-engflow-auth-token:LONG_JWT_STRING
http_headers = <AUTH_HTTP_HEADERS>
```

To obtain the value of `LONG_JWT_STRING`, log into https://<CLUSTER_NAME>.cluster.engflow.com/gettingstarted and use the value of `x-engflow-auth-token` in section `Method 2: JWT`.
To obtain the value of `<AUTH_HTTP_HEADERS>`, log into https://<CLUSTER_NAME>.cluster.engflow.com/gettingstarted and obtain the value of `x-engflow-auth-token` in section `Method 2: JWT`, take note of this value. Then set `AUTH_HTTP_HEADERS` with the value `x-engflow-auth-method:jwt-v0,x-engflow-auth-token:<JWT_TOKEN_FROM_GETTINGSTARTED_PAGE>.

Note for CI runs, the auth method used is [Github Tokens](https://docs.engflow.com/re/config/authentication.html#github-tokens).
9 changes: 9 additions & 0 deletions infra/auth-buck2.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#!/usr/bin/env bash
set -xe

# Modify the buck2/**/.buckconfig files to set the test cluster to opal.
find ./ -type f -name '.buckconfig' -exec sed -i 's/<CLUSTER_NAME>/opal/g' {} \;

# Modify the buck2/**/.buckconfig files to set the "Authorization=Bearer $GITHUB_TOKEN" in the http_headers.
find ./ -type f -name '.buckconfig' -exec sed -i 's/<AUTH_HTTP_HEADERS>/Authorization:Bearer $GITHUB_TOKEN/g' {} \;

12 changes: 12 additions & 0 deletions infra/setup-buck2.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#!/usr/bin/env bash
set -xe

# Get the Buck2 binary.
curl -L -O https://github.com/facebook/buck2/releases/download/latest/buck2-x86_64-unknown-linux-musl.zst

# Unpack the binary.
unzstd buck2-x86_64-unknown-linux-musl.zst

# Change its name and make it executable.
mv buck2-x86_64-unknown-linux-musl buck2-exe
chmod +x buck2-exe
23 changes: 23 additions & 0 deletions infra/test-buck2.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#!/usr/bin/env bash
set -xe
shopt -s expand_aliases
Copy link
Contributor

@TheGrizzlyDev TheGrizzlyDev Dec 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could have also used a function FYI

Copy link
Contributor Author

@nlopezgi nlopezgi Dec 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I will consider that option in the general case. In this case I think using a function might be too much, but if the script starts getting more complex as we add more samples I will switch to using a function.


alias buck2="$(realpath buck2-exe)"

# Run cpp example
cd buck2/cpp
buck2 build //:cpp_lib
buck2 test //:cpp_test
cd ..

# Run python example
cd python
buck2 build //main:check_main
buck2 test //hello:hello_unittest_test
cd ..

# Run go example
cd golang
buck2 build //go:hello
buck2 test //go/greeting:greeting_test
cd ..
Loading