Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.github/workflows: adapting last v2 CI changes #2435

Merged
merged 11 commits into from
Dec 22, 2023
Merged
24 changes: 13 additions & 11 deletions .github/workflows/apps/appsec-test-contrib-submodules.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,22 +7,29 @@ set -e
# .github/workflows/appsec.yml.

echo "Running appsec tests for:"
echo " V2_BRANCH=$V2_BRANCH"
echo " GODEBUG=$GODEBUG"
echo " GOEXPERIMENT=$GOEXPERIMENT"
echo " CGO_ENABLED=$CGO_ENABLED"
echo " DD_APPSEC_ENABLED=$DD_APPSEC_ENABLED"
echo " DD_APPSEC_WAF_TIMEOUT=$DD_APPSEC_WAF_TIMEOUT"

function gotestsum_runner() {
report=$1; shift
report=$1; shift; shift
gotestsum --junitfile "$report" -- -v "$@"
}

function docker_runner() {
# ignore the first argument, which is the JUnit report
shift
docker run --platform=$PLATFORM -v $PWD:$PWD -w $PWD -eCGO_ENABLED=$CGO_ENABLED -eDD_APPSEC_ENABLED=$DD_APPSEC_ENABLED -eDD_APPSEC_WAF_TIMEOUT=$DD_APPSEC_WAF_TIMEOUT golang go test -v "$@"
# capture the working directory for the test run
WD=$(realpath "$1"); shift
docker run \
--platform="$PLATFORM" \
-v "$PWD":"$PWD" -w "$WD" \
-eCGO_ENABLED="$CGO_ENABLED" \
-eDD_APPSEC_ENABLED="$DD_APPSEC_ENABLED" \
-eDD_APPSEC_WAF_TIMEOUT="$DD_APPSEC_WAF_TIMEOUT" \
golang go test -v "$@"
}

runner="gotestsum_runner"
Expand All @@ -32,16 +39,11 @@ if [[ "$1" == "docker" ]]; then
[[ -z "$PLATFORM" ]] && PLATFORM="linux/arm64"
fi

$runner "$JUNIT_REPORT.xml" ./appsec/... ./internal/appsec/...
$runner "$JUNIT_REPORT.xml" "." ./appsec/... ./internal/appsec/...

SCOPES=("gin-gonic/gin" "google.golang.org/grpc" "net/http" "gorilla/mux" "go-chi/chi" "go-chi/chi.v5" "labstack/echo.v4")
for SCOPE in "${SCOPES[@]}"; do
contrib=$(basename "$SCOPE")
if [[ "$V2_BRANCH" == "true" ]]; then
cd "./v2/contrib/$SCOPE"
$runner "$JUNIT_REPORT.$contrib.xml" .
cd -
else
$runner "$JUNIT_REPORT.$contrib.xml" "./contrib/$SCOPE/..."
fi
echo "Running appsec tests for contrib/$SCOPE"
$runner "$JUNIT_REPORT.$contrib.xml" "." "./contrib/$SCOPE/..."
done
19 changes: 0 additions & 19 deletions .github/workflows/apps/test-contrib-submodules.sh

This file was deleted.

9 changes: 0 additions & 9 deletions .github/workflows/appsec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@ on:
env:
DD_APPSEC_WAF_TIMEOUT: 5s
JUNIT_REPORT: /tmp/gotestsum-report
V2_BRANCH: false
jobs:
native:
strategy:
Expand Down Expand Up @@ -198,13 +197,5 @@ jobs:

smoke-tests:
uses: DataDog/appsec-go-test-app/.github/workflows/smoke-tests.yml@main
if: inputs.ref != 'refs/heads/v2-dev'
with:
dd-trace-go-version: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}

smoke-tests-v2:
# TODO(darccio): change to main branch on v2 release
uses: DataDog/appsec-go-test-app/.github/workflows/smoke-tests.yml@dario.castane/AIT-3705/dd-trace-go.v2
if: inputs.ref == 'refs/heads/v2-dev'
with:
dd-trace-go-version: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
10 changes: 0 additions & 10 deletions .github/workflows/govulncheck.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,6 @@ on:
- cron: '00 00 * * *'
workflow_dispatch:

env:
V2_BRANCH: false

jobs:
govulncheck-tests:
runs-on: ubuntu-latest
Expand All @@ -34,15 +31,8 @@ jobs:
- name: Run govulncheck
run: govulncheck ./ddtrace/... ./appsec/... ./profiler/... ./internal/...
- name: Run govulncheck-contribs
if: ${{ env.V2_BRANCH != 'true' }}
run: |
# Excluding legacy contrib grpc.v12
go list -f '{{.Dir}}' ./contrib/... | grep -v -e grpc.v12 | while read dir ; do
govulncheck -C $dir .
done
- name: Run govulncheck-contribs (v2)
if: ${{ env.V2_BRANCH == 'true' }}
run: |
find ./v2/contrib -mindepth 2 -type f -name go.mod -exec dirname {} \; | while read dir ; do
govulncheck -C $dir .
done
10 changes: 0 additions & 10 deletions .github/workflows/multios-unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,6 @@ on:
required: true
type: string

env:
V2_BRANCH: false

jobs:
test-multi-os:
runs-on: "${{ (inputs.go-version == '1.19' && inputs.runs-on == 'windows-latest') && 'windows-2019' || inputs.runs-on }}"
Expand All @@ -50,17 +47,10 @@ jobs:
if: inputs.runs-on == 'macos-latest'
run: brew install coreutils
- name: "Runner ${{ matrix.runner-index }}: Test Core and Contrib (No Integration Tests)"
if: ${{ env.V2_BRANCH != 'true' }}
shell: bash
run: |
go list ./... | grep -v -e grpc.v12 -e google.golang.org/api -e sarama -e confluent-kafka-go -e cmemprof | sort >packages.txt
gotestsum --junitfile ${REPORT} -- $(cat packages.txt) -v -coverprofile=coverage.txt -covermode=atomic -timeout 15m
- name: "Runner ${{ matrix.runner-index }}: Test Core and Contrib (No Integration Tests) v2"
if: ${{ env.V2_BRANCH == 'true' }}
shell: bash
run: |
go list ./... | grep -v -e google.golang.org/api -e sarama -e confluent-kafka-go -e cmemprof | sort >packages.txt
gotestsum --junitfile ${REPORT} -- $(cat packages.txt) -v -coverprofile=coverage.txt -covermode=atomic -timeout 15m
- name: Upload the results to Datadog CI App
if: always()
uses: ./.github/actions/dd-ci-upload
Expand Down
21 changes: 1 addition & 20 deletions .github/workflows/parametric-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,28 +28,18 @@ jobs:
group: "APM Larger Runners"
env:
TEST_LIBRARY: golang
V2_BRANCH: false
steps:
- name: Checkout system tests
uses: actions/checkout@v3
if: ${{ env.V2_BRANCH != 'true' }}
with:
repository: 'DataDog/system-tests'
ref: ${{ inputs.ref }}

# TODO(darccio): remove ref on v2 release
- name: Checkout system tests (v2)
uses: actions/checkout@v3
if: ${{ env.V2_BRANCH == 'true' }}
with:
repository: 'DataDog/system-tests'
ref: refs/heads/dario.castane/AIT-3705/dd-trace-go.v2

- name: Checkout dd-trace-go
uses: actions/checkout@v3
with:
ref: ${{ inputs.branch_ref || github.ref }}
path: 'binaries/dd-trace-go'
path: 'binaries/dd-trace-go'

#This It will be removed after merge the system-tests PR: https://github.com/DataDog/system-tests/pull/1948
- name: Checkout dd-trace-go (Deprecated.)
Expand All @@ -64,19 +54,10 @@ jobs:

#This It will be removed after merge the system-tests PR: https://github.com/DataDog/system-tests/pull/1948
- name: Patch dd-trace-go version (Deprecated)
if: ${{ env.V2_BRANCH != 'true' }}
run: |
cd utils/build/docker/golang/parametric/
echo "replace gopkg.in/DataDog/dd-trace-go.v1 => ./dd-trace-go" >> go.mod
go mod tidy

#This It will be removed after merge the system-tests PR: https://github.com/DataDog/system-tests/pull/1948
- name: Patch dd-trace-go version (v2)(Deprecated)
if: ${{ env.V2_BRANCH == 'true' }}
run: |
cd utils/build/docker/golang/parametric/
echo "replace github.com/DataDog/dd-trace-go/v2 => ./dd-trace-go" >> go.mod
go mod tidy

- name: Build runner
uses: ./.github/actions/install_runner
Expand Down
5 changes: 4 additions & 1 deletion .github/workflows/smoke-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ on:
branches:
- '**'

env:
TEST_RESULTS: /tmp/test-results # path to where test results will be saved

jobs:
go-get-u:
# Run go get -u to upgrade dd-trace-go dependencies to their
Expand All @@ -39,7 +42,7 @@ jobs:
cache: true
- name: go get -u
run: |
go get -u $PACKAGES
go get -u -t $PACKAGES
go mod tidy
- name: Compile dd-trace-go
run: go build $PACKAGES
Expand Down
10 changes: 0 additions & 10 deletions .github/workflows/system-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -95,24 +95,14 @@ jobs:
DD_API_KEY: ${{ secrets.DD_API_KEY }}
SYSTEM_TESTS_E2E_DD_API_KEY: ${{ secrets.SYSTEM_TESTS_E2E_DD_API_KEY }}
SYSTEM_TESTS_E2E_DD_APP_KEY: ${{ secrets.SYSTEM_TESTS_E2E_DD_APP_KEY }}
V2_BRANCH: false
name: Test (${{ matrix.weblog-variant }}, ${{ matrix.scenario }})
steps:
- name: Checkout system tests
uses: actions/checkout@v3
if: ${{ env.V2_BRANCH != 'true' }}
with:
repository: 'DataDog/system-tests'
ref: ${{ inputs.ref }}

# TODO(darccio): remove ref on v2 release
- name: Checkout system tests (v2)
uses: actions/checkout@v3
if: ${{ env.V2_BRANCH == 'true' }}
with:
repository: 'DataDog/system-tests'
ref: refs/heads/dario.castane/AIT-3705/dd-trace-go.v2

- name: Checkout dd-trace-go
uses: actions/checkout@v3
with:
Expand Down
13 changes: 1 addition & 12 deletions .github/workflows/unit-integration-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,10 @@ jobs:
with:
ref: ${{ inputs.ref || github.ref }}

- name: Setup Go
uses: ./.github/actions/setup-go
with:
go-version: ${{ inputs.go-version }}

- name: golangci-lint
uses: reviewdog/action-golangci-lint@v2
with:
go_version: ${{ inputs.go-version }}
golangci_lint_version: v1.52.2
fail_on_error: true
reporter: github-pr-review
Expand All @@ -51,7 +47,6 @@ jobs:
env:
TEST_RESULTS: /tmp/test-results # path to where test results will be saved
INTEGRATION: true
V2_BRANCH: false
services:
datadog-agent:
image: datadog/agent:latest
Expand Down Expand Up @@ -193,16 +188,11 @@ jobs:
go-version: ${{ inputs.go-version }}

- name: Test Contrib
if: ${{ env.V2_BRANCH != 'true' }}
run: |
mkdir -p $TEST_RESULTS
PACKAGE_NAMES=$(go list ./contrib/... | grep -v -e grpc.v12 -e google.golang.org/api)
gotestsum --junitfile ${TEST_RESULTS}/gotestsum-report.xml -- $PACKAGE_NAMES -v -race -coverprofile=coverage.txt -covermode=atomic

- name: Test Contrib Submodules
if: always()
run: ./.github/workflows/apps/test-contrib-submodules.sh

- name: Upload the results to Datadog CI App
if: always()
continue-on-error: true
Expand Down Expand Up @@ -253,7 +243,6 @@ jobs:
go test -v ./contrib/google.golang.org/api/...

- name: Testing outlier gRPC v1.2
if: ${{ env.V2_BRANCH != 'true' }}
run: |
# This hacky approach is necessary because running the tests regularly
# do not allow using grpc-go@v1.2.0 alongside sketches-go@v1.0.0
Expand Down
17 changes: 9 additions & 8 deletions .github/workflows/v2-nightly.yml
Original file line number Diff line number Diff line change
@@ -1,53 +1,54 @@
name: v2 Nightly Tests
on:
workflow_dispatch: # manually
schedule:
- cron: "0 0 * * *"

jobs:
appsec-v2:
uses: ./.github/workflows/appsec.yml
uses: DataDog/dd-trace-go/.github/workflows/appsec.yml@v2-dev
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes them tremendously painful to update (since it needs to be on the branch before it can be tested); but unfortunately GitHub offers no better way...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not fond of this too, but you said it: GitHub offers no better way. Thanks for the approval!

secrets: inherit
with:
ref: refs/heads/v2-dev

codeql-v2:
uses: ./.github/workflows/codeql-analysis.yml
uses: DataDog/dd-trace-go/.github/workflows/codeql-analysis.yml@v2-dev
secrets: inherit
with:
ref: refs/heads/v2-dev

govulncheck-v2:
uses: ./.github/workflows/govulncheck.yml
uses: DataDog/dd-trace-go/.github/workflows/govulncheck.yml@v2-dev
secrets: inherit
with:
ref: refs/heads/v2-dev

main-branch-tests-v2:
uses: ./.github/workflows/main-branch-tests.yml
uses: DataDog/dd-trace-go/.github/workflows/main-branch-tests.yml@v2-dev
secrets: inherit
with:
ref: refs/heads/v2-dev

parametric-v2:
uses: ./.github/workflows/parametric-tests.yml
uses: DataDog/dd-trace-go/.github/workflows/parametric-tests.yml@v2-dev
secrets: inherit
with:
ref: refs/heads/v2-dev

smoke-v2:
uses: ./.github/workflows/smoke-tests.yml
uses: DataDog/dd-trace-go/.github/workflows/smoke-tests.yml@v2-dev
secrets: inherit
with:
ref: refs/heads/v2-dev

system-v2:
uses: ./.github/workflows/system-tests.yml
uses: DataDog/dd-trace-go/.github/workflows/system-tests.yml@v2-dev
secrets: inherit
with:
branch_ref: refs/heads/v2-dev

test-apps-v2:
uses: ./.github/workflows/test-apps.yml
uses: DataDog/dd-trace-go/.github/workflows/test-apps.yml@v2-dev
secrets: inherit
with:
ref: refs/heads/v2-dev