Skip to content

chore: use matrix for the CI APIC-255 #75

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 51 commits into from
Jan 13, 2022
Merged

chore: use matrix for the CI APIC-255 #75

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
291705a
first try
millotp Jan 10, 2022
887dfe7
escape
millotp Jan 10, 2022
8d34990
json
millotp Jan 10, 2022
8339cee
try things out
millotp Jan 11, 2022
44f0382
generate array
millotp Jan 11, 2022
4a7c0a9
quotes
millotp Jan 11, 2022
ff018c6
jq
millotp Jan 11, 2022
4884ce9
log
millotp Jan 11, 2022
a6a237b
logic
millotp Jan 11, 2022
cca5039
forgort s on output
millotp Jan 11, 2022
ce70e8d
try toJSON
millotp Jan 11, 2022
cebc54e
try with array
millotp Jan 11, 2022
34db1f0
using openapitools.json
millotp Jan 11, 2022
29e01e7
test with git
millotp Jan 11, 2022
e0449cb
typo
millotp Jan 11, 2022
50254f9
try with empty array
millotp Jan 11, 2022
46a5882
skip if no specs
millotp Jan 11, 2022
6d126a2
fix
millotp Jan 11, 2022
b481de7
swap 1 and 0
millotp Jan 11, 2022
7b2af63
log run
millotp Jan 11, 2022
47c0e61
condition
millotp Jan 11, 2022
8eb4776
working?
millotp Jan 11, 2022
c7a7bef
add clients
millotp Jan 11, 2022
de11397
messed up
millotp Jan 11, 2022
b1382d6
use json
millotp Jan 11, 2022
f2318f9
argjson
millotp Jan 11, 2022
94e9916
log
millotp Jan 11, 2022
7aca413
iterate
millotp Jan 11, 2022
912606a
simple iteration
millotp Jan 11, 2022
1ef1c9e
compact
millotp Jan 11, 2022
8783570
escape
millotp Jan 11, 2022
b43aa35
try log
millotp Jan 11, 2022
68d8e97
try fixing json bug
millotp Jan 11, 2022
c7e0289
change name
millotp Jan 11, 2022
e4b968e
remove quotesé
millotp Jan 11, 2022
7767984
this is taking way too long
millotp Jan 11, 2022
6e35b59
check
millotp Jan 11, 2022
6020cfa
add java
millotp Jan 11, 2022
39f2862
merge
millotp Jan 12, 2022
893ec6f
needs ?
millotp Jan 12, 2022
1b0a198
use local script
millotp Jan 12, 2022
b48ae9e
use -d
millotp Jan 12, 2022
a760f43
log
millotp Jan 12, 2022
f0b1e89
test split
millotp Jan 12, 2022
70549dc
use one echo
millotp Jan 12, 2022
1cf77be
wrong commit
millotp Jan 12, 2022
059481b
remove echo
millotp Jan 12, 2022
1f21520
use script for java too
millotp Jan 12, 2022
ddf5319
review and spec script
millotp Jan 12, 2022
f9c8aaa
merge
millotp Jan 13, 2022
560fead
review
millotp Jan 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
141 changes: 75 additions & 66 deletions .github/actions/setup/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,84 +29,93 @@ runs:

echo "::set-output name=SPECS_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- specs | wc -l)"
echo "::set-output name=COMMON_SPECS_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- specs/common | wc -l)"
echo "::set-output name=SEARCH_SPECS_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- specs/search | wc -l)"
echo "::set-output name=RECOMMEND_SPECS_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- specs/recommend | wc -l)"
echo "::set-output name=QUERY_SUGGESTIONS_SPECS_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- specs/query-suggestions | wc -l)"
echo "::set-output name=PERSO_SPECS_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- specs/personalization | wc -l)"
echo "::set-output name=INSIGHTS_SPECS_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- specs/insights | wc -l)"
echo "::set-output name=ANALYTICS_SPECS_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- specs/analytics | wc -l)"
echo "::set-output name=ABTESTING_SPECS_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- specs/abtesting | wc -l)"

echo "::set-output name=TESTS_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- tests | wc -l)"

echo "::set-output name=SCRIPTS_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- scripts | wc -l)"

echo "::set-output name=JS_CLIENT_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- clients/algoliasearch-client-javascript | wc -l)"
echo "::set-output name=JS_SEARCH_CLIENT_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- clients/algoliasearch-client-javascript/client-search | wc -l)"
echo "::set-output name=JS_RECOMMEND_CLIENT_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- clients/algoliasearch-client-javascript/recommend | wc -l)"
echo "::set-output name=JS_QUERY_SUGGESTIONS_CLIENT_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- clients/algoliasearch-client-javascript/client-query-suggestions | wc -l)"
echo "::set-output name=JS_PERSO_CLIENT_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- clients/algoliasearch-client-javascript/client-personalization | wc -l)"
echo "::set-output name=JS_ANALYTICS_CLIENT_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- clients/algoliasearch-client-javascript/client-analytics | wc -l)"
echo "::set-output name=JS_ABTESTING_CLIENT_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- clients/algoliasearch-client-javascript/client-abtesting | wc -l)"
echo "::set-output name=JS_INSIGHTS_CLIENT_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- clients/algoliasearch-client-javascript/client-insights | wc -l)"
echo "::set-output name=JS_TEMPLATE_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- templates/javascript | wc -l)"

echo "::set-output name=JAVA_CLIENT_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- clients/algoliasearch-client-java-2 | wc -l)"
echo "::set-output name=JAVA_TEMPLATE_CHANGED::$(git diff --shortstat origin/${{ github.base_ref }}..HEAD -- templates/java | wc -l)"

- name: Compute specs matrix
id: spec-matrix
shell: bash
run: |
base_changed=${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 }}

matrix=$(./scripts/ci/create-spec-matrix.sh $base_changed ${{ github.base_ref }})

if [[ $matrix == '{"client":["no-run"]}' ]]; then
run="false"
else
run="true"
fi
Comment on lines +51 to +55
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this part could be done in the scripts, no?

I think we should reduce the logic in the run as it's easier to debug/write in the scripts

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't get it to work with github actions, I cannot split the output properly with any command, maybe bash is different.

Copy link
Member

@shortcuts shortcuts Jan 13, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking again, it would impact adding jq logic to handle double returns, you can skip this comment

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't get it to work with github actions, I cannot split the output properly with any command, maybe bash is different.

My guess would be to return a json object with run and matrix and split the result with jq to set the output, but not sure if it's worth compared to a simple if else


echo "::set-output name=MATRIX::$matrix"
echo "::set-output name=RUN_SPECS::$run"

- name: Compute the JS client build matrix
id: js-matrix
shell: bash
run: |
base_changed=${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.JS_TEMPLATE_CHANGED > 0 }}

matrix=$(./scripts/ci/create-client-matrix.sh javascript $base_changed ${{ github.base_ref }})

if [[ $matrix == '{"client":["no-run"]}' ]]; then
run="false"
else
run="true"
fi

echo "::set-output name=MATRIX::$matrix"
echo "::set-output name=RUN_CLIENT::$run"

- name: Compute the Java client build matrix
id: java-matrix
shell: bash
run: |
base_changed=${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.JAVA_TEMPLATE_CHANGED > 0 }}

matrix=$(./scripts/ci/create-client-matrix.sh java $base_changed ${{ github.base_ref }})

if [[ $matrix == '{"client":["no-run"]}' ]]; then
run="false"
else
run="true"
fi

echo "::set-output name=MATRIX::$matrix"
echo "::set-output name=RUN_CLIENT::$run"

outputs:
# specs variables
RUN_SPECS_SEARCH:
description: Determine if the `specs_search` job should run
value: ${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.SEARCH_SPECS_CHANGED > 0 }}
RUN_SPECS_RECOMMEND:
description: Determine if the `specs_recommend` job should run
value: ${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.RECOMMEND_SPECS_CHANGED > 0 }}
RUN_SPECS_QUERY_SUGGESTIONS:
description: Determine if the `specs_query_suggestions` job should run
value: ${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.QUERY_SUGGESTIONS_SPECS_CHANGED > 0 }}
RUN_SPECS_PERSO:
description: Determine if the `specs_perso` job should run
value: ${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.PERSO_SPECS_CHANGED > 0 }}
RUN_SPECS_INSIGHTS:
description: Determine if the `specs_insights` job should run
value: ${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.INSIGHTS_SPECS_CHANGED > 0 }}
RUN_SPECS_ANALYTICS:
description: Determine if the `specs_analytics` job should run
value: ${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.ANALYTICS_SPECS_CHANGED > 0 }}
RUN_SPECS_ABTESTING:
description: Determine if the `specs_abtesting` job should run
value: ${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.ABTESTING_SPECS_CHANGED > 0 }}

# js client variables
RUN_JS_CLIENT_SEARCH:
description: Determine if the `client_javascript_search` job should run
value: ${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.SEARCH_SPECS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.JS_SEARCH_CLIENT_CHANGED > 0 || steps.diff.outputs.JS_TEMPLATE_CHANGED > 0 }}
RUN_JS_CLIENT_RECOMMEND:
description: Determine if the `client_javascript_recommend` job should run
value: ${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.RECOMMEND_SPECS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.JS_RECOMMEND_CLIENT_CHANGED > 0 || steps.diff.outputs.JS_TEMPLATE_CHANGED > 0 }}
RUN_JS_CLIENT_QUERY_SUGGESTIONS:
description: Determine if the `client_javascript_query_suggestions` job should run
value: ${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.QUERY_SUGGESTIONS_SPECS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.JS_QUERY_SUGGESTIONS_CLIENT_CHANGED > 0 || steps.diff.outputs.JS_TEMPLATE_CHANGED > 0 }}
RUN_JS_CLIENT_PERSO:
description: Determine if the `client_javascript_perso` job should run
value: ${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.PERSO_SPECS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.JS_PERSO_CLIENT_CHANGED > 0 || steps.diff.outputs.JS_TEMPLATE_CHANGED > 0 }}
RUN_JS_CLIENT_ANALYTICS:
description: Determine if the `client_javascript_analytics` job should run
value: ${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.ANALYTICS_SPECS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.JS_ANALYTICS_CLIENT_CHANGED > 0 || steps.diff.outputs.JS_TEMPLATE_CHANGED > 0 }}
RUN_JS_CLIENT_ABTESTING:
description: Determine if the `client_javascript_abtesting` job should run
value: ${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.ABTESTING_SPECS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.JS_ABTESTING_CLIENT_CHANGED > 0 || steps.diff.outputs.JS_TEMPLATE_CHANGED > 0 }}
RUN_JS_CLIENT_INSIGHTS:
description: Determine if the `client_javascript_insights` job should run
value: ${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.INSIGHTS_SPECS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.JS_INSIGHTS_CLIENT_CHANGED > 0 || steps.diff.outputs.JS_TEMPLATE_CHANGED > 0 }}

# java client variables
RUN_JAVA_CLIENT:
description: Determine if the `client_java_*` job should run
value: ${{ github.ref == 'refs/heads/main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.COMMON_SPECS_CHANGED > 0 || steps.diff.outputs.SEARCH_SPECS_CHANGED > 0 || steps.diff.outputs.SCRIPTS_CHANGED > 0 || steps.diff.outputs.JAVA_CLIENT_CHANGED > 0 || steps.diff.outputs.JAVA_TEMPLATE_CHANGED > 0 }}

# cts variables
RUN_SPECS:
description: Whether to run specs or not
value: ${{ steps.spec-matrix.outputs.RUN_SPECS }}

SPECS_MATRIX:
description: Generate the matrix for specs
value: ${{ steps.spec-matrix.outputs.MATRIX }}

RUN_JS:
description: Whether to run js client or not
value: ${{ steps.js-matrix.outputs.RUN_CLIENT }}

JS_MATRIX:
description: Generate the matrix for the Javascript client
value: ${{ steps.js-matrix.outputs.MATRIX }}

RUN_JAVA:
description: Whether to run java client or not
value: ${{ steps.java-matrix.outputs.RUN_CLIENT }}

JAVA_MATRIX:
description: Generate the matrix for the Java client
value: ${{ steps.java-matrix.outputs.MATRIX }}

RUN_CTS:
description: Determine if the `cts` job should run
value: ${{ github.ref_name == 'main' || steps.diff.outputs.GITHUB_ACTIONS_CHANGED > 0 || steps.diff.outputs.SPECS_CHANGED > 0 || steps.diff.outputs.TESTS_CHANGED > 0 || steps.diff.outputs.JS_CLIENT_CHANGED > 0 || steps.diff.outputs.JS_TEMPLATE_CHANGED > 0 || steps.diff.outputs.JAVA_CLIENT_CHANGED > 0 || steps.diff.outputs.JAVA_TEMPLATE_CHANGED > 0 }}
Loading