-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize build and release pipelines #557
Merged
Merged
Changes from all commits
Commits
Show all changes
81 commits
Select commit
Hold shift + click to select a range
720cfc2
split docker build in stages to minimize size
9dcf921
add builder tag
8a2d90a
use redis image instead of redis-stack
3262ce4
single line apt-get
24bd4a3
WIP - restructuring build pipeline
b4481d1
fix build image
9a7b88a
trying fix for image name
90a8ccf
remove copy bin
7312669
trying save cache after build
7527f17
change image artifact upload method
9d45b8d
re-add caching
68199ea
temp - remove arm arch
8b9375b
rm non-required digests
48e7bb5
rm digest
d017d03
fix cache and path
978d789
checkout repo
c529773
deps
659578b
trying to split compiler image for caching purposes
47fb5cc
trying fix compiler image
191a71f
trying fix compiler image
eefe533
change base image name
053fa7b
trying remove buildx
daad110
trying to get docker to use local image
beed0d2
remove args
e7db9d5
add service port
a4804fc
fix image name
1f4e861
fix cache image
2c31ab7
run tests using compiler image
b71fc6c
remove image type
24cb827
load compiler image
4af1e8f
tag compiler image
90161c9
testing image save
ddeed97
trying oci format
67380ff
type=docker
13885a1
trying builx save
9d54cb0
remove cache
e017340
test remove save
a79f88e
export compiler image
6acf97a
trying pull image first
1f5a57c
re-tag image
23a407c
tag test image
2d82bf2
list images
74c8723
try set base image name
0a80c82
add registry
eb1384c
trying pull image
2e91e59
push image
be0b2ed
fix test dir
683e4b0
add arm64
9c1a7ae
create release image flow
aae8b65
fix branch
1331118
fix file name
e19ae70
fix push
2bd7a46
fix digest folder
66ba934
tag image
1b81535
list images
74b6fd9
image name based on architecture
118bef1
move around steps
099f777
remove cache skip
12a8ad9
fix tar name
9b614c9
fix bool value
07b09a2
Merge branch 'master' into optimize-docker-image
dudizimber 83388b8
- arg order
820301f
release on push to master and release
472ae7f
fix target platform argument
cda49af
fix image ref
b8ae20d
add push to master
55a6368
add base image arg
22fadda
add target platform for test
83f378b
test in both architectures
275e3b0
tests after build
97f0c9b
remove test folder requirement as it uses the compiler image
b0054b4
fix file
742152f
test with image platform
0c15daf
add platform to test image
f13527d
setup qemu for tests in arm
dc40987
- change trigger to pull request
49f6d21
test caching compiler image
bb50d9c
remove lib caching
2c49ec7
Revert "remove lib caching"
c6f113b
Revert "test caching compiler image"
25fc200
Merge branch 'master' into optimize-docker-image
AviAvni File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,169 +1,261 @@ | ||
name: Build on ubuntu | ||
|
||
on: | ||
push: | ||
workflow_dispatch: | ||
release: | ||
types: [published] | ||
pull_request: | ||
branches: [master] | ||
push: | ||
branches: | ||
- master | ||
|
||
concurrency: | ||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} | ||
cancel-in-progress: true | ||
|
||
jobs: | ||
build: | ||
runs-on: ubuntu-latest | ||
container: falkordb/falkordb-build:latest | ||
services: | ||
registry: | ||
image: registry:2 | ||
ports: | ||
- 5000:5000 | ||
strategy: | ||
fail-fast: false | ||
matrix: | ||
platform: | ||
- linux/amd64 | ||
- linux/arm64 | ||
env: | ||
ARCH_SUFFIX: ${{ matrix.platform == 'linux/amd64' && 'x64' || 'arm64v8' }} | ||
steps: | ||
- name: Safe dir | ||
run: git config --global --add safe.directory '*' | ||
|
||
- name: Safe dir | ||
run: git config --global --add safe.directory '*' | ||
|
||
- uses: actions/checkout@v3 | ||
with: | ||
set-safe-directory: '*' | ||
submodules: recursive | ||
|
||
- name: Cache GraphBLAS ${{ matrix.platform }} | ||
if: github.event_name == 'push' | ||
id: cache_graphblas | ||
uses: actions/cache@v3 | ||
with: | ||
path: /FalkorDB/bin/linux-${{ matrix.platform == 'linux/amd64' && 'x64' || 'arm64v8' }}-release/GraphBLAS | ||
key: graphblas-${{ matrix.platform == 'linux/amd64' && 'x64' || 'arm64v8' }}-${{ hashFiles('./deps/GraphBLAS/Include/GraphBLAS.h') }} | ||
|
||
- name: Cache parser ${{ matrix.platform }} | ||
if: github.event_name == 'push' | ||
id: cache_parser | ||
uses: actions/cache@v3 | ||
with: | ||
path: /FalkorDB/bin/linux-${{ matrix.platform == 'linux/amd64' && 'x64' || 'arm64v8' }}-release/libcypher-parser | ||
key: parser-${{ matrix.platform == 'linux/amd64' && 'x64' || 'arm64v8' }}-${{ hashFiles('./deps/libcypher-parser/lib/src/parser.c') }} | ||
|
||
- name: Cache search ${{ matrix.platform }} | ||
if: github.event_name == 'push' | ||
id: cache_search | ||
uses: actions/cache@v3 | ||
with: | ||
path: /FalkorDB/bin/linux-${{ matrix.platform == 'linux/amd64' && 'x64' || 'arm64v8' }}-release/search-static | ||
key: search-${{ matrix.platform == 'linux/amd64' && 'x64' || 'arm64v8' }}-${{ hashFiles('./deps/RediSearch/src/version.h') }} | ||
|
||
# Make sure the working directory is static between runs | ||
# and always the same for the CMake cache | ||
- name: Relocate source | ||
run: | | ||
mkdir -p /FalkorDB | ||
mv $GITHUB_WORKSPACE/* /FalkorDB | ||
|
||
- name: Set up QEMU | ||
uses: docker/setup-qemu-action@v3 | ||
|
||
- name: Set up Docker Buildx | ||
uses: docker/setup-buildx-action@v2 | ||
|
||
- name: Login to Docker Hub | ||
uses: docker/login-action@v3 | ||
with: | ||
username: ${{ secrets.DOCKER_USERNAME }} | ||
password: ${{ secrets.DOCKER_PASSWORD }} | ||
|
||
- name: Build and push | ||
id: build | ||
uses: docker/build-push-action@v5 | ||
with: | ||
context: /FalkorDB | ||
file: /FalkorDB/build/docker/Dockerfile | ||
platforms: ${{ matrix.platform }} | ||
outputs: type=image,name=falkordb/falkordb,push-by-digest=true,name-canonical=true,push=true | ||
|
||
- name: Export digest | ||
run: | | ||
mkdir -p /tmp/digests | ||
digest="${{ steps.build.outputs.digest }}" | ||
touch "/tmp/digests/${digest#sha256:}" | ||
|
||
- name: Upload digest | ||
uses: actions/upload-artifact@v3 | ||
with: | ||
name: digests | ||
path: /tmp/digests/* | ||
if-no-files-found: error | ||
retention-days: 1 | ||
|
||
- name: Copy bin from docker on PUSH | ||
if: github.event_name == 'push' | ||
working-directory: /FalkorDB | ||
run: | | ||
id=$(docker create --platform=${{ matrix.platform }} falkordb/falkordb@${{ steps.build.outputs.digest }}) | ||
docker cp $id:/FalkorDB/bin . | ||
docker rm -v $id | ||
|
||
merge: | ||
runs-on: ubuntu-latest | ||
needs: build | ||
steps: | ||
- name: Download digests | ||
uses: actions/download-artifact@v3 | ||
- uses: actions/checkout@v3 | ||
with: | ||
name: digests | ||
path: /tmp/digests | ||
- name: Set up Docker Buildx | ||
uses: docker/setup-buildx-action@v3 | ||
- name: Login to Docker Hub | ||
uses: docker/login-action@v3 | ||
set-safe-directory: "*" | ||
submodules: recursive | ||
|
||
- name: Cache GraphBLAS ${{ matrix.platform }} | ||
id: cache_graphblas | ||
uses: actions/cache@v3 | ||
with: | ||
username: ${{ secrets.DOCKER_USERNAME }} | ||
password: ${{ secrets.DOCKER_PASSWORD }} | ||
- name: Create manifest list and push on PUSH | ||
if: github.event_name == 'push' | ||
working-directory: /tmp/digests | ||
run: | | ||
docker buildx imagetools create -t falkordb/falkordb:${{ github.ref_name == 'master' && 'master -t falkordb/falkordb:edge' || github.ref_name }} \ | ||
$(printf 'falkordb/falkordb@sha256:%s ' *) | ||
- name: Create manifest list and push on RELEASE | ||
if: github.event_name == 'release' | ||
working-directory: /tmp/digests | ||
path: /FalkorDB/bin/linux-${{ env.ARCH_SUFFIX }}-release/GraphBLAS | ||
key: graphblas-${{ env.ARCH_SUFFIX }}-${{ hashFiles('./deps/GraphBLAS/Include/GraphBLAS.h') }} | ||
|
||
- name: Cache parser ${{ matrix.platform }} | ||
id: cache_parser | ||
uses: actions/cache@v3 | ||
with: | ||
path: /FalkorDB/bin/linux-${{ env.ARCH_SUFFIX }}-release/libcypher-parser | ||
key: parser-${{ env.ARCH_SUFFIX }}-${{ hashFiles('./deps/libcypher-parser/lib/src/parser.c') }} | ||
|
||
- name: Cache search ${{ matrix.platform }} | ||
id: cache_search | ||
uses: actions/cache@v3 | ||
with: | ||
path: /FalkorDB/bin/linux-${{ env.ARCH_SUFFIX }}-release/search-static | ||
key: search-${{ env.ARCH_SUFFIX }}-${{ hashFiles('./deps/RediSearch/src/version.h') }} | ||
|
||
# Make sure the working directory is static between runs | ||
# and always the same for the CMake cache | ||
- name: Relocate source | ||
run: | | ||
docker buildx imagetools create -t falkordb/falkordb:${{ github.event.release.tag_name }} -t falkordb/falkordb:latest \ | ||
$(printf 'falkordb/falkordb@sha256:%s ' *) | ||
- name: Inspect image | ||
mkdir -p /FalkorDB | ||
mv $GITHUB_WORKSPACE/* /FalkorDB | ||
|
||
- name: Set up QEMU | ||
uses: docker/setup-qemu-action@v3 | ||
|
||
- name: Set up Docker Buildx | ||
uses: docker/setup-buildx-action@v2 | ||
with: | ||
# network=host driver-opt needed to push to local registry | ||
driver-opts: network=host | ||
|
||
- name: Build compiler image | ||
id: build_compiler | ||
uses: docker/build-push-action@v5 | ||
with: | ||
context: /FalkorDB | ||
file: /FalkorDB/build/docker/Dockerfile.compiler | ||
platforms: ${{ matrix.platform }} | ||
tags: localhost:5000/falkordb/falkordb-compiler | ||
# outputs: type=docker,dest=/tmp/falkordb-compiler.tar | ||
push: true | ||
build-args: | | ||
TARGETPLATFORM=${{ matrix.platform }} | ||
dudizimber marked this conversation as resolved.
Show resolved
Hide resolved
Comment on lines
+78
to
+89
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The build process for the compiler image is correctly configured. However, consider the previous comment about caching build files instead of the build itself for potentially better efficiency. |
||
|
||
- name: Copy bin from docker for caching | ||
working-directory: /FalkorDB | ||
if: steps.cache_graphblas.outputs.cache-hit == false || steps.cache_parser.outputs.cache-hit == false || steps.cache_search.outputs.cache-hit == false | ||
run: | | ||
docker buildx imagetools inspect falkordb/falkordb:latest | ||
id=$(docker create --platform=${{ matrix.platform }} localhost:5000/falkordb/falkordb-compiler) | ||
docker cp $id:/FalkorDB/bin /FalkorDB/bin | ||
docker rm -v $id | ||
ls -l -R /FalkorDB/bin | ||
|
||
- name: Build tests image | ||
uses: docker/build-push-action@v5 | ||
with: | ||
context: . | ||
file: /FalkorDB/tests/Dockerfile | ||
outputs: type=docker,dest=/tmp/falkordb-tests-${{ env.ARCH_SUFFIX }}.tar | ||
tags: falkordb/falkordb-tests | ||
platforms: ${{ matrix.platform }} | ||
build-args: | | ||
BASE_IMAGE=localhost:5000/falkordb/falkordb-compiler | ||
TARGETPLATFORM=${{ matrix.platform }} | ||
|
||
- name: Upload tests image | ||
uses: actions/upload-artifact@v3 | ||
with: | ||
name: falkordb-tests-${{ env.ARCH_SUFFIX }} | ||
path: /tmp/falkordb-tests-${{ env.ARCH_SUFFIX }}.tar | ||
if-no-files-found: error | ||
|
||
- name: Build base image | ||
id: build | ||
uses: docker/build-push-action@v5 | ||
with: | ||
context: /FalkorDB | ||
file: /FalkorDB/build/docker/Dockerfile | ||
platforms: ${{ matrix.platform }} | ||
outputs: type=docker,dest=/tmp/falkordb-${{ env.ARCH_SUFFIX }}.tar | ||
tags: falkordb/falkordb-${{ env.ARCH_SUFFIX }} | ||
build-args: | | ||
BASE_IMAGE=localhost:5000/falkordb/falkordb-compiler | ||
TARGETPLATFORM=${{ matrix.platform }} | ||
|
||
- name: Upload image | ||
uses: actions/upload-artifact@v3 | ||
with: | ||
name: falkordb-${{ env.ARCH_SUFFIX }} | ||
path: /tmp/falkordb-${{ env.ARCH_SUFFIX }}.tar | ||
if-no-files-found: error | ||
|
||
unit-tests: | ||
needs: build | ||
runs-on: ubuntu-latest | ||
container: falkordb/falkordb:${{ github.ref_name }} | ||
needs: merge | ||
strategy: | ||
fail-fast: false | ||
matrix: | ||
platform: | ||
- linux/amd64 | ||
- linux/arm64 | ||
env: | ||
ARCH_SUFFIX: ${{ matrix.platform == 'linux/amd64' && 'x64' || 'arm64v8' }} | ||
steps: | ||
- name: Download image | ||
uses: actions/download-artifact@v3 | ||
with: | ||
name: falkordb-tests-${{ env.ARCH_SUFFIX }} | ||
path: /tmp | ||
|
||
- name: Set up QEMU | ||
if: ${{ matrix.platform == 'linux/arm64' }} | ||
uses: docker/setup-qemu-action@v3 | ||
|
||
- name: Load image | ||
id: load_image | ||
run: | | ||
docker load --input /tmp/falkordb-tests-${{ env.ARCH_SUFFIX }}.tar | ||
|
||
- name: Unit tests | ||
working-directory: /FalkorDB | ||
run: make unit-tests CLEAR_LOGS=0 | ||
run: | | ||
docker run --platform ${{ matrix.platform }} --rm falkordb/falkordb-tests make CLEAR_LOGS=0 unit-tests | ||
|
||
flow-tests: | ||
needs: build | ||
runs-on: ubuntu-latest | ||
container: falkordb/falkordb:${{ github.ref_name }} | ||
needs: merge | ||
strategy: | ||
fail-fast: false | ||
matrix: | ||
platform: | ||
- linux/amd64 | ||
- linux/arm64 | ||
env: | ||
ARCH_SUFFIX: ${{ matrix.platform == 'linux/amd64' && 'x64' || 'arm64v8' }} | ||
steps: | ||
- name: Download image | ||
uses: actions/download-artifact@v3 | ||
with: | ||
name: falkordb-tests-${{ env.ARCH_SUFFIX }} | ||
path: /tmp | ||
|
||
- name: Set up QEMU | ||
if: ${{ matrix.platform == 'linux/arm64' }} | ||
uses: docker/setup-qemu-action@v3 | ||
|
||
- name: Load image | ||
id: load_image | ||
run: | | ||
docker load --input /tmp/falkordb-tests-${{ env.ARCH_SUFFIX }}.tar | ||
|
||
- name: Flow tests | ||
working-directory: /FalkorDB | ||
run: make CLEAR_LOGS=0 PARALLEL=4 flow-tests | ||
run: | | ||
docker run --platform ${{ matrix.platform }} --rm falkordb/falkordb-tests make CLEAR_LOGS=0 PARALLEL=4 flow-tests | ||
|
||
tck-tests: | ||
needs: build | ||
runs-on: ubuntu-latest | ||
container: falkordb/falkordb:${{ github.ref_name }} | ||
needs: merge | ||
strategy: | ||
fail-fast: false | ||
matrix: | ||
platform: | ||
- linux/amd64 | ||
- linux/arm64 | ||
env: | ||
ARCH_SUFFIX: ${{ matrix.platform == 'linux/amd64' && 'x64' || 'arm64v8' }} | ||
steps: | ||
- name: Download image | ||
uses: actions/download-artifact@v3 | ||
with: | ||
name: falkordb-tests-${{ env.ARCH_SUFFIX }} | ||
path: /tmp | ||
|
||
- name: Set up QEMU | ||
if: ${{ matrix.platform == 'linux/arm64' }} | ||
uses: docker/setup-qemu-action@v3 | ||
|
||
- name: Load image | ||
id: load_image | ||
run: | | ||
docker load --input /tmp/falkordb-tests-${{ env.ARCH_SUFFIX }}.tar | ||
|
||
- name: TCK tests | ||
working-directory: /FalkorDB | ||
run: make CLEAR_LOGS=0 tck-tests | ||
run: | | ||
docker run --platform ${{ matrix.platform }} --rm falkordb/falkordb-tests make CLEAR_LOGS=0 tck-tests | ||
|
||
fuzz-tests: | ||
needs: build | ||
runs-on: ubuntu-latest | ||
container: falkordb/falkordb:${{ github.ref_name }} | ||
needs: merge | ||
strategy: | ||
fail-fast: false | ||
matrix: | ||
platform: | ||
- linux/amd64 | ||
- linux/arm64 | ||
env: | ||
ARCH_SUFFIX: ${{ matrix.platform == 'linux/amd64' && 'x64' || 'arm64v8' }} | ||
steps: | ||
- name: Download image | ||
uses: actions/download-artifact@v3 | ||
with: | ||
name: falkordb-tests-${{ env.ARCH_SUFFIX }} | ||
path: /tmp | ||
|
||
- name: Set up QEMU | ||
if: ${{ matrix.platform == 'linux/arm64' }} | ||
uses: docker/setup-qemu-action@v3 | ||
|
||
- name: Load image | ||
id: load_image | ||
run: | | ||
docker load --input /tmp/falkordb-tests-${{ env.ARCH_SUFFIX }}.tar | ||
|
||
- name: Fuzz tests | ||
working-directory: /FalkorDB | ||
run: make fuzz TIMEOUT=180 | ||
run: | | ||
docker run --platform ${{ matrix.platform }} --rm falkordb/falkordb-tests make fuzz TIMEOUT=180 |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider adding a filter for
types
underpull_request
to limit the workflow trigger to specific activities, enhancing control and efficiency.