Add docker-compose-airflow #5757
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is almost exactly the Airflow docker-compose, except we add a lakeFS connector as part of setup. To use: ```sh docker compose -f docker-compose.yml -f docker-compose-airflow.yml up -d ``` == Why a separate docker-compose file? == 1. Airflow requires _multiple_ services. This makes patching the one required docker-compose file easier. 1. Anyway many (most?) users won't want all these services. Using profiles to load Airflow inside the same docker-compose file would only cause _more_ differences, and add no value to users. == How was this tested? == I brought up all services in _both_ docker-compose files using `docker compose`. Then I copied `treeverse/airflow-provider-lakeFS/lakefs_provider/example_dags/lakefs-dag.py` into `./dags/`. I created a repo example-repo on http://localhost:8080/ (the lakeFS instance). I had to restart airflow-scheduler, but after that it found my DAG and I was able to run the lakeFS example dag http://localhost:8000/ (the Airflow instance).
arielshaqed
added
new-feature
arielshaqed
commented
Apr 27, 2023
| AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK: 'true' | ||
| # WARNING: Use _PIP_ADDITIONAL_REQUIREMENTS option ONLY for a quick checks | ||
| # for other purpose (development, test and especially production usage) build/extend Airflow image. | ||
| _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-} airflow-provider-lakefs>=0.45.0 |
There was a problem hiding this comment.
Reviewer please note: this is the only change to the Airflow docker-compose file!
|
|
@arielshaqed thanks for the note. Is there a readme or docs to go with this? |
You're right, there will be. Documentation for the Everything Bagel docker-compose is somewhat lacking already. No excuse for keeping it that way, though. PTAL. |
rmoff
approved these changes
Apr 28, 2023
There was a problem hiding this comment.
LGTM, thanks @arielshaqed.
EB docs and general state is closely on my radar for attention soon :)
Jonathan-Rosenberg
approved these changes
May 1, 2023
|
Thanks! Pulling... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is almost exactly the Airflow docker-compose, except we add a lakeFS
connector as part of setup.
To use:
Why a separate docker-compose file?
required docker-compose file easier.
to load Airflow inside the same docker-compose file would only cause
more differences, and add no value to users.
How was this tested?
I brought up all services in both docker-compose files using
docker compose. Then I copiedtreeverse/airflow-provider-lakeFS/lakefs_provider/example_dags/lakefs-dag.pyinto
./dags/. I created a repo example-repo onhttp://localhost:8080/ (the lakeFS instance). I had to restart
airflow-scheduler, but after that it found my DAG and I was able to run the
lakeFS example dag http://localhost:8000/ (the Airflow instance).
Fixes #5677.