Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker Compose based CI setup #1311

Closed
wants to merge 184 commits into from
Closed

Conversation

kszucs
Copy link
Member

@kszucs kszucs commented Jan 29, 2018

  • separated postgres from impala
  • reduces duplicated service definitions
  • makes a circleci build reproducible locally
  • moved every testing/ci resources inder ci directory

TODO:

  • udfs kill impala daemon

- IBIS_TEST_CLICKHOUSE_PORT=9000
- IBIS_TEST_CLICKHOUSE_DATABASE=ibis_testing
- GOOGLE_BIGQUERY_PROJECT_ID=ibis-gbq
- GOOGLE_APPLICATION_CREDENTIALS=/tmp/gcloud-service-key.json
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be /tmp/ibis/gcloud-service-key.json to match the shell commands.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the volumes section /tmp/ibis host directory is mounted at /tmp inside the container.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, didn't catch that (even though it's right below). Thanks for pointing it out.

context: ..
dockerfile: ci/Dockerfile
args:
PYTHON: ${PYTHON_VERSION-3.6}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use the ${parameter:-word} syntax so that this will substitute word if parameter is null.
shell

@@ -243,7 +243,7 @@ def load(data, udf, data_dir, overwrite):
try:
load_impala_data(con, data_dir, overwrite)
finally:
shutil.rmtree(tmp_dir)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this not work?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced with plumbum.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice.

@kszucs kszucs changed the title [WIP] Docker Compose based CI setup Docker Compose based CI setup Jan 31, 2018
@kszucs
Copy link
Member Author

kszucs commented Jan 31, 2018

@cpcloud It's ready on my side. You might cancel the appveyor build except the last one.

@cpcloud
Copy link
Member

cpcloud commented Jan 31, 2018

appveyor build passing here: https://ci.appveyor.com/project/kszucs/ibis/build/1.0.507

@cpcloud cpcloud closed this in 5b083cf Jan 31, 2018
@cpcloud
Copy link
Member

cpcloud commented Jan 31, 2018

thanks @kszucs great work!

@wesm
Copy link
Member

wesm commented Jan 31, 2018

Cool stuff! I also enjoyed the commit messages

@kszucs
Copy link
Member Author

kszucs commented Jan 31, 2018

Thanks! The development guide probably needs an update too.

@kszucs kszucs deleted the compose branch May 30, 2022 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci Continuous Integration issues or PRs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants