New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker Compose based CI setup #1311

Closed
wants to merge 184 commits into
base: master
from

Conversation

Projects
None yet
3 participants
@kszucs
Member

kszucs commented Jan 29, 2018

  • separated postgres from impala
  • reduces duplicated service definitions
  • makes a circleci build reproducible locally
  • moved every testing/ci resources inder ci directory

TODO:

  • udfs kill impala daemon

kszucs added some commits Jan 31, 2018

- IBIS_TEST_CLICKHOUSE_PORT=9000
- IBIS_TEST_CLICKHOUSE_DATABASE=ibis_testing
- GOOGLE_BIGQUERY_PROJECT_ID=ibis-gbq
- GOOGLE_APPLICATION_CREDENTIALS=/tmp/gcloud-service-key.json

This comment has been minimized.

@cpcloud

cpcloud Jan 31, 2018

Member

This needs to be /tmp/ibis/gcloud-service-key.json to match the shell commands.

This comment has been minimized.

@kszucs

kszucs Jan 31, 2018

Member

In the volumes section /tmp/ibis host directory is mounted at /tmp inside the container.

This comment has been minimized.

@cpcloud

cpcloud Jan 31, 2018

Member

Cool, didn't catch that (even though it's right below). Thanks for pointing it out.

context: ..
dockerfile: ci/Dockerfile
args:
PYTHON: ${PYTHON_VERSION-3.6}

This comment has been minimized.

@cpcloud

cpcloud Jan 31, 2018

Member

Let's use the ${parameter:-word} syntax so that this will substitute word if parameter is null.
shell

@@ -243,7 +243,7 @@ def load(data, udf, data_dir, overwrite):
try:
load_impala_data(con, data_dir, overwrite)
finally:
shutil.rmtree(tmp_dir)

This comment has been minimized.

@cpcloud

cpcloud Jan 31, 2018

Member

Does this not work?

This comment has been minimized.

@kszucs

kszucs Jan 31, 2018

Member

Replaced with plumbum.

This comment has been minimized.

@cpcloud

cpcloud Jan 31, 2018

Member

Nice.

kszucs added some commits Jan 31, 2018

@kszucs kszucs changed the title from [WIP] Docker Compose based CI setup to Docker Compose based CI setup Jan 31, 2018

@kszucs

This comment has been minimized.

Member

kszucs commented Jan 31, 2018

@cpcloud It's ready on my side. You might cancel the appveyor build except the last one.

@cpcloud

This comment has been minimized.

Member

cpcloud commented Jan 31, 2018

@cpcloud cpcloud closed this in 5b083cf Jan 31, 2018

@cpcloud

This comment has been minimized.

Member

cpcloud commented Jan 31, 2018

thanks @kszucs great work!

@wesm

This comment has been minimized.

Member

wesm commented Jan 31, 2018

Cool stuff! I also enjoyed the commit messages

@kszucs

This comment has been minimized.

Member

kszucs commented Jan 31, 2018

Thanks! The development guide probably needs an update too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment