Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: Fix load data stage for Linux CI #2069

Merged
merged 7 commits into from
Feb 11, 2020
Merged

BUG: Fix load data stage for Linux CI #2069

merged 7 commits into from
Feb 11, 2020

Conversation

anmyachev
Copy link
Contributor

Main problem inside the string: $(DOCKER_RUN) -e LOGLEVEL=$(LOGLEVEL) ibis ./ci/load-data.sh $(BACKENDS). BACKENDS var contains all the backends, but ./ci/load-data.sh can load for specific backend. There was no error checking in this script and it was not noticeable.

LinuxTest py36:Load test datasets before fix 
2020-02-02T19:27:46.1265072Z ##[section]Starting: Load test datasets
2020-02-02T19:27:46.1268292Z ==============================================================================
2020-02-02T19:27:46.1268531Z Task         : Bash
2020-02-02T19:27:46.1268634Z Description  : Run a Bash script on macOS, Linux, or Windows
2020-02-02T19:27:46.1268791Z Version      : 3.163.1
2020-02-02T19:27:46.1268894Z Author       : Microsoft Corporation
2020-02-02T19:27:46.1268995Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
2020-02-02T19:27:46.1269164Z ==============================================================================
2020-02-02T19:27:46.2436085Z Generating script.
2020-02-02T19:27:46.2444329Z Script contents:
2020-02-02T19:27:46.2444721Z make load PYTHON_VERSION=$PYTHON_VERSION
2020-02-02T19:27:46.2457433Z ========================== Starting Command Output ===========================
2020-02-02T19:27:46.2474353Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/523702ec-2667-4b28-8901-70bcd6989901.sh
2020-02-02T19:27:46.2656803Z # load datasets for testing purpose
2020-02-02T19:27:46.2670394Z PYTHON_VERSION=3.6 docker-compose -f "/home/vsts/work/1/s/ci/docker-compose.yml" run --rm -e LOGLEVEL=WARNING ibis ./ci/load-data.sh clickhouse impala kudu-master kudu-tserver mysql omniscidb parquet postgres sqlite
2020-02-02T19:27:48.6549486Z    760 datamgr              INFO     MainThread                Downloading https://github.com/ibis-project/testing-data/archive/master.zip to /tmp/ibis-testing-data.zip...
2020-02-02T19:27:48.6716330Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-02-02T19:27:48.6717046Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-02-02T19:27:48.6717390Z 
2020-02-02T19:27:48.7836195Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-02-02T19:27:48.7837219Z 100   130    0   130    0     0   1171      0 --:--:-- --:--:-- --:--:--  1181
2020-02-02T19:27:49.3300708Z 
2020-02-02T19:27:50.2622131Z 100 35847    0 35847    0     0  54396      0 --:--:-- --:--:-- --:--:-- 54396
2020-02-02T19:27:51.9333956Z 100 5995k    0 5995k    0     0  3770k      0 --:--:--  0:00:01 --:--:-- 6394k
2020-02-02T19:27:52.9347230Z 100 15.6M    0 15.6M    0     0  4908k      0 --:--:--  0:00:03 --:--:-- 6137k
2020-02-02T19:27:53.9359070Z 100 15.6M    0 15.6M    0     0  3755k      0 --:--:--  0:00:04 --:--:-- 4432k
2020-02-02T19:27:54.9371853Z 100 15.6M    0 15.6M    0     0  3041k      0 --:--:--  0:00:05 --:--:-- 3469k
2020-02-02T19:27:55.3111429Z 100 15.6M    0 15.6M    0     0  2555k      0 --:--:--  0:00:06 --:--:-- 2849k
2020-02-02T19:27:56.7595508Z 100 15.6M    0 15.6M    0     0  2411k      0 --:--:--  0:00:06 --:--:-- 1984k
2020-02-02T19:27:57.7607339Z 100 75.6M    0 75.6M    0     0  9573k      0 --:--:--  0:00:08 --:--:-- 12.4M
2020-02-02T19:27:58.7618678Z 100 75.6M    0 75.6M    0     0  8519k      0 --:--:--  0:00:09 --:--:-- 12.4M
2020-02-02T19:27:59.2616024Z 100 75.6M    0 75.6M    0     0  7674k      0 --:--:--  0:00:10 --:--:-- 12.4M
2020-02-02T19:28:00.3613660Z 100  100M    0  100M    0     0  9752k      0 --:--:--  0:00:10 --:--:-- 19.7M
2020-02-02T19:28:01.3626521Z 100  123M    0  123M    0     0  10.6M      0 --:--:--  0:00:11 --:--:-- 21.4M
2020-02-02T19:28:02.3639737Z 100  123M    0  123M    0     0   9.7M      0 --:--:--  0:00:12 --:--:-- 10.5M
2020-02-02T19:28:03.3652432Z 100  123M    0  123M    0     0  9271k      0 --:--:--  0:00:13 --:--:-- 10.5M
2020-02-02T19:28:04.9463545Z 100  123M    0  123M    0     0  8639k      0 --:--:--  0:00:14 --:--:-- 10.5M
2020-02-02T19:28:05.9476397Z 100  183M    0  183M    0     0  11.2M      0 --:--:--  0:00:16 --:--:-- 14.5M
2020-02-02T19:28:06.2618470Z 100  183M    0  183M    0     0  10.6M      0 --:--:--  0:00:17 --:--:-- 10.6M
2020-02-02T19:28:07.2620195Z 100  185M    0  185M    0     0  10.5M      0 --:--:--  0:00:17 --:--:-- 12.5M
2020-02-02T19:28:08.3194878Z 100  227M    0  227M    0     0  12.2M      0 --:--:--  0:00:18 --:--:-- 21.0M
2020-02-02T19:28:09.3200407Z 100  231M    0  231M    0     0  11.8M      0 --:--:--  0:00:19 --:--:-- 21.7M
2020-02-02T19:28:09.5758054Z 100  231M    0  231M    0     0  11.2M      0 --:--:--  0:00:20 --:--:-- 11.0M
2020-02-02T19:28:09.5766371Z 100  268M    0  268M    0     0  12.8M      0 --:--:--  0:00:20 --:--:-- 23.4M
2020-02-02T19:28:09.5799915Z  21686 datamgr              INFO     MainThread                Extracting archive to /tmp/ibis-testing-data
2020-02-02T19:28:15.1368221Z   2387 datamgr              INFO     MainThread                Initializing ClickHouse...
2020-02-02T19:28:15.1506667Z   2314 datamgr              INFO     MainThread                Initializing PostgreSQL...
2020-02-02T19:28:15.1537790Z   2298 datamgr              INFO     MainThread                Initializing SQLite...
2020-02-02T19:28:15.1764669Z   2305 datamgr              INFO     MainThread                Initializing MySQL...
2020-02-02T19:28:15.6574560Z Usage: datamgr.py [OPTIONS] COMMAND [ARGS]...
2020-02-02T19:28:15.6576327Z 
2020-02-02T19:28:15.6576525Z Options:
2020-02-02T19:28:15.6577276Z   -q, --quiet / --verbose
2020-02-02T19:28:15.6577678Z   --help                   Show this message and exit.
2020-02-02T19:28:15.6578027Z 
2020-02-02T19:28:15.6578139Z Commands:
2020-02-02T19:28:15.6578267Z   bigquery
2020-02-02T19:28:15.6578439Z   clickhouse
2020-02-02T19:28:15.6578552Z   download
2020-02-02T19:28:15.6578725Z   mysql
2020-02-02T19:28:15.6578850Z   omniscidb
2020-02-02T19:28:15.6579009Z   parquet
2020-02-02T19:28:15.6579132Z   postgres
2020-02-02T19:28:15.6579289Z   sqlite
2020-02-02T19:28:15.6672587Z Usage: datamgr.py [OPTIONS] COMMAND [ARGS]...
2020-02-02T19:28:15.6673315Z 
2020-02-02T19:28:15.6673570Z Options:
2020-02-02T19:28:15.6674697Z   -q, --quiet / --verbose
2020-02-02T19:28:15.6675773Z   --help                   Show this message and exit.
2020-02-02T19:28:15.6676102Z 
2020-02-02T19:28:15.6676387Z Commands:
2020-02-02T19:28:15.6676660Z   bigquery
2020-02-02T19:28:15.6677439Z   clickhouse
2020-02-02T19:28:15.6677756Z   download
2020-02-02T19:28:15.6678063Z   mysql
2020-02-02T19:28:15.6678352Z   omniscidb
2020-02-02T19:28:15.6678621Z   parquet
2020-02-02T19:28:15.6679011Z   postgres
2020-02-02T19:28:15.6679187Z   sqlite
2020-02-02T19:28:16.7316157Z   3428 datamgr              INFO     MainThread                Initializing OmniSci...
2020-02-02T19:28:17.6656205Z /opt/conda/envs/ibis-env/lib/python3.6/site-packages/pyarrow/pandas_compat.py:383: FutureWarning: RangeIndex._start is deprecated and will be removed in a future version. Use RangeIndex.start instead
2020-02-02T19:28:17.6657259Z   'start': level._start,
2020-02-02T19:28:17.6657845Z /opt/conda/envs/ibis-env/lib/python3.6/site-packages/pyarrow/pandas_compat.py:384: FutureWarning: RangeIndex._stop is deprecated and will be removed in a future version. Use RangeIndex.stop instead
2020-02-02T19:28:17.6658349Z   'stop': level._stop,
2020-02-02T19:28:17.6658911Z /opt/conda/envs/ibis-env/lib/python3.6/site-packages/pyarrow/pandas_compat.py:385: FutureWarning: RangeIndex._step is deprecated and will be removed in a future version. Use RangeIndex.step instead
2020-02-02T19:28:17.6659345Z   'step': level._step
2020-02-02T19:29:02.6230272Z 
2020-02-02T19:29:02.6291002Z ##[section]Finishing: Load test datasets
LinuxTest py36:Load test datasets after fix 
2020-02-03T14:22:11.3625119Z ##[section]Starting: Load test datasets
2020-02-03T14:22:11.3627822Z ==============================================================================
2020-02-03T14:22:11.3627907Z Task         : Bash
2020-02-03T14:22:11.3627973Z Description  : Run a Bash script on macOS, Linux, or Windows
2020-02-03T14:22:11.3628032Z Version      : 3.163.1
2020-02-03T14:22:11.3628094Z Author       : Microsoft Corporation
2020-02-03T14:22:11.3628150Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
2020-02-03T14:22:11.3628229Z ==============================================================================
2020-02-03T14:22:11.4969174Z Generating script.
2020-02-03T14:22:11.4977691Z Script contents:
2020-02-03T14:22:11.4977880Z make load PYTHON_VERSION=$PYTHON_VERSION
2020-02-03T14:22:11.4993199Z ========================== Starting Command Output ===========================
2020-02-03T14:22:11.5009585Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/b54dcf02-fef8-4ec8-bd6e-c8214ef67038.sh
2020-02-03T14:22:11.5233033Z # load datasets for testing purpose
2020-02-03T14:22:11.5250454Z PYTHON_VERSION=3.6 docker-compose -f "/home/vsts/work/1/s/ci/docker-compose.yml" run --rm -e LOGLEVEL=WARNING ibis ./ci/load-data.sh clickhouse impala mysql omniscidb parquet postgres sqlite
2020-02-03T14:22:14.1495659Z    851 datamgr              INFO     MainThread                Downloading https://github.com/ibis-project/testing-data/archive/master.zip to /tmp/ibis-testing-data.zip...
2020-02-03T14:22:14.1669258Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-02-03T14:22:14.1669389Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-02-03T14:22:14.1669452Z 
2020-02-03T14:22:14.5497076Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-02-03T14:22:14.5497503Z 100   130    0   130    0     0    339      0 --:--:-- --:--:-- --:--:--   339
2020-02-03T14:22:14.8512368Z 
2020-02-03T14:22:16.0241776Z 100 11814    0 11814    0     0  17246      0 --:--:-- --:--:-- --:--:-- 17246
2020-02-03T14:22:16.8515304Z 100 2612k    0 2612k    0     0  1406k      0 --:--:--  0:00:01 --:--:-- 2219k
2020-02-03T14:22:17.8582555Z 100 14.6M    0 14.6M    0     0  5571k      0 --:--:--  0:00:02 --:--:-- 7474k
2020-02-03T14:22:18.8592680Z 100 15.6M    0 15.6M    0     0  4336k      0 --:--:--  0:00:03 --:--:-- 5322k
2020-02-03T14:22:19.8605942Z 100 15.6M    0 15.6M    0     0  3411k      0 --:--:--  0:00:04 --:--:-- 3991k
2020-02-03T14:22:20.8616287Z 100 15.6M    0 15.6M    0     0  2811k      0 --:--:--  0:00:05 --:--:-- 3194k
2020-02-03T14:22:21.8509386Z 100 15.6M    0 15.6M    0     0  2391k      0 --:--:--  0:00:06 --:--:-- 2769k
2020-02-03T14:22:23.2440107Z 100 58.6M    0 58.6M    0     0  7818k      0 --:--:--  0:00:07 --:--:-- 9025k
2020-02-03T14:22:24.2452146Z 100 75.6M    0 75.6M    0     0  8529k      0 --:--:--  0:00:09 --:--:-- 11.1M
2020-02-03T14:22:24.8612091Z 100 75.6M    0 75.6M    0     0  7682k      0 --:--:--  0:00:10 --:--:-- 11.1M
2020-02-03T14:22:26.4717952Z 100 95.9M    0 95.9M    0     0  9190k      0 --:--:--  0:00:10 --:--:-- 16.0M
2020-02-03T14:22:27.4732298Z 100  123M    0  123M    0     0  10.0M      0 --:--:--  0:00:12 --:--:-- 19.3M
2020-02-03T14:22:28.4744876Z 100  123M    0  123M    0     0  9539k      0 --:--:--  0:00:13 --:--:-- 11.6M
2020-02-03T14:22:28.8508619Z 100  123M    0  123M    0     0  8872k      0 --:--:--  0:00:14 --:--:-- 9466k
2020-02-03T14:22:29.8509078Z 100  133M    0  133M    0     0  9338k      0 --:--:--  0:00:14 --:--:-- 12.6M
2020-02-03T14:22:30.9784621Z 100  178M    0  178M    0     0  11.3M      0 --:--:--  0:00:15 --:--:-- 16.5M
2020-02-03T14:22:31.8651463Z 100  183M    0  183M    0     0  10.9M      0 --:--:--  0:00:16 --:--:-- 13.2M
2020-02-03T14:22:33.1214962Z 100  221M    0  221M    0     0  12.5M      0 --:--:--  0:00:17 --:--:-- 22.1M
2020-02-03T14:22:34.0482915Z 100  231M    0  231M    0     0  12.2M      0 --:--:--  0:00:18 --:--:-- 23.2M
2020-02-03T14:22:34.8517169Z 100  231M    0  231M    0     0  11.6M      0 --:--:--  0:00:19 --:--:-- 18.8M
2020-02-03T14:22:34.8561755Z 100  267M    0  267M    0     0  12.9M      0 --:--:--  0:00:20 --:--:-- 17.7M
2020-02-03T14:22:34.8562202Z 100  268M    0  268M    0     0  12.9M      0 --:--:--  0:00:20 --:--:-- 21.9M
2020-02-03T14:22:34.8662117Z  21561 datamgr              INFO     MainThread                Extracting archive to /tmp/ibis-testing-data
2020-02-03T14:22:40.0779532Z   1709 datamgr              INFO     MainThread                Initializing MySQL...
2020-02-03T14:22:40.1784419Z   1789 datamgr              INFO     MainThread                Initializing PostgreSQL...
2020-02-03T14:22:40.2432307Z   1843 datamgr              INFO     MainThread                Initializing ClickHouse...
2020-02-03T14:22:40.2665404Z   1879 datamgr              INFO     MainThread                Initializing SQLite...
2020-02-03T14:22:46.9340230Z   8560 datamgr              INFO     MainThread                Initializing OmniSci...
2020-02-03T14:22:49.1711912Z /opt/conda/envs/ibis-env/lib/python3.6/site-packages/pyarrow/pandas_compat.py:383: FutureWarning: RangeIndex._start is deprecated and will be removed in a future version. Use RangeIndex.start instead
2020-02-03T14:22:49.1712513Z   'start': level._start,
2020-02-03T14:22:49.1712965Z /opt/conda/envs/ibis-env/lib/python3.6/site-packages/pyarrow/pandas_compat.py:384: FutureWarning: RangeIndex._stop is deprecated and will be removed in a future version. Use RangeIndex.stop instead
2020-02-03T14:22:49.1713227Z   'stop': level._stop,
2020-02-03T14:22:49.1713640Z /opt/conda/envs/ibis-env/lib/python3.6/site-packages/pyarrow/pandas_compat.py:385: FutureWarning: RangeIndex._step is deprecated and will be removed in a future version. Use RangeIndex.step instead
2020-02-03T14:22:49.1713896Z   'step': level._step
2020-02-03T14:23:32.3977568Z 
2020-02-03T14:23:32.3982455Z ##[section]Finishing: Load test datasets

@anmyachev anmyachev requested a review from xmnlab February 3, 2020 15:29
xmnlab
xmnlab reviewed Feb 5, 2020
View reviewed changes
ci/load-data.sh Show resolved Hide resolved
xmnlab
xmnlab reviewed Feb 5, 2020
View reviewed changes
Copy link
Contributor

@xmnlab xmnlab left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good. I just added a question. @anmyachev could you add a release note please? thanks for working on this bugfix!

@xmnlab xmnlab changed the title [BUG] Fix load data stage for Linux CI BUG: Fix load data stage for Linux CI Feb 5, 2020
@xmnlab xmnlab self-assigned this Feb 5, 2020
@xmnlab xmnlab added bug Incorrect behavior inside of ibis ci Continuous Integration issues or PRs labels Feb 5, 2020
@jreback
Copy link
Contributor

jreback commented Feb 11, 2020

@anmyachev can you merge master, ping on green.

@anmyachev
Copy link
Contributor Author

@jreback ping on green

@jreback jreback added this to the Next Feature Release milestone Feb 11, 2020
@jreback jreback merged commit 376aee8 into ibis-project:master Feb 11, 2020
@jreback
Copy link
Contributor

jreback commented Feb 11, 2020

thanks @anmyachev

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xmnlab xmnlab xmnlab left review comments

@jreback jreback jreback approved these changes

Assignees

@xmnlab xmnlab

Labels
bug Incorrect behavior inside of ibis ci Continuous Integration issues or PRs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@anmyachev @jreback @xmnlab