Fix multiple CI issues #1696

xmnlab · 2019-01-28T03:33:24Z

The test_insert fail should be investigated #1695

xmnlab · 2019-01-28T04:02:26Z

@cpcloud @kszucs

it seems now it is raising some errors related to #1689

maybe we should add this skip on #1689? what do you think about that?

…e_insert_test

xmnlab · 2019-01-29T16:46:27Z

I am adding commits from #1689

xmnlab · 2019-01-29T16:57:13Z

cc @cpcloud

xmnlab · 2019-01-29T18:58:05Z

Some messages from the logs:

python35_test

ibis/tests/all/conftest.py:29: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
ibis/tests/backends.py:55: in __init__
    self.connection = self.connect(data_directory)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <ibis.tests.backends.BigQuery object at 0x7f0d1b56d780>
data_directory = PosixPath('/tmp/ibis-testing-data')

    def connect(self, data_directory):
        ga = pytest.importorskip('google.auth')
    
        project_id = os.environ.get('GOOGLE_BIGQUERY_PROJECT_ID')
        if project_id is None:
            pytest.skip('Environment variable GOOGLE_BIGQUERY_PROJECT_ID '
                        'not defined')
        elif not project_id:
            pytest.skip('Environment variable GOOGLE_BIGQUERY_PROJECT_ID '
                        'is empty')
    
        dataset_id = 'testing'
        try:
>           return ibis.bigquery.connect(project_id, dataset_id)
E           AttributeError: module 'ibis.bigquery' has no attribute 'connect'

python36_conda_build

Rewriting env in output: {'PREFIX': '/opt/conda/conda-bld/ibis-framework_1548781116998/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place',
 'SRC_DIR': '/opt/conda/conda-bld/ibis-framework_1548781116998/test_tmp'}
/opt/conda/conda-bld/ibis-framework_1548781116998/test_tmp/conda_test_runner.sh: line 3:   132 Segmentation fault      (core dumped) "/opt/conda/conda-bld/ibis-framework_1548781116998/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/bin/python" -s "/opt/conda/conda-bld/ibis-framework_1548781116998/test_tmp/run_test.py"
Tests failed for ibis-framework-0.14.0+38.gd127d9c-py36_0.tar.bz2 - moving package to /opt/conda/conda-bld/broken
WARNING:conda_build.build:Tests failed for ibis-framework-0.14.0+38.gd127d9c-py36_0.tar.bz2 - moving package to /opt/conda/conda-bld/broken
TESTS FAILED: ibis-framework-0.14.0+38.gd127d9c-py36_0.tar.bz2
Traceback (most recent call last):
  File "/ibis/ci/feedstock.py", line 129, in <module>
    cli()
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/ibis/ci/feedstock.py", line 124, in test
    ctx.invoke(build)
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/ibis/ci/feedstock.py", line 97, in build
    stderr=click.get_binary_stream('stderr'))
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/plumbum/commands/base.py", line 103, in __call__
    return self.run(args, **kwargs)[1]
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/plumbum/commands/base.py", line 240, in run
    return p.run()
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/plumbum/commands/base.py", line 201, in runner
    return run_proc(p, retcode, timeout)
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/plumbum/commands/processes.py", line 232, in run_proc
    return _check_process(proc, retcode, timeout, stdout, stderr)
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/plumbum/commands/processes.py", line 23, in _check_process
    proc.verify(retcode, timeout, stdout, stderr)
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/plumbum/machines/base.py", line 26, in verify
    stderr)
plumbum.commands.processes.ProcessExecutionError: Command line: ['/opt/conda/condabin/conda', 'build', '/tmp/ibis-framework-feedstock/recipe', '--channel', 'conda-forge', '--python', '3.6']

python36_docs

docker-compose run ibis ci/load-data.sh
/ibis/ci/load-data.sh: line 19:    13 Segmentation fault      (core dumped) python $CWD/datamgr.py download
/ibis/ci/load-data.sh: line 33:    20 Segmentation fault      (core dumped) python "${CWD}"/impalamgr.py load --data
/ibis/ci/load-data.sh: line 33:    21 Segmentation fault      (core dumped) python "${CWD}"/datamgr.py ${argcommands[${arg}]}
/ibis/ci/load-data.sh: line 33:    22 Segmentation fault      (core dumped) python "${CWD}"/datamgr.py ${argcommands[${arg}]}
/ibis/ci/load-data.sh: line 33:    23 Segmentation fault      (core dumped) python "${CWD}"/datamgr.py ${argcommands[${arg}]}
/ibis/ci/load-data.sh: line 33:    24 Segmentation fault      (core dumped) python "${CWD}"/datamgr.py ${argcommands[${arg}]}
/ibis/ci/load-data.sh: line 33:    25 Segmentation fault      (core dumped) python "${CWD}"/datamgr.py ${argcommands[${arg}]}
/ibis/ci/load-data.sh: line 33:    26 Segmentation fault      (core dumped) python "${CWD}"/datamgr.py ${argcommands[${arg}]}
Exited with code 1

pipelines

WindowsTest py36

[gw1] win32 -- Python 3.6.6 C:\Users\VssAdministrator\.conda\envs\ibis_3.6\python.exe
2019-01-29T17:00:12.6337665Z ibis\file\tests\conftest.py:81: in parquet
2019-01-29T17:00:12.6338121Z     pq.write_table(table, str(f))
2019-01-29T17:00:12.6339066Z C:\Users\VssAdministrator\.conda\envs\ibis_3.6\lib\site-packages\pyarrow\parquet.py:1159: in write_table
2019-01-29T17:00:12.6339134Z     **kwargs) as writer:
2019-01-29T17:00:12.6339994Z C:\Users\VssAdministrator\.conda\envs\ibis_3.6\lib\site-packages\pyarrow\parquet.py:369: in __init__
2019-01-29T17:00:12.6340150Z     sink = self.file_handle = filesystem.open(path, 'wb')
2019-01-29T17:00:12.6340962Z C:\Users\VssAdministrator\.conda\envs\ibis_3.6\lib\site-packages\pyarrow\filesystem.py:242: in open
2019-01-29T17:00:12.6341056Z     return open(path, mode=mode)
2019-01-29T17:00:12.6342122Z E   FileNotFoundError: [Errno 2] No such file or directory: '\\Users\\VssAdministrator\\AppData\\Local\\Temp\\pytest-of-VssAdministrator\\pytest-0\\popen-gw1\\test_creation1\\pq\\open.parquet'

WindowsCondaBuild py35

%SRC_DIR%>set "INCLUDE=%PREFIX%\Library\include;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt;" 
2019-01-29T16:49:59.6916807Z The system cannot find the path specified.
2019-01-29T16:50:00.3097998Z 
2019-01-29T16:50:00.3098324Z CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
2019-01-29T16:50:00.3098434Z If using 'conda activate' from a batch script, change your
2019-01-29T16:50:00.3098534Z invocation to 'CALL conda.bat activate'.
2019-01-29T16:50:00.3098567Z 
2019-01-29T16:50:00.3098614Z To initialize your shell, run
2019-01-29T16:50:00.3098646Z 
2019-01-29T16:50:00.3098692Z     $ conda init <SHELL_NAME>
2019-01-29T16:50:00.3098744Z 
2019-01-29T16:50:00.3098790Z Currently supported shells are:
2019-01-29T16:50:00.3098850Z   - bash
2019-01-29T16:50:00.3098911Z   - cmd.exe
2019-01-29T16:50:00.3098957Z   - fish
2019-01-29T16:50:00.3099003Z   - tcsh
2019-01-29T16:50:00.3099046Z   - xonsh
2019-01-29T16:50:00.3099096Z   - zsh
2019-01-29T16:50:00.3099169Z   - powershell
2019-01-29T16:50:00.3099200Z 
2019-01-29T16:50:00.3099250Z See 'conda init --help' for more information and options.
2019-01-29T16:50:00.3099282Z 
2019-01-29T16:50:00.3099353Z IMPORTANT: You may need to close and restart your shell after running 'conda init'.

WindowsCondaBuild py36

2019-01-29T16:49:44.6558839Z %SRC_DIR%>set "INCLUDE=%PREFIX%\Library\include;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt;" 
2019-01-29T16:49:45.2160159Z 
2019-01-29T16:49:45.2160730Z CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
2019-01-29T16:49:45.2160932Z If using 'conda activate' from a batch script, change your
2019-01-29T16:49:45.2161111Z invocation to 'CALL conda.bat activate'.
2019-01-29T16:49:45.2161227Z 
2019-01-29T16:49:45.2161542Z To initialize your shell, run
2019-01-29T16:49:45.2161655Z 
2019-01-29T16:49:45.2161785Z     $ conda init <SHELL_NAME>
2019-01-29T16:49:45.2161893Z 
2019-01-29T16:49:45.2162038Z Currently supported shells are:
2019-01-29T16:49:45.2162181Z   - bash
2019-01-29T16:49:45.2162306Z   - cmd.exe
2019-01-29T16:49:45.2162447Z   - fish
2019-01-29T16:49:45.2162570Z   - tcsh
2019-01-29T16:49:45.2162695Z   - xonsh
2019-01-29T16:49:45.2162833Z   - zsh
2019-01-29T16:49:45.2162954Z   - powershell
2019-01-29T16:49:45.2163063Z 
2019-01-29T16:49:45.2163211Z See 'conda init --help' for more information and options.
2019-01-29T16:49:45.2163323Z 
2019-01-29T16:49:45.2163458Z IMPORTANT: You may need to close and restart your shell after running 'conda init'.

codecov · 2019-01-29T20:36:50Z

Codecov Report

Merging #1696 into master will decrease coverage by 2.25%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1696      +/-   ##
==========================================
- Coverage   89.59%   87.34%   -2.26%     
==========================================
  Files         186      186              
  Lines       27257    27262       +5     
  Branches     2309     2310       +1     
==========================================
- Hits        24422    23813     -609     
- Misses       2433     3044     +611     
- Partials      402      405       +3

Impacted Files	Coverage Δ
ibis/pandas/aggcontext.py	`93.97% <ø> (ø)`	⬆️
ibis/pandas/execution/tests/test_temporal.py	`100% <100%> (ø)`	⬆️
ibis/clickhouse/tests/test_client.py	`93.33% <100%> (-6.67%)`	⬇️
ibis/file/tests/test_parquet.py	`100% <100%> (ø)`	⬆️
ibis/file/tests/conftest.py	`100% <100%> (+13.63%)`	⬆️
ibis/sql/postgres/tests/test_functions.py	`99.8% <100%> (ø)`	⬆️
ibis/bigquery/tests/test_client.py	`25.87% <0%> (-73.55%)`	⬇️
ibis/bigquery/tests/test_compiler.py	`39.57% <0%> (-60.43%)`	⬇️
ibis/bigquery/udf/tests/test_udf_execute.py	`28.71% <0%> (-53.47%)`	⬇️
ibis/bigquery/client.py	`42.67% <0%> (-52.59%)`	⬇️
... and 21 more

kszucs · 2019-01-29T20:38:31Z

Had to disable conda build steps until We update the recipe in the conda forge feedstock.
There were a lot of changes around conda-forge lately: compiler migration, dropped python3.5 support and ibis also dropped python2.7.

The conda build tests are fetching and updating the upstream conda forge feedstock to build the local ibis source which are highly incompatible because of the changes mentioned above.

kszucs · 2019-01-29T21:07:14Z

Right after this PR We should update our builds to test against python 3.7 #1544

xmnlab · 2019-01-29T21:50:50Z

awesome! thank you so much @kszucs !

kszucs · 2019-01-29T21:51:37Z

Not done yet, there must be a strange dependency problem in the docs build. Investigating...

xmnlab · 2019-01-29T22:15:54Z

@kszucs I changed the version of pyarrow and pymapd on requrirements-docs-3.6.yml

kszucs · 2019-01-29T22:27:56Z

I think there is another issue: conda-forge/parquet-cpp-feedstock#55

…sion

kszucs · 2019-01-30T17:34:15Z

@cpcloud @xmnlab please review
I'm merging a little later.

xmnlab

@kszucs It looks awesome! thanks for the hard work!

kszucs · 2019-01-30T20:31:22Z

Sadly I had to disable multiple tests, but not critical ones - We'll fix them later.

kszucs · 2019-01-31T14:41:48Z

Ahh flake8 was just updated, causing build errors on master. The fix is in #1659

xmnlab · 2019-01-31T15:00:42Z

thanks for the update @kszucs the others PR I am working.

cpcloud and others added 2 commits January 13, 2019 17:50

Fix pandas tests

3a55093

Added pytest skip on clickhouse test_insert method

6a78866

This was referenced Jan 28, 2019

Changed from pygdf to cudf #1694

Closed

test: investigate clickhouse test_insert fail #1695

Closed

Merge remote-tracking branch 'cpcloud/fix-master' into skip_clickhous…

d127d9c

…e_insert_test

kszucs added 2 commits January 29, 2019 21:19

conda-forge has dropped support for python 3.5

29a86a0

disable conda forge feedstock builds

5354f32

simplify conda env handling

4358a28

parquet compat with arrow 0.12

ebc7380

Changed version of pymapd and pyarrow for ci docs

e325270

restructure docker setup and conda env installation; pinned arrow ver…

fb8f336

…sion

kszucs changed the title ~~Added pytest skip on clickhouse test_insert method~~ Fix multiple CI issues Jan 29, 2019

kszucs and others added 7 commits January 30, 2019 00:00

inpin pyarrow and pymapd

99a59ee

don't install conda-build because it messes up the conda env

508a240

Added make command

f232e81

resolve segfault

c8bae5b

build-essentials

49b2e94

added missing character

5cd33b1

documentation build

9b8d9d5

kszucs added 8 commits January 30, 2019 02:33

don't force pull images

7cb5e1d

debug azure pipelines: assert that parquet tmp test directory exists

311cea1

python 3.7

1526d12

python 3.7

4cf5bee

install impyla from git; show running containers in docs build

74586fd

factor out config env variables

3c62791

compose format 3

cdd0a41

impyla

409efcf

kszucs mentioned this pull request Jan 30, 2019

Cannot write parquet file on windows with pyarrow 0.12 #1698

Closed

disable failing tests

612315e

This was referenced Jan 30, 2019

Strftime quoting error with postgres backend on python 3.7 #1699

Closed

Re-enable conda-forge CI build steps #1700

Closed

kszucs added 3 commits January 30, 2019 17:18

skip on win32

0f76568

debug platform

7201243

skip the whole test

0afbbd9

xmnlab commented Jan 30, 2019

View reviewed changes

kszucs closed this in 8feae7a Jan 31, 2019

This was referenced Jan 31, 2019

BUG: Fix pandas tests in master #1689

Closed

CI: Add python 3.7 support #1544

Closed

xmnlab deleted the skip_clickhouse_insert_test branch December 21, 2020 18:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix multiple CI issues #1696

Fix multiple CI issues #1696

xmnlab commented Jan 28, 2019 •

edited

Loading

xmnlab commented Jan 28, 2019

xmnlab commented Jan 29, 2019

xmnlab commented Jan 29, 2019

xmnlab commented Jan 29, 2019

codecov bot commented Jan 29, 2019 •

edited

Loading

kszucs commented Jan 29, 2019 •

edited

Loading

kszucs commented Jan 29, 2019

xmnlab commented Jan 29, 2019

kszucs commented Jan 29, 2019

xmnlab commented Jan 29, 2019

kszucs commented Jan 29, 2019

kszucs commented Jan 30, 2019

xmnlab left a comment

kszucs commented Jan 30, 2019

kszucs commented Jan 31, 2019

xmnlab commented Jan 31, 2019

Fix multiple CI issues #1696

Fix multiple CI issues #1696

Conversation

xmnlab commented Jan 28, 2019 • edited Loading

xmnlab commented Jan 28, 2019

xmnlab commented Jan 29, 2019

xmnlab commented Jan 29, 2019

xmnlab commented Jan 29, 2019

pipelines

codecov bot commented Jan 29, 2019 • edited Loading

Codecov Report

kszucs commented Jan 29, 2019 • edited Loading

kszucs commented Jan 29, 2019

xmnlab commented Jan 29, 2019

kszucs commented Jan 29, 2019

xmnlab commented Jan 29, 2019

kszucs commented Jan 29, 2019

kszucs commented Jan 30, 2019

xmnlab left a comment

Choose a reason for hiding this comment

kszucs commented Jan 30, 2019

kszucs commented Jan 31, 2019

xmnlab commented Jan 31, 2019

xmnlab commented Jan 28, 2019 •

edited

Loading

codecov bot commented Jan 29, 2019 •

edited

Loading

kszucs commented Jan 29, 2019 •

edited

Loading