Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix multiple CI issues #1696

Closed

Conversation

Projects
None yet
3 participants
@xmnlab
Copy link
Collaborator

xmnlab commented Jan 28, 2019

The test_insert fail should be investigated #1695

cpcloud and others added some commits Jan 13, 2019

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

xmnlab commented Jan 28, 2019

@cpcloud @kszucs

it seems now it is raising some errors related to #1689

maybe we should add this skip on #1689? what do you think about that?

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

xmnlab commented Jan 29, 2019

I am adding commits from #1689

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

xmnlab commented Jan 29, 2019

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

xmnlab commented Jan 29, 2019

Some messages from the logs:

python35_test

ibis/tests/all/conftest.py:29: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
ibis/tests/backends.py:55: in __init__
    self.connection = self.connect(data_directory)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <ibis.tests.backends.BigQuery object at 0x7f0d1b56d780>
data_directory = PosixPath('/tmp/ibis-testing-data')

    def connect(self, data_directory):
        ga = pytest.importorskip('google.auth')
    
        project_id = os.environ.get('GOOGLE_BIGQUERY_PROJECT_ID')
        if project_id is None:
            pytest.skip('Environment variable GOOGLE_BIGQUERY_PROJECT_ID '
                        'not defined')
        elif not project_id:
            pytest.skip('Environment variable GOOGLE_BIGQUERY_PROJECT_ID '
                        'is empty')
    
        dataset_id = 'testing'
        try:
>           return ibis.bigquery.connect(project_id, dataset_id)
E           AttributeError: module 'ibis.bigquery' has no attribute 'connect'

python36_conda_build

Rewriting env in output: {'PREFIX': '/opt/conda/conda-bld/ibis-framework_1548781116998/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place',
 'SRC_DIR': '/opt/conda/conda-bld/ibis-framework_1548781116998/test_tmp'}
/opt/conda/conda-bld/ibis-framework_1548781116998/test_tmp/conda_test_runner.sh: line 3:   132 Segmentation fault      (core dumped) "/opt/conda/conda-bld/ibis-framework_1548781116998/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/bin/python" -s "/opt/conda/conda-bld/ibis-framework_1548781116998/test_tmp/run_test.py"
Tests failed for ibis-framework-0.14.0+38.gd127d9c-py36_0.tar.bz2 - moving package to /opt/conda/conda-bld/broken
WARNING:conda_build.build:Tests failed for ibis-framework-0.14.0+38.gd127d9c-py36_0.tar.bz2 - moving package to /opt/conda/conda-bld/broken
TESTS FAILED: ibis-framework-0.14.0+38.gd127d9c-py36_0.tar.bz2
Traceback (most recent call last):
  File "/ibis/ci/feedstock.py", line 129, in <module>
    cli()
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/ibis/ci/feedstock.py", line 124, in test
    ctx.invoke(build)
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/ibis/ci/feedstock.py", line 97, in build
    stderr=click.get_binary_stream('stderr'))
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/plumbum/commands/base.py", line 103, in __call__
    return self.run(args, **kwargs)[1]
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/plumbum/commands/base.py", line 240, in run
    return p.run()
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/plumbum/commands/base.py", line 201, in runner
    return run_proc(p, retcode, timeout)
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/plumbum/commands/processes.py", line 232, in run_proc
    return _check_process(proc, retcode, timeout, stdout, stderr)
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/plumbum/commands/processes.py", line 23, in _check_process
    proc.verify(retcode, timeout, stdout, stderr)
  File "/opt/conda/envs/ibis-dev-3.6/lib/python3.6/site-packages/plumbum/machines/base.py", line 26, in verify
    stderr)
plumbum.commands.processes.ProcessExecutionError: Command line: ['/opt/conda/condabin/conda', 'build', '/tmp/ibis-framework-feedstock/recipe', '--channel', 'conda-forge', '--python', '3.6']

python36_docs

docker-compose run ibis ci/load-data.sh
/ibis/ci/load-data.sh: line 19:    13 Segmentation fault      (core dumped) python $CWD/datamgr.py download
/ibis/ci/load-data.sh: line 33:    20 Segmentation fault      (core dumped) python "${CWD}"/impalamgr.py load --data
/ibis/ci/load-data.sh: line 33:    21 Segmentation fault      (core dumped) python "${CWD}"/datamgr.py ${argcommands[${arg}]}
/ibis/ci/load-data.sh: line 33:    22 Segmentation fault      (core dumped) python "${CWD}"/datamgr.py ${argcommands[${arg}]}
/ibis/ci/load-data.sh: line 33:    23 Segmentation fault      (core dumped) python "${CWD}"/datamgr.py ${argcommands[${arg}]}
/ibis/ci/load-data.sh: line 33:    24 Segmentation fault      (core dumped) python "${CWD}"/datamgr.py ${argcommands[${arg}]}
/ibis/ci/load-data.sh: line 33:    25 Segmentation fault      (core dumped) python "${CWD}"/datamgr.py ${argcommands[${arg}]}
/ibis/ci/load-data.sh: line 33:    26 Segmentation fault      (core dumped) python "${CWD}"/datamgr.py ${argcommands[${arg}]}
Exited with code 1

pipelines

WindowsTest py36

[gw1] win32 -- Python 3.6.6 C:\Users\VssAdministrator\.conda\envs\ibis_3.6\python.exe
2019-01-29T17:00:12.6337665Z ibis\file\tests\conftest.py:81: in parquet
2019-01-29T17:00:12.6338121Z     pq.write_table(table, str(f))
2019-01-29T17:00:12.6339066Z C:\Users\VssAdministrator\.conda\envs\ibis_3.6\lib\site-packages\pyarrow\parquet.py:1159: in write_table
2019-01-29T17:00:12.6339134Z     **kwargs) as writer:
2019-01-29T17:00:12.6339994Z C:\Users\VssAdministrator\.conda\envs\ibis_3.6\lib\site-packages\pyarrow\parquet.py:369: in __init__
2019-01-29T17:00:12.6340150Z     sink = self.file_handle = filesystem.open(path, 'wb')
2019-01-29T17:00:12.6340962Z C:\Users\VssAdministrator\.conda\envs\ibis_3.6\lib\site-packages\pyarrow\filesystem.py:242: in open
2019-01-29T17:00:12.6341056Z     return open(path, mode=mode)
2019-01-29T17:00:12.6342122Z E   FileNotFoundError: [Errno 2] No such file or directory: '\\Users\\VssAdministrator\\AppData\\Local\\Temp\\pytest-of-VssAdministrator\\pytest-0\\popen-gw1\\test_creation1\\pq\\open.parquet'

WindowsCondaBuild py35

%SRC_DIR%>set "INCLUDE=%PREFIX%\Library\include;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt;" 
2019-01-29T16:49:59.6916807Z The system cannot find the path specified.
2019-01-29T16:50:00.3097998Z 
2019-01-29T16:50:00.3098324Z CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
2019-01-29T16:50:00.3098434Z If using 'conda activate' from a batch script, change your
2019-01-29T16:50:00.3098534Z invocation to 'CALL conda.bat activate'.
2019-01-29T16:50:00.3098567Z 
2019-01-29T16:50:00.3098614Z To initialize your shell, run
2019-01-29T16:50:00.3098646Z 
2019-01-29T16:50:00.3098692Z     $ conda init <SHELL_NAME>
2019-01-29T16:50:00.3098744Z 
2019-01-29T16:50:00.3098790Z Currently supported shells are:
2019-01-29T16:50:00.3098850Z   - bash
2019-01-29T16:50:00.3098911Z   - cmd.exe
2019-01-29T16:50:00.3098957Z   - fish
2019-01-29T16:50:00.3099003Z   - tcsh
2019-01-29T16:50:00.3099046Z   - xonsh
2019-01-29T16:50:00.3099096Z   - zsh
2019-01-29T16:50:00.3099169Z   - powershell
2019-01-29T16:50:00.3099200Z 
2019-01-29T16:50:00.3099250Z See 'conda init --help' for more information and options.
2019-01-29T16:50:00.3099282Z 
2019-01-29T16:50:00.3099353Z IMPORTANT: You may need to close and restart your shell after running 'conda init'.

WindowsCondaBuild py36

2019-01-29T16:49:44.6558839Z %SRC_DIR%>set "INCLUDE=%PREFIX%\Library\include;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt;" 
2019-01-29T16:49:45.2160159Z 
2019-01-29T16:49:45.2160730Z CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
2019-01-29T16:49:45.2160932Z If using 'conda activate' from a batch script, change your
2019-01-29T16:49:45.2161111Z invocation to 'CALL conda.bat activate'.
2019-01-29T16:49:45.2161227Z 
2019-01-29T16:49:45.2161542Z To initialize your shell, run
2019-01-29T16:49:45.2161655Z 
2019-01-29T16:49:45.2161785Z     $ conda init <SHELL_NAME>
2019-01-29T16:49:45.2161893Z 
2019-01-29T16:49:45.2162038Z Currently supported shells are:
2019-01-29T16:49:45.2162181Z   - bash
2019-01-29T16:49:45.2162306Z   - cmd.exe
2019-01-29T16:49:45.2162447Z   - fish
2019-01-29T16:49:45.2162570Z   - tcsh
2019-01-29T16:49:45.2162695Z   - xonsh
2019-01-29T16:49:45.2162833Z   - zsh
2019-01-29T16:49:45.2162954Z   - powershell
2019-01-29T16:49:45.2163063Z 
2019-01-29T16:49:45.2163211Z See 'conda init --help' for more information and options.
2019-01-29T16:49:45.2163323Z 
2019-01-29T16:49:45.2163458Z IMPORTANT: You may need to close and restart your shell after running 'conda init'.

kszucs added some commits Jan 29, 2019

@codecov

This comment has been minimized.

Copy link

codecov bot commented Jan 29, 2019

Codecov Report

Merging #1696 into master will decrease coverage by 2.25%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1696      +/-   ##
==========================================
- Coverage   89.59%   87.34%   -2.26%     
==========================================
  Files         186      186              
  Lines       27257    27262       +5     
  Branches     2309     2310       +1     
==========================================
- Hits        24422    23813     -609     
- Misses       2433     3044     +611     
- Partials      402      405       +3
Impacted Files Coverage Δ
ibis/pandas/aggcontext.py 93.97% <ø> (ø) ⬆️
ibis/pandas/execution/tests/test_temporal.py 100% <100%> (ø) ⬆️
ibis/clickhouse/tests/test_client.py 93.33% <100%> (-6.67%) ⬇️
ibis/file/tests/test_parquet.py 100% <100%> (ø) ⬆️
ibis/file/tests/conftest.py 100% <100%> (+13.63%) ⬆️
ibis/sql/postgres/tests/test_functions.py 99.8% <100%> (ø) ⬆️
ibis/bigquery/tests/test_client.py 25.87% <0%> (-73.55%) ⬇️
ibis/bigquery/tests/test_compiler.py 39.57% <0%> (-60.43%) ⬇️
ibis/bigquery/udf/tests/test_udf_execute.py 28.71% <0%> (-53.47%) ⬇️
ibis/bigquery/client.py 42.67% <0%> (-52.59%) ⬇️
... and 21 more
@kszucs

This comment has been minimized.

Copy link
Member

kszucs commented Jan 29, 2019

Had to disable conda build steps until We update the recipe in the conda forge feedstock.
There were a lot of changes around conda-forge lately: compiler migration, dropped python3.5 support and ibis also dropped python2.7.

The conda build tests are fetching and updating the upstream conda forge feedstock to build the local ibis source which are highly incompatible because of the changes mentioned above.

@kszucs

This comment has been minimized.

Copy link
Member

kszucs commented Jan 29, 2019

Right after this PR We should update our builds to test against python 3.7 #1544

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

xmnlab commented Jan 29, 2019

awesome! thank you so much @kszucs !

@kszucs

This comment has been minimized.

Copy link
Member

kszucs commented Jan 29, 2019

Not done yet, there must be a strange dependency problem in the docs build. Investigating...

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

xmnlab commented Jan 29, 2019

@kszucs I changed the version of pyarrow and pymapd on requrirements-docs-3.6.yml

@kszucs

This comment has been minimized.

Copy link
Member

kszucs commented Jan 29, 2019

I think there is another issue: conda-forge/parquet-cpp-feedstock#55

@kszucs kszucs changed the title Added pytest skip on clickhouse test_insert method Fix multiple CI issues Jan 29, 2019

kszucs and others added some commits Jan 29, 2019

kszucs added some commits Jan 30, 2019

kszucs added some commits Jan 30, 2019

@kszucs

This comment has been minimized.

Copy link
Member

kszucs commented Jan 30, 2019

@cpcloud @xmnlab please review
I'm merging a little later.

@xmnlab
Copy link
Collaborator Author

xmnlab left a comment

@kszucs It looks awesome! thanks for the hard work!

@kszucs

This comment has been minimized.

Copy link
Member

kszucs commented Jan 30, 2019

Sadly I had to disable multiple tests, but not critical ones - We'll fix them later.

@kszucs

This comment has been minimized.

Copy link
Member

kszucs commented Jan 31, 2019

Ahh flake8 was just updated, causing build errors on master. The fix is in #1659

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

xmnlab commented Jan 31, 2019

thanks for the update @kszucs the others PR I am working.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.