[BUG] b'In function ddlCreateTableService: cannot create the table: Could not create table'

[ivenzor]

(rapids_blazing) sh-4.2$ conda list | grep blazing
blazingdb-toolchain 0.4.0 py37hf484d3e_0 blazingsql
blazingsql-calcite 0.4.0 py37_0 blazingsql
blazingsql-communication 0.4.0 py37_80 blazingsql
blazingsql-io 0.4.0 py37_31 blazingsql
blazingsql-orchestrator 0.4.0 py37_19 blazingsql
blazingsql-protocol 0.4.0 py37_25 blazingsql
blazingsql-python 0.4.0 cuda10.0_py37_14 blazingsql/label/cuda10.0
blazingsql-ral 0.4.0 cuda10.0_py37_5 blazingsql/label/cuda10.0

Yesterday I was able to create a table based on a cudf but today I'm having some errors. I have already created again the whole instance and did again all the steps but I'm facing some errors similar to the following:
b'In function ddlCreateTableService: cannot create the table: Could not create table'

There is no more information. Any ideas how I can trace the error?
In addition, it seems in version 0.4.2 there will be some changes in how BlazingContext launches processes, could it be related to this? When this release will be conda installable?

If I execute again BlazingContext() and try to create the table I can get two kinds of different errors:
1)
Already connected to the Orchestrator
b'In function ddlCreateTableService: cannot create the table: Connection to server failed.'

WARNING: blazingsql-orchestrator was not automativally started, its probably already running
WARNING: blazingsql-engine was not automativally started, its probably already running
WARNING: blazingsql-algebra was not automativally started, its probably already running
Already connected to the Orchestrator
Unexpected error on create_table, can only concatenate str (not "tuple") to str

[editaxz]

Hello @ivenzor ,

The issue you were experiencing may have been due to some versioning issues in conda and the packages you were using were not synched with each other. Actually, those were supposed to be 0.4.2, but the version number was not updated properly and hence it was causing some packages to not play nicely with each other.

We just released version 0.4.3, now we should point to the latest version that is correct. It should look like this:

(cuda10.0) edith@edithpc:~$ conda list |grep "blazing"
blazingdb-toolchain 0.4.0 py37hf484d3e_0 blazingsql
blazingsql-calcite 0.4.3 py37_0 blazingsql
blazingsql-communication 0.4.3 py37_0 blazingsql
blazingsql-io 0.4.3 py37_0 blazingsql
blazingsql-orchestrator 0.4.3 py37_0 blazingsql
blazingsql-protocol 0.4.3 py37_0 blazingsql
blazingsql-python 0.4.3 cuda10.0_py37_0 blazingsql/label/cuda10.0
blazingsql-ral 0.4.3 cuda10.0_py37_0 blazingsql/label/cuda10.0

This should work with the latest version (0.4.3 , cuda 10, py37):
from blazingsql import BlazingContext
bc = BlazingContext()
nation = bc.create_table('nation', '/tmp/nation_0_0.parquet')

[ivenzor]

I installed the new versions but still got errors:
(rapids_blazing) sh-4.2$ conda list | grep blazing
blazingdb-toolchain 0.4.0 py37hf484d3e_0 blazingsql
blazingsql-calcite 0.4.3 py37_0 blazingsql
blazingsql-communication 0.4.3 py37_0 blazingsql
blazingsql-io 0.4.3 py37_0 blazingsql
blazingsql-orchestrator 0.4.3 py37_0 blazingsql
blazingsql-protocol 0.4.3 py37_0 blazingsql
blazingsql-python 0.4.3 cuda10.0_py37_0 blazingsql/label/cuda10.0
blazingsql-ral 0.4.3 cuda10.0_py37_0 blazingsql/label/cuda10.0

I tried to create a test table based on a cudf:
from blazingsql import BlazingContext
bc = BlazingContext()
bc.create_table('test', df2)

Error:
connection established
b'In function ddlCreateTableService: cannot create the table: Connection to server failed.'

If I execute it a second time:
WARNING: blazingsql-orchestrator was not automativally started, its probably already running
WARNING: blazingsql-engine was not automativally started, its probably already running
WARNING: blazingsql-algebra was not automativally started, its probably already running
Already connected to the Orchestrator
Unexpected error on create_table, can only concatenate str (not "tuple") to str

If I execute it a third time:
Already connected to the Orchestrator
b'In function ddlCreateTableService: cannot create the table: Connection to server failed.'

[ivenzor]

It's now working today again... I don't quite get why it sometimes work and why sometimes it doesn't.

The only difference I can think of is the following:

• When I have successfully created a table is when I imported and called BlazingContext() at the begining of the notebook.
• I have not been succesfully when I import and called BlazingContext and create the table in the same cell.

My hypothesis is the following:
It could be that when I import and call BlazingContext() at the beginning of the notebook there is enough time for BlazingSQL to initialize required processes and any other required tasks and hence, when I reach the point to actually create the table everything is ready.
In the other hand, it could be that when I import and call BlazingContext() in the very same cell that I try to create the table, the BlazingSQL processes are not ready yet and hence the errors.

Does this make any sense to you?

[wmalpica]

@ivenzor You are absolutely correct. There are some internal sleep timers that are supposed to allow all the systems to boot up properly in the right order, but we are finding that different systems with different hardware need more time than what has been setup right now. This is an issue I am working on right now, to try to move away from arbitrary timers and setup something more robust. In the mean time, I would suggest adding additional sleep delay after you create the BlazingContext(). If you still are still having issues and you could increase the sleep timers that are used as part of the initialization of the BlazingContext(). This would have to be done manually in the code though. I hope to have this issue resolved very soon. I appologize for the inconvenience.

[wmalpica]

@ivenzor Also, there is something else going on where the java process blazingsql-algebra somehow holds on to the socket even after termination for a period of time. This makes it so that if you run a script that creates the BlazingContext more than onece in quick succession, the second time around it will think that blazingsql-algebra is already running and not launch it. I am also working on this.

To work around this in the mean time, you can launch the processes manually :
$CONDA_PREFIX/bin/blazingsql-orchestrator 9100 8889 127.0.0.1 8890
java -jar $CONDA_PREFIX/lib/blazingsql-algebra.jar -p 8890
$CONDA_PREFIX/bin/blazingsql-engine 1 0 127.0.0.1 9100 127.0.0.1 9001 8891 lo

[aucahuasi]

Quick notice: just to let you know we are working on this one +1

[ivenzor]

Many thanks for your feedback. For the time being I will just create the BlazingContext at the beggining of the notebook and it should be fine.

[wmalpica]

Hello @ivenzor Just wanted to give you an update on this issue. We have made a lot of changes to make it so that this is no longer a problem. If you are interested in building from source, we can help you out with that and you can get access to the new features right now. Our next release that will be available via a conda install is still some time away because have several big projects underway and we are also updating to use cudf 0.10.

[ivenzor]

@williamBlazing Thanks for your update. I was also looking forward for next version (0.4.5?) for the fix for issue #82 .
Support for cudf 0.10 will be very welcome.

I could try to build from source while the next conda version is released. Should I follow the instructions in: https://github.com/BlazingDB/pyBlazing/tree/develop#buildinstall-from-source-conda-environment ?

Best regards,

[wmalpica]

Hello @ivenzor since we are currently using rapidsai-nightly version of cudf, these are the instrucctions I recommend for building from source:

conda activate bzsqlenv
conda install -c conda-forge maven flatbuffers gtest gmock rapidjson
conda install -c conda-forge -c rapidsai-nightly/label/cuda10.0 cudf=0.10.0a191009
conda install -c rapidsai-nightly/label/cuda10.0 dask-cudf=0.10.0a191009
conda install -c conda-forge -c blazingsql blazingsql-toolchain
cd $CONDA_PREFIX
git clone https://github.com/BlazingDB/pyBlazing.git
cd pyBlazing/scripts
build-all.sh```

[gumdropsteve]

Running into this (or similar) issue right now in JupyterLab

• error is rotating between
  • b'In function ddlCreateTableService: cannot create the table: Could not create table'
  • b'In function ddlCreateTableService: cannot create the table: Could not drop table'

code block being executed

bc.create_table('taxi', gdf)

grep blazing

!conda list | grep blazing

blazingsql-calcite        0.4.4                        19    blazingsql-nightly
blazingsql-communication  0.4.4                        16    blazingsql-nightly
blazingsql-io             0.4.4                         3    blazingsql-nightly
blazingsql-orchestrator   0.4.4                        31    blazingsql-nightly
blazingsql-protocol       0.4.4                   py37_54    blazingsql-nightly
blazingsql-python         0.4.4           cuda10.0_py37_69    blazingsql-nightly
blazingsql-ral            0.4.4               cuda10.0_98    blazingsql-nightly
blazingsql-toolchain      0.4.4                         9    blazingsql-nightly

[aucahuasi]

Hi @gumdropsteve thanks! could you please send us more info so we can replicate the issue here? I know @williamBlazing has been doing some fixes related to this.

[gumdropsteve]

EDIT: seems to have been resolved by adjustment seen in #93 , resolved by #94

Hi @aucahuasi here is a link to a copy saved in Google Colab: https://colab.research.google.com/drive/1INmk9-HGpySMY_N62-_AnhSuV9JrA0Q0

The notebook found there is not the same as the one I was using before but is producing the same errors, and those errors are being triggered by the same bc.create_table() method.

Also now running (locally) on a different machine, so here's the output of !conda list | grep blazing

blazingsql-calcite        0.4.4                        24    blazingsql-nightly
blazingsql-communication  0.4.4              cuda10.0_107    blazingsql-nightly/label/cuda10.0
blazingsql-io             0.4.4                         3    blazingsql-nightly
blazingsql-orchestrator   0.4.4                        38    blazingsql-nightly
blazingsql-protocol       0.4.4                   py37_54    blazingsql-nightly
blazingsql-python         0.4.4           cuda10.0_py37_70    blazingsql-nightly/label/cuda10.0
blazingsql-ral            0.4.4              cuda10.0_135    blazingsql-nightly/label/cuda10.0
blazingsql-toolchain      0.4.4                        10    blazingsql-nightly

[aucahuasi]

Hey @ivenzor can you try again please?

[aucahuasi]

Hi @ivenzor these PRs solve this issue

#138
https://github.com/BlazingDB/blazingdb-ral/pull/245

You can try our nightly packages:

conda install -c blazingsql-nightly/label/cuda10.0 -c blazingsql-nightly -c rapidsai-nightly -c conda-forge -c defaults python=3.7 blazingsql

Thanks!