Several issues on OS X

[cbusseata]

Hi, I REALLY want to use your emulator (also, thank you for taking the initiative to put this together!) but I am having several issues with it.

1. I tried using this exact configuration as described here ([Question] Does emulator support streaming data with BigQuery Storage Write API? #29 (comment)) but I see:

bigquery_1         | unknown flag `c'
spdb_bigquery_1 exited with code 1

2. When trying to use --data-from-yaml when starting the docker container, I always get an error about missing file:

> ls -l data.yaml
-rw-r--r--    1 cb871d    10050   316 Sep 10 17:46 data.yaml
> docker run -it ghcr.io/goccy/bigquery-emulator:latest --project=test --data-from-yaml=./data.yaml
open ./data.yaml: no such file or directory

3. This issue magically disappeared, I am not sure why, but previously when I successfully started the docker container, I would just keep getting SQL connection errors.

I'd start the container with:

> docker run -it ghcr.io/goccy/bigquery-emulator:latest --project=test --port=9050 --log-level=debug --dataset=dataset1
[bigquery-emulator] listening at 0.0.0.0:9050

And any http request, e.g., http://0.0.0.0:9050/bigquery/v2/projects would get the following response:
"sql: connection is already closed"

Is there some prerequisites perhaps not listed?

[goccy]

Hi, As mentioned in #29, the streaming api doesn't work yet, so this code won't work.
Second, if you specify data.yaml with a relative path, it must be relative to the path where you're running the bigquery-emulator binary. Therefore, please specify the absolute path here.
The third problem is difficult to reproduce, so it would be helpful if you could provide the reproduction code.

[cbusseata]

Hey thanks for the reply. Here's what I've found.

Ignore point #3, whatever was going on, it's not happening anymore, probably just my machine.

1. I managed to get around this by creating a proxy server using node http proxy and express. With that I was able to capture the chunked data, turn it into a json string and make the request to the emulator without any issues.

2. I tried the absolute path, but this never worked just running the docker container with run -it for some reason. What did work was creating a new dockerfile from your image and running the emulator command from there - then I was able to import data from yaml.

I was trying to make this work as part of a spring boot dev environment. I am trying to use Flyway to create my schema and the simba bigquery JDBC driver. Unfortunately this isn't going to work for a couple of reasons, not sure if it's on your roadmap, but here goes.

The simba ODBC driver keeps the concept of preparing/executing statements. "Preparing" a statement actually consists of making a job insert POST request to /bigquery/v2/projects/{projectId}/jobs with "dryRun": true in the request body. Unfortunately, this repository doesn't seem to support that.

The other issue I see is that Flyway relies on the INFORMATION_SCHEMA for some things - I partially got around this by manually creating a INFORMATION_SCHEMA.TABLES table using --data-from-yaml, but then I ran into the dryRun issue described above.

There was also certain statistics property information that Flyway was looking for that was not in the response from the jobs API endpoint.

Do you have any plans to support dryRun? Need help?

[goccy]

Thank you for the information ! I will support dryRun option and information_schema later, and I will try to support dryRun as soon as possible !

[goccy]

I supported dryRun option in v0.1.12

[joselitofilho]

@goccy, many thanks for your work on BQ emulator. Amazing work.
I'm getting the same error as @cbusseata trying to use --data-from-yaml on Mac M1:

2. When trying to use --data-from-yaml when starting the docker container, I always get an error about missing file:

$ ls -l ./test
> -rw-r--r--  ./test/data.yaml

$ docker run -it --platform linux/x86_64 ghcr.io/goccy/bigquery-emulator:latest --project=myproject --port=8080 --data-from-yaml=./test/data.yaml
> open ./test/data.yaml: no such file or directory

Is there anything I can do? Let me know if I can help you with something.

[goccy]

@joselitofilho test/data.yaml should be on docker container, not host os, is that ok ?

[kankaiz]

hi @cbusseata
I came cross into this thread as I wanted to so similar thing, which is try to use Flyway to connect the emulator.
did you end up with a working solution??

thanks in advance!