MySQL not finding terracotta table

[angelosnm]

I have prepared a custom Docker image (similar to the existing) and I get the below error when connecting it with a terracotta MySQL database.

sqlalchemy.exc.NoSuchTableError: terracotta

The data that have been ingested in the databse and the client side are using terracotta version 0.8.2 (according to previous similar issue)

[j08lue]

Can you please share your Dockerfile and a at least a rough sketch of how you initialized and populated the database and configured Terracotta? Could it be that you are pointing at a different schema or something?

[angelosnm]

Sure!

The data are being ingested as documented (https://terracotta-python.readthedocs.io/en/latest/tutorials/aws.html#populate-data-storage-and-database)

Regarding the Dockerfile, please check at this fork: https://github.com/SCiO-systems/terracotta/blob/easier-docker/Dockerfile

Then I'm running the container as per below:

docker run -d -p5000:5000 --name my-terracotta -e RIVER_PATH="mysql://user:pass@mydb.com/terracotta_test" -e DRIVER_PROVIDER=mysql my-terracotta-image

[dionhaefner]

docker run -d -p5000:5000 --name my-terracotta -e RIVER_PATH="mysql://user:pass@mydb.com/terracotta_test" -e DRIVER_PROVIDER=mysql my-terracotta-image

RIVER_PATH -> DRIVER_PATH?

[angelosnm]

docker run -d -p5000:5000 --name my-terracotta -e RIVER_PATH="mysql://user:pass@mydb.com/terracotta_test" -e DRIVER_PROVIDER=mysql my-terracotta-image

RIVER_PATH -> DRIVER_PATH?

Nope...I was hoping for that too but it was just a typo here 😆

The full log is the below:

[-] Exception on /datasets [GET]
Traceback (most recent call last):
  File "/app/terracotta/drivers/relational_meta_store.py", line 50, in convert_exceptions
    yield
  File "/usr/local/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/app/terracotta/drivers/relational_meta_store.py", line 192, in db_version
    terracotta_table = sqla.Table(
  File "<string>", line 2, in __new__
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
    return fn(*args, **kwargs)  # type: ignore[no-any-return]
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/schema.py", line 436, in __new__
    return cls._new(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/schema.py", line 490, in _new
    with util.safe_reraise():
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/schema.py", line 486, in _new
    table.__init__(name, metadata, *args, _no_init=False, **kw)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/schema.py", line 866, in __init__
    self._autoload(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/schema.py", line 898, in _autoload
    conn_insp.reflect_table(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 1538, in reflect_table
    raise exc.NoSuchTableError(table_name)
sqlalchemy.exc.NoSuchTableError: terracotta

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 872, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 870, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 855, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
  File "/app/terracotta/server/datasets.py", line 94, in get_datasets
    "datasets": datasets(keys, page=page, limit=limit),
  File "/usr/local/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/app/terracotta/handlers/datasets.py", line 23, in datasets
    with driver.connect():
  File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/app/terracotta/drivers/relational_meta_store.py", line 159, in connect
    self._connection_callback()
  File "/app/terracotta/drivers/relational_meta_store.py", line 178, in _connection_callback
    db_version = self.db_version
  File "/usr/local/lib/python3.10/contextlib.py", line 78, in inner
    with self._recreate_cm():
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/app/terracotta/drivers/relational_meta_store.py", line 52, in convert_exceptions
    raise exceptions.InvalidDatabaseError(error_message) from exception
terracotta.exceptions.InvalidDatabaseError: Could not connect to database. Make sure that the given path points to a valid Terracotta database, and that you ran driver.create().

To my understanding the code is searching for a table named terracotta and somehow it cannot be detected although this exists on the MySQL side.

[angelosnm]

Found it!

I was incorrectly parsing the environment variables. Instead of DRIVER_PATH I used TC_DRIVER_PATH and instead of DRIVER_PROVIDER I used TC_DRIVER_PROVIDER

Also I installed inside pymysql package inside the Dockerfile as it is not included in the environment of the code base!

[dionhaefner]

So you're saying that Terracotta just crashes with InvalidDatabaseError if no driver is given at all? That's not good, we should definitely give an error much earlier than that.

[dionhaefner]

@charalamm You want to look into this one?

[charalamm]

@dionhaefner I made a PR #330 that checks if the database url it is empty.