Redash db migration from 10.1.0.b50633 to 25.1.0 Fails

[batzsalmassi]

Issue Summary

The database migration command docker-compose run --rm server manage db upgrade for upgrading to 25.1.0 version fails on fix_hash
Traceback (most recent call last):

Steps to Reproduce

docker-compose down
change image version in docker-compose.yml to 25.1.0
docker-compose run --rm server manage db upgrade
result:

root@redash:/opt/redash# docker compose run --rm server manage db upgrade
[+] Creating 2/0
✔ Container redash-postgres-1 Running 0.0s
✔ Container redash-redis-1 Running 0.0s
[2025-02-10 13:00:07,051][PID:1][INFO][xmlschema] Resource 'XMLSchema.xsd' is already loaded
[2025-02-10 13:00:08,335][PID:1][INFO][alembic.runtime.migration] Context impl PostgresqlImpl.
[2025-02-10 13:00:08,335][PID:1][INFO][alembic.runtime.migration] Will assume transactional DDL.
[2025-02-10 13:00:08,380][PID:1][INFO][alembic.runtime.migration] Running upgrade 89bc7873a3e0 -> fd4fc850d7ea, Convert user details to jsonb and move user profile image url into details column
[2025-02-10 13:00:08,429][PID:1][INFO][alembic.runtime.migration] Running upgrade fd4fc850d7ea -> 1038c2174f5d, Make case insensitive hash of query text
[2025-02-10 13:00:08,796][PID:1][INFO][alembic.runtime.migration] Running upgrade 1038c2174f5d -> 7ce5925f832b, create sqlalchemy_searchable expressions
[2025-02-10 13:00:08,805][PID:1][INFO][alembic.runtime.migration] Running upgrade 7ce5925f832b -> 7205816877ec, change type of json fields from varchar to json
[2025-02-10 13:00:10,473][PID:1][INFO][alembic.runtime.migration] Running upgrade 7205816877ec -> 9e8c841d1a30, fix_hash
Traceback (most recent call last):
File "/app/manage.py", line 9, in
manager()
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 357, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flask_migrate/cli.py", line 134, in upgrade
_upgrade(directory, revision, sql, tag, x_arg)
File "/usr/local/lib/python3.10/site-packages/flask_migrate/init.py", line 95, in wrapped
f(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flask_migrate/init.py", line 280, in upgrade
command.upgrade(config, revision, sql=sql, tag=tag)
File "/usr/local/lib/python3.10/site-packages/alembic/command.py", line 403, in upgrade
script.run_env()
File "/usr/local/lib/python3.10/site-packages/alembic/script/base.py", line 583, in run_env
util.load_python_file(self.dir, "env.py")
File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "/app/migrations/env.py", line 93, in
run_migrations_online()
File "/app/migrations/env.py", line 85, in run_migrations_online
context.run_migrations()
File "", line 8, in run_migrations
File "/usr/local/lib/python3.10/site-packages/alembic/runtime/environment.py", line 948, in run_migrations
self.get_context().run_migrations(**kw)
File "/usr/local/lib/python3.10/site-packages/alembic/runtime/migration.py", line 627, in run_migrations
step.migration_fn(**kw)
File "/app/migrations/versions/9e8c841d1a30_fix_hash.py", line 55, in upgrade
new_hash = update_query_hash(record)
File "/app/migrations/versions/9e8c841d1a30_fix_hash.py", line 33, in update_query_hash
return query_runner.gen_query_hash(query_text, should_apply_auto_limit)
AttributeError: 'NoneType' object has no attribute 'gen_query_hash'

I try to upgrade from redash/redash:10.1.0.b50633

Technical details:

Redash Version: 25.1.0
Browser/OS: Chrome/macOS
How did you install Redash: Docker

[batzsalmassi]

@arikfr Do you have any idea of what can be done here in order to resolve this issue?

[arikfr]

@batzsalmassi This happens because we're trying to generate a hash for a query that has no data source assigned to it (so the data source id is null).

I will make a fix for this, but you can work around it by either deleting these queries or assigning some data source to them.

[batzsalmassi]

@batzsalmassi This happens because we're trying to generate a hash for a query that has no data source assigned to it (so the data source id is null).

I will make a fix for this, but you can work around it by either deleting these queries or assigning some data source to them.

Thank you so much for the reply!!
I'll try to find the empty queries and delete them, I'm waiting for the SAML Login issue to be fixed on V25.1 and after that I'll try to perform the upgrade again.

[arikfr]

@batzsalmassi what SAML issue are you referring to?

[batzsalmassi]

@arikfr I've raised this issue last week 7333
and @eradman has opened a PR that should fix the issue PR-7327

[take-take]

I encountered the same error mentioned in this issue while trying to upgrade to 25.1.0.

Today, after reading this comment, I tried deleting the queries with data_source_id is null along with their dependent data and then attempted the upgrade again, but the same error occurred.

I've been continuously upgrading my Redash instance since around v2.

It turned out that some old queries were still linked to an unused Python data source, which seemed to be the cause of the issue. Since those queries were no longer needed, I deleted them just like the ones with data_source_id is null.

After that, the upgrade to 25.1.0 completed successfully.

Thank you!

[batzsalmassi]

@take-take Thanks for the comment and share that it worked for you.
Did you logged into the Postgres DB itself to delete the queries or from the Redash portal?

[batzsalmassi]

Looping back to this issue since we didn't manage to find the queries with the empty data_source.
@arikfr Can you help maybe with how can we find these all queries with data_source_id = null?
I did try to add prints to the 9e8c841d1a30_fix_hash.py script to explore the problematic queries, but it's didn't print any new error and got the same error that I've attached in the issue body

[batzsalmassi]

@arikfr @eradman - FYI
After a long investigation, we managed to identify the root cause of our upgrade issue.
The common factor among the problematic queries is that they are of the dynamodb_sql type.
It appears that after the upgrade, the option to add a DynamoDB data source is no longer available.
Does the new Redash version not support DynamoDB data sources anymore?