AssertionError while running superset db upgrade from 1.5.1 to 2.1.0

[racoobina]

Environment

superset previous version: 1.5.1
superset installing version: 2.1.0
python version: 3.9.13
OS: CentOS 7.7

How to reproduce the bug

1. pip install apache-superset --upgrade
2. export FLASK_APP=superset
3. In superset_config.py the library werkzeug.contrib.cache replaced by cachelib.redis (because of previous error ModuleNotFoundError: No module named 'werkzeug.contrib'):

#from werkzeug.contrib.cache import RedisCache
from cachelib.redis import RedisCache

RESULTS_BACKEND = RedisCache(
    host='localhost', port=6379, key_prefix='superset_results')

CACHE_CONFIG = {'CACHE_TYPE':'redis', 'CACHE_REDIS_HOST':'localhost', 'CACHE_REDIS_PORT':6379, 'CACHE_KEY_PREFIX':'sql-cache'}
TABLE_NAMES_CACHE_CONFIG = {'CACHE_TYPE':'redis', 'CACHE_REDIS_HOST':'localhost', 'CACHE_REDIS_PORT':6379, 'CACHE_KEY_PREFIX':'table-cache'}

4. superset db upgrade

Expected results

upgraded superset bd

Actual results

/o/superset# superset db upgrade
Loaded your LOCAL configuration at [/opt/superset/superset_config.py]
logging was configured successfully
2023-04-29 13:43:02,550:INFO:superset.utils.logging_configurator:logging was configured successfully
2023-04-29 13:43:02,555:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
We haven't found any Content Security Policy (CSP) defined in the configurations. Please make sure to configure CSP using the TALISMAN_ENABLED and TALISMAN_CONFIG keys or any other external software. Failing to configure CSP have serious security implications. Check https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP for more information. You can disable this warning using the CONTENT_SECURITY_POLICY_WARNING key.
2023-04-29 13:43:02,556:WARNING:superset.initialization:We haven't found any Content Security Policy (CSP) defined in the configurations. Please make sure to configure CSP using the TALISMAN_ENABLED and TALISMAN_CONFIG keys or any other external software. Failing to configure CSP have serious security implications. Check https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP for more information. You can disable this warning using the CONTENT_SECURITY_POLICY_WARNING key.
Falling back to the built-in cache, that stores data in the metadata database, for the following cache: `FILTER_STATE_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
2023-04-29 13:43:02,558:WARNING:superset.utils.cache_manager:Falling back to the built-in cache, that stores data in the metadata database, for the following cache: `FILTER_STATE_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
Falling back to the built-in cache, that stores data in the metadata database, for the following cache: `EXPLORE_FORM_DATA_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
2023-04-29 13:43:02,561:WARNING:superset.utils.cache_manager:Falling back to the built-in cache, that stores data in the metadata database, for the following cache: `EXPLORE_FORM_DATA_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
/opt/superset/lib/python3.9/site-packages/sqlalchemy/orm/relationships.py:3435: SAWarning: relationship 'SqlaTable.slices' will copy column tables.id to column slices.datasource_id, which conflicts with relationship(s): 'Slice.table' (copies tables.id to slices.datasource_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards.   The 'overlaps' parameter may be used to remove this warning.
  util.warn(
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 58df9d617f14 -> 2ed890b36b94, rm_time_range_endpoints_from_qc
INFO  [alembic.runtime.migration] Running upgrade 2ed890b36b94 -> b0d0249074e4, deprecate time_range_endpoints v2
INFO  [alembic.runtime.migration] Running upgrade 2ed890b36b94 -> 8b841273bec3, sql_lab_models_database_constraint_updates
INFO  [alembic.runtime.migration] Running upgrade 8b841273bec3, b0d0249074e4 -> 9d8a8d575284, merge point
INFO  [alembic.runtime.migration] Running upgrade 9d8a8d575284 -> cecc6bf46990, rm_time_range_endpoints_2
INFO  [alembic.runtime.migration] Running upgrade cecc6bf46990 -> ad07e4fdbaba, rm_time_range_endpoints_from_qc_3
slices updated with no time_range_endpoints: 223
INFO  [alembic.runtime.migration] Running upgrade ad07e4fdbaba -> a9422eeaae74, new_dataset_models_take_2
>> Copy 26 physical tables to sl_tables...
>> Copy 408 SqlaTable to sl_datasets...
   Copy dataset owners...
   Link physical datasets with tables...
>> Copy 4,002 table columns to sl_columns...
   Link all columns to sl_datasets...
>> Copy 656 metrics to sl_columns...
   Link metric columns to datasets...
>> Run postprocessing on 4,658 columns
   [Column 1 to 4,658] 3,401 may be updated
   Assign table column relations...
Traceback (most recent call last):
  File "/opt/superset/bin/superset", line 8, in <module>
    sys.exit(superset())
  File "/opt/superset/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/superset/lib/python3.9/site-packages/flask/cli.py", line 567, in main
    return super().main(*args, **kwargs)
  File "/opt/superset/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/superset/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/superset/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/superset/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/superset/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/superset/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/superset/lib/python3.9/site-packages/flask/cli.py", line 407, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/opt/superset/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/superset/lib/python3.9/site-packages/flask_migrate/cli.py", line 149, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/opt/superset/lib/python3.9/site-packages/flask_migrate/__init__.py", line 98, in wrapped
    f(*args, **kwargs)
  File "/opt/superset/lib/python3.9/site-packages/flask_migrate/__init__.py", line 185, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/opt/superset/lib/python3.9/site-packages/alembic/command.py", line 320, in upgrade
    script.run_env()
  File "/opt/superset/lib/python3.9/site-packages/alembic/script/base.py", line 563, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/opt/superset/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
    module = load_module_py(module_id, path)
  File "/opt/superset/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/opt/superset/lib/python3.9/site-packages/superset/extensions/../migrations/env.py", line 126, in <module>
    run_migrations_online()
  File "/opt/superset/lib/python3.9/site-packages/superset/extensions/../migrations/env.py", line 118, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/opt/superset/lib/python3.9/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/opt/superset/lib/python3.9/site-packages/alembic/runtime/migration.py", line 620, in run_migrations
    step.migration_fn(**kw)
  File "/opt/superset/lib/python3.9/site-packages/superset/migrations/versions/2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py", line 903, in upgrade
    postprocess_datasets(session)
  File "/opt/superset/lib/python3.9/site-packages/superset/migrations/versions/2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py", line 532, in postprocess_datasets
    assert session.query(func.count()).select_from(joined_tables).scalar() == total
AssertionError

[sebastianliebscher]

I am not 100% sure if this will resolve it, but what if you upgrade step-by-step? So in your case: 1.5.1 --> 1.5.2 --> 1.5.3 --> 2.0.0 --> 2.0.1 --> 2.1.0

[racoobina]

Well, in this case I get the same error on 1.5.3 --> 2.0.0 step

[racoobina]

Resolved by commented assert line, but it caused deletion of None permission:

Delete None permissions
Going to delete a data access permission [datasource_access on [None].User1-Untitled Query-3Lc3vBC2L]
Going to delete a data access permission [schema_access on [None].[read]]