fix: remove pydash merge

[betodealmeida]

SUMMARY

#15405 builds the configuration for Superset using pydash.objects.merge to combine superset_config.py into superset/config.py.

This fails when a RESULTS_BACKEND is configured, since pydash tries to deepcopy all the objects, but a cachelib.redis.RedisCache (a valid backend) can't be deepcopied because it has a lock:

Traceback (most recent call last):
  File "/Users/beto/.pyenv/versions/superset/bin/superset", line 33, in <module>
    sys.exit(load_entry_point('apache-superset', 'console_scripts', 'superset')())
  File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/click/core.py", line 1256, in invoke
    Command.invoke(self, ctx)
  File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/flask/cli.py", line 425, in decorator
    with __ctx.ensure_object(ScriptInfo).load_app().app_context():
  File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/flask/cli.py", line 381, in load_app
    app = call_factory(self, self.create_app)
  File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/flask/cli.py", line 119, in call_factory
    return app_factory()
  File "/Users/beto/Projects/incubator-superset/superset/app.py", line 52, in create_app
    raise ex
  File "/Users/beto/Projects/incubator-superset/superset/app.py", line 41, in create_app
    config = init_config()
  File "/Users/beto/Projects/incubator-superset/superset/app.py", line 62, in init_config
    return merge(config, override_conf)
  File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/pydash-5.0.1-py3.8.egg/pydash/objects.py", line 859, in merge
    return merge_with(obj, *sources)
  File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/pydash-5.0.1-py3.8.egg/pydash/objects.py", line 907, in merge_with
    sources = [copy.deepcopy(source) for source in sources]
  File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/pydash-5.0.1-py3.8.egg/pydash/objects.py", line 907, in <listcomp>
    sources = [copy.deepcopy(source) for source in sources]
  File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle '_thread.lock' object
Exception ignored in: <function Redis.__del__ at 0x1109a7790>
Traceback (most recent call last):
  File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/redis/client.py", line 885, in __del__
    self.close()
  File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/redis/client.py", line 888, in close
    conn = self.connection
AttributeError: 'Redis' object has no attribute 'connection'

Additionally, the merge strategy is not what we want to do. For example, we can have languages configured in superset/config.py:

LANGUAGES = {
    "en": {"flag": "us", "name": "English"},
    "es": {"flag": "es", "name": "Spanish"},
    "it": {"flag": "it", "name": "Italian"},
    "fr": {"flag": "fr", "name": "French"},
    "zh": {"flag": "cn", "name": "Chinese"},
    "ja": {"flag": "jp", "name": "Japanese"},
    "de": {"flag": "de", "name": "German"},
    "pt": {"flag": "pt", "name": "Portuguese"},
    "pt_BR": {"flag": "br", "name": "Brazilian Portuguese"},
    "ru": {"flag": "ru", "name": "Russian"},
    "ko": {"flag": "kr", "name": "Korean"},
    "sl": {"flag": "si", "name": "Slovenian"},
}

If a user wants to have only Brazilian Portuguese in their Superset instance they would add to their config:

LANGUAGES = {
    "pt_BR": {"flag": "br", "name": "Brazilian Portuguese"},
}

But the merge would bring back all languages, resulting in a config that looks like this:

LANGUAGES = {
    "en": {"flag": "us", "name": "English"},
    "es": {"flag": "es", "name": "Spanish"},
    "it": {"flag": "it", "name": "Italian"},
    "fr": {"flag": "fr", "name": "French"},
    "zh": {"flag": "cn", "name": "Chinese"},
    "ja": {"flag": "jp", "name": "Japanese"},
    "de": {"flag": "de", "name": "German"},
    "pt": {"flag": "pt", "name": "Portuguese"},
    "pt_BR": {"flag": "br", "name": "Brazilian Portuguese"},
    "ru": {"flag": "ru", "name": "Russian"},
    "ko": {"flag": "kr", "name": "Korean"},
    "sl": {"flag": "si", "name": "Slovenian"},
}

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

• Has associated issue:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[codecov]

Codecov Report

Merging #15435 (ed9b8f5) into master (21d1fb5) will decrease coverage by 0.15%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master   #15435      +/-   ##
==========================================
- Coverage   77.14%   76.98%   -0.16%     
==========================================
  Files         975      975              
  Lines       50662    50662              
  Branches     6228     6228              
==========================================
- Hits        39081    39004      -77     
- Misses      11370    11447      +77     
  Partials      211      211              

Flag	Coverage Δ
hive	?
mysql	81.73% <100.00%> (ø)
postgres	81.75% <100.00%> (ø)
python	81.83% <100.00%> (-0.30%)	⬇️
sqlite	81.37% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/app.py	68.42% <100.00%> (ø)
superset/db_engines/hive.py	0.00% <0.00%> (-82.15%)	⬇️
superset/db_engine_specs/hive.py	69.20% <0.00%> (-17.21%)	⬇️
superset/views/database/mixins.py	81.03% <0.00%> (-1.73%)	⬇️
superset/db_engine_specs/presto.py	83.36% <0.00%> (-1.06%)	⬇️
superset/db_engine_specs/base.py	87.97% <0.00%> (-0.41%)	⬇️
superset/connectors/sqla/models.py	88.22% <0.00%> (-0.24%)	⬇️
superset/utils/core.py	88.97% <0.00%> (-0.13%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 21d1fb5...ed9b8f5. Read the comment docs.

[ktmud]

Instead of deep-merging all config values, have we considered merging only the selected config keys that actually needed it?

cc @amitmiran137