chore(key-value): use json serialization for main resources

[villebro]

SUMMARY

Since the permalink and app resources were already using simple primitive types that serialize well with json.dumps, we don't really need to use pickle to serialize them. Therefore, to simplify the main key-value resources, we replace pickle with json where applicable. However, the metastore cache will still use pickle, as it needs to be able to handle arbitrary binary types. This should be ok, as it's opt-in, so it won't be enabled by default.

To test this permalinks were created on master branch, the branch checked out, the database migrated, and the permalinks validated to work as before.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[codecov]

Codecov Report

Merging #23888 (a4c8202) into master (3528f41) will decrease coverage by 1.76%.
The diff coverage is 97.10%.

❗ Current head a4c8202 differs from pull request most recent head 5543be7. Consider uploading reports for the commit 5543be7 to get more accurate results

@@            Coverage Diff             @@
##           master   #23888      +/-   ##
==========================================
- Coverage   68.10%   66.35%   -1.76%     
==========================================
  Files        1940     1940              
  Lines       75016    75057      +41     
  Branches     8154     8155       +1     
==========================================
- Hits        51092    49803    -1289     
- Misses      21841    23171    +1330     
  Partials     2083     2083              

Flag	Coverage Δ
hive	?
mysql	78.82% <97.05%> (+0.01%)	⬆️
postgres	78.90% <97.05%> (+0.01%)	⬆️
presto	?
python	79.03% <97.05%> (-3.65%)	⬇️
sqlite	77.42% <97.05%> (+0.01%)	⬆️
unit	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...nd/src/explore/components/RunQueryButton/index.tsx	100.00% <ø> (ø)
superset/dashboards/permalink/commands/create.py	92.85% <ø> (ø)
superset/explore/permalink/commands/create.py	90.90% <ø> (ø)
superset/explore/permalink/commands/get.py	91.42% <ø> (ø)
superset/key_value/types.py	95.12% <90.90%> (-4.88%)	⬇️
...c/SqlLab/components/RunQueryActionButton/index.tsx	79.41% <100.00%> (+0.62%)	⬆️
superset/dashboards/permalink/commands/base.py	100.00% <100.00%> (ø)
superset/dashboards/permalink/commands/get.py	81.25% <100.00%> (ø)
superset/explore/permalink/commands/base.py	100.00% <100.00%> (ø)
superset/extensions/metastore_cache.py	92.45% <100.00%> (-5.63%)	⬇️
... and 8 more

... and 90 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[michael-s-molina]

LGTM

[michael-s-molina]

Nice tests

[dpgaspar]

really cool solution!

[vin01]

It makes me sad that people are getting CVEs for stuff like this and even trying to fly it as a "RCE". 😞

• https://www.horizon3.ai/apache-superset-part-ii-rce-credential-harvesting-and-more/

An attacker with write access to the metadata database can insert an arbitrary pickle payload into the store, and then trigger deserialization of it, leading to remote code execution.

With write access to the metadata database, I don't think any further steps are needed to compromise anything here. Such CVEs should be outright rejected.