Bug/backfill fixes

[giancarloaf]

Progress on #15 - backfilling all tables.

Changes

• Avoid error when parsing empty metadata in import_all.get_page()
• Extract pageid from older metadata format (i.e. pageid rather than page_id)
• Catch AttributeError when parsing features

Previous failed backfill jobs

March

  File "/usr/local/lib/python3.8/site-packages/modules/import_all.py", line 62, in get_page
    features = get_features(page, wptid)
  File "/usr/local/lib/python3.8/site-packages/modules/import_all.py", line 191, in get_features
    + get_feature_names(blink_features.get("AnimatedCSSFeatures"), "animated-css")
  File "/usr/local/lib/python3.8/site-packages/modules/import_all.py", line 161, in get_feature_names
    for (key, value) in feature_map.items():
AttributeError: 'list' object has no attribute 'items'

[TypeError: unsupported operand type(s) for <<: 'NoneType' and 'int'](https://console.cloud.google.com/errors/CMj-xeyat-PO5QE?time=P1D&project=httparchive)

April

  File "/usr/local/lib/python3.8/site-packages/modules/import_all.py", line 62, in get_page
    features = get_features(page, wptid)
  File "/usr/local/lib/python3.8/site-packages/modules/import_all.py", line 191, in get_features
    + get_feature_names(blink_features.get("AnimatedCSSFeatures"), "animated-css")
  File "/usr/local/lib/python3.8/site-packages/modules/import_all.py", line 161, in get_feature_names
    for (key, value) in feature_map.items():
AttributeError: 'list' object has no attribute 'items'

TypeError: 'NoneType' object is not subscriptable

at .aggregate_stats ( [/usr/local/lib/python3.8/site-packages/modules/transformation.py:541](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.8%2Fsite-packages%2Fmodules%2Ftransformation.py&line=541&project=httparchive) )
at .generate_pages ( [/usr/local/lib/python3.8/site-packages/modules/transformation.py:172](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.8%2Fsite-packages%2Fmodules%2Ftransformation.py&line=172&project=httparchive) )
at .get_page ( [/usr/local/lib/python3.8/site-packages/modules/import_all.py:67](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.8%2Fsite-packages%2Fmodules%2Fimport_all.py&line=67&project=httparchive) )

Successful backfill jobs, post-update

March
April

[github-actions]

Coverage report

The coverage rate went from 39% to 40% ⬆️
The branch rate is 34%.

0% of new lines are covered.

Diff Coverage details (click to unfold)

modules/transformation.py

0% of new lines are covered (28% of the complete file).

Missing lines: 110

modules/import_all.py

0% of new lines are covered (14% of the complete file).

Missing lines: 176