Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug: Incosistent Metadata after updating the Online Key #491

Closed
1 task done
kairoaraujo opened this issue Apr 17, 2024 · 0 comments · Fixed by #501
Closed
1 task done

Bug: Incosistent Metadata after updating the Online Key #491

kairoaraujo opened this issue Apr 17, 2024 · 0 comments · Fixed by #501
Assignees
@kairoaraujo @MVrachev
Labels
bug Something isn't working CRITICAL Used for issues that brakes functionality

Comments

@kairoaraujo
Copy link
Member

kairoaraujo commented Apr 17, 2024 •
edited
Loading

What happened?

The Online Key is used for signing the Targets, Snapshop, Timestamp, and the Target Delegated Roles (succinct roles bins-<id>.json)

The bug occurs when the Online Key is updated, generating an inconsistency while the TUF client tries to download a Target File (artifact).

After an investigation, the root cause of the bug is that we bump the high-level Target roles but don't update the keys, in this case, the Online Key in the Target Metadata (targets.signed.delegations.keys and targets.signed.delegations.succinct_roles.keyids).

The below test shows the failure and the inconsistency.
I modified our Functional Tests for Metadata Update to rotate the online key.

INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: d213ca176460459aa0a2c58f32858d1e
INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: 4c974c106e054d758bae664749b6923b
INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: f2050b62642d4be8aeb75b755ae83405
INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: d3a171d70e7f4d53a1155ab3afc9a902
INFO     tests.functional.metadata.test_update:test_update.py:90 [METADATA UPDATE] Submiting Root Metadata Update
INFO     tests.functional.metadata.test_update:test_update.py:116 [METADATA UPDATE]  Metadata Updated by 32ceccb5382e4010a96dda9abad4b383
INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: 7330cc86307c4f64a87ee7b643363185
INFO     tests.functional.metadata.test_update:test_update.py:146 [METADATA UPDATE] {"data":{"task_id":"32ceccb5382e4010a96dda9abad4b383","state":"SUCCESS","result":{"message":"Metadata Update Processed","status":true,"task":"metadata_update","last_update":"2024-04-17T12:05:33.071964Z","details":{"role":"root"}}},"message":"Task state."}
INFO     tests.functional.metadata.test_update:test_update.py:153 [METADATA UPDATE] Update Metadata to 2.root.json finished
INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: 467cd789f552414ca21734de9702dd99
INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: ac1a7ee237844acfbe861975fae02e9e
INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: bb7aff18b7a14d6a9243664bd3b5e39e
INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: 68a28dcbd4e94dc184d3e3d4440348d8
INFO     tests.functional.metadata.test_update:test_update.py:175 [METADATA UPDATE] Metadata Update available (2.root.json)
INFO     tests.functional.metadata.test_update:test_update.py:49 Stop adding artifacts. Total requests: 9
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 1/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 2/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 3/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 4/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 5/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 6/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 7/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 8/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 9/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:195 Verifying test/condescending_golick-0.tar.gz
============================================================================= warnings summary ==============================================================================
tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed
tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed
tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed
  /usr/local/lib/python3.10/site-packages/pytest_bdd/compat.py:46: PytestDeprecationWarning: A private pytest class or function was used.
    fd = FixtureDef(

tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed
tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed
tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed
  /usr/local/lib/python3.10/site-packages/pytest_html/basereport.py:356: DeprecationWarning: The 'py' module is deprecated and support will be removed in a future release.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------------------------------------------------- generated json file: /rstuf-runner/rstuf-umbrella/test-report.json -----------------------------------------------------
============================================================================= slowest durations =============================================================================
16.34s call     tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed
------------------------------------------------ Generated html report: file:///rstuf-runner/rstuf-umbrella/test-report.html ------------------------------------------------
========================================================================== short test summary info ==========================================================================
FAILED tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed - tuf.api.exceptions.UnsignedMetadataError: bins-1 was signed by 0/1 keys
====================================================================== 1 failed, 6 warnings in 16.42s =======================================================================

Below, I added a simple fix that shows the consistency.


tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed <- ../../usr/local/lib/python3.10/site-packages/pytest_bdd/scenario.py 
------------------------------------------------------------------------------- live log call -------------------------------------------------------------------------------
2024-04-17 11:58:37 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:37 [    INFO] Added task_id: 5791656a48694886b80d4f8540d3df8d (test_update.py:45)
2024-04-17 11:58:38 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:38 [    INFO] Added task_id: 9c3c85acf42b4713b742162130799da1 (test_update.py:45)
2024-04-17 11:58:38 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:38 [    INFO] Added task_id: d6331645182e417eb5f51aa60276e8bd (test_update.py:45)
2024-04-17 11:58:39 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:39 [    INFO] Added task_id: 79806abceb95462fa42f4056e334a498 (test_update.py:45)
2024-04-17 11:58:39 [    INFO] [METADATA UPDATE] Submiting Root Metadata Update (test_update.py:90)
2024-04-17 11:58:39 [    INFO] [METADATA UPDATE]  Metadata Updated by ed2a1afb04414feb8e39fc876d1e3ae1 (test_update.py:116)
2024-04-17 11:58:40 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:40 [    INFO] Added task_id: 64c6df38a2b2421f8f9b9afe71901f0b (test_update.py:45)
2024-04-17 11:58:40 [    INFO] [METADATA UPDATE] {"data":{"task_id":"ed2a1afb04414feb8e39fc876d1e3ae1","state":"SUCCESS","result":{"message":"Metadata Update Processed","status":true,"task":"metadata_update","last_update":"2024-04-17T11:58:40.200730Z","details":{"role":"root"}}},"message":"Task state."} (test_update.py:146)
2024-04-17 11:58:40 [    INFO] [METADATA UPDATE] Update Metadata to 2.root.json finished (test_update.py:153)
2024-04-17 11:58:40 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:40 [    INFO] Added task_id: 87d6476a0dbd43b4b3562cceac8241af (test_update.py:45)
2024-04-17 11:58:41 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:41 [    INFO] Added task_id: 5859bac4faa5404cb730fb4e139c9d16 (test_update.py:45)
2024-04-17 11:58:41 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:41 [    INFO] Added task_id: 40a514a128ef49d9b40ad8a37be9736f (test_update.py:45)
2024-04-17 11:58:42 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:42 [    INFO] Added task_id: 48a7d7ca1c644657bf848d4685147204 (test_update.py:45)
2024-04-17 11:58:42 [    INFO] [METADATA UPDATE] Metadata Update available (2.root.json) (test_update.py:175)
2024-04-17 11:58:42 [    INFO] Stop adding artifacts. Total requests: 9 (test_update.py:49)
2024-04-17 11:58:46 [    INFO] Task 1/9 finshed! (test_update.py:188)
2024-04-17 11:58:47 [    INFO] Task 2/9 finshed! (test_update.py:188)
2024-04-17 11:58:51 [    INFO] Task 3/9 finshed! (test_update.py:188)
2024-04-17 11:58:51 [    INFO] Task 4/9 finshed! (test_update.py:188)
2024-04-17 11:58:52 [    INFO] Task 5/9 finshed! (test_update.py:188)
2024-04-17 11:58:52 [    INFO] Task 6/9 finshed! (test_update.py:188)
2024-04-17 11:58:52 [    INFO] Task 7/9 finshed! (test_update.py:188)
2024-04-17 11:58:52 [    INFO] Task 8/9 finshed! (test_update.py:188)
2024-04-17 11:58:52 [    INFO] Task 9/9 finshed! (test_update.py:188)
2024-04-17 11:58:52 [    INFO] Verifying test/great_kapitsa-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/serene_blackburn-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/amazing_edison-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/adoring_tu-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/optimistic_chatterjee-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/amazing_visvesvaraya-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/optimistic_gould-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/lucid_dijkstra-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/stoic_gates-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/mystifying_colden-9.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/practical_brattain-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/pedantic_dirac-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/modest_joliot-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/youthful_dirac-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/amazing_khayyam-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/vigorous_wu-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/nostalgic_jepsen-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/dazzling_shirley-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/affectionate_edison-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/dazzling_ellis-9.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/sweet_colden-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/vigilant_rosalind-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/busy_saha-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/elated_leavitt-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/goofy_herschel-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/affectionate_brattain-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/tender_wiles-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/hardcore_austin-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/cranky_jang-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/ecstatic_faraday-9.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/beautiful_euler-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/compassionate_johnson-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/loving_chaplygin-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/gracious_feistel-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/xenodochial_buck-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/bold_haibt-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/pedantic_wright-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/quizzical_haslett-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/kind_heyrovsky-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/keen_ritchie-9.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/sad_johnson-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/keen_feistel-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/ecstatic_cannon-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/xenodochial_haslett-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/zen_wu-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/upbeat_jackson-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/boring_ptolemy-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/amazing_darwin-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/elastic_cartwright-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/inspiring_dubinsky-9.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/strange_snyder-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/nostalgic_wescoff-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/eloquent_jang-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/quizzical_bhabha-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/competent_hamilton-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/vigorous_mccarthy-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/pensive_banach-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/charming_bohr-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/lucid_lalande-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/zealous_chatelet-9.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/awesome_rosalind-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/youthful_dijkstra-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/modest_merkle-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/elated_noyce-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/sad_carson-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/quirky_leavitt-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/naughty_varahamihira-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/awesome_darwin-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/recursing_lovelace-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/stupefied_kalam-9.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/kind_solomon-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/naughty_colden-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/suspicious_darwin-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/sleepy_golick-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/brave_nash-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/elegant_benz-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/condescending_brown-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/optimistic_gould-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/boring_vaughan-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/suspicious_torvalds-9.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/condescending_mendeleev-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/elastic_bassi-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/boring_napier-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/elegant_sutherland-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/zealous_shaw-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/ecstatic_chatterjee-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/wonderful_feynman-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/xenodochial_dubinsky-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/inspiring_robinson-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/fervent_ride-9.tar.gz (test_update.py:195)


Feature: Update metadata
    Scenario: Updating Root metadata full signed
        Given RSTUF is running and operational
        Then the RSTUF is receiving multiple requests
        When the RSTUF key holders send a fully signed metadata
        Then the API requester should get status code '202' with 'task_id'
        Then the API requester gets from endpoint 'GET /api/v1/task' status 'SUCCESS'
        Then the '2.root.json' will be available in the TUF Metadata
        Then the user downloads will not have inconsistency during this process
    PASSED

Credit for the bug: @matglas

Note: Another bug was identified during the investigation.

  • The root persistency in the storage should be done in the end together with the timestamp during the Metadata Update

repository-service-tuf-worker/repository_service_tuf_worker/repository.py

Lines 1304 to 1311 in 70f548f

# root metadata and online key are updated
# 1. persist the new root
# 2. bump all target roles
self._persist(new_root, Root.type)
logging.info(
f"Updating root metadata: {new_root.signed.version}"
)
self._run_online_roles_bump(force=True)

What steps did you take?

  1. Generate a Ceremony rstuf admin ceremony
  2. Add targets
  3. Generate a Metadata Update and rotate the online key rstuf admin metadata update
  4. Add targets
  5. Try to download targets using a TUF client

What behavior did you expect?

Update the Online Key without breaking the TUF Metadata consistency for the TUF clients.

Include a Functional Test to avoid regression

Relevant log output

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct
@kairoaraujo kairoaraujo added bug Something isn't working CRITICAL Used for issues that brakes functionality labels Apr 17, 2024
@kairoaraujo kairoaraujo self-assigned this Apr 19, 2024
This was referenced Apr 20, 2024
Closed
Merged
Closed
Open
Merged
@MVrachev MVrachev self-assigned this Apr 29, 2024
@MVrachev MVrachev mentioned this issue Apr 29, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kairoaraujo kairoaraujo

@MVrachev MVrachev

Labels
bug Something isn't working CRITICAL Used for issues that brakes functionality
Projects
No open projects
[Sprint 17]: 05 April - 19 April 2024
Status: In Progress
[Sprint 18]: 19 April - 17 May 2024
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix two bugs & do a small refactoring MVrachev/repository-service-tuf-worker
2 participants
@kairoaraujo @MVrachev