Retrieval protocol

[fjarri]

Type of PR:

• Bugfix
• Code quality

Required reviews:

• 3

What this does:

• removes the unused bob parameter of TreasureMap.construct_by_publisher() (introduced in Immutable treasure maps #2773)
• removes Bob.matching_nodes_among()
• removes EncryptedTreasureMap datastore model
• removes the UmbralMessageKit alias of PolicyMessageKit
• moves TLSHostingPower to powers.py from server.py (a proper place for it, also helps avoid some import cycles)
• renames PolicyMessageKit to MessageKit. This is the externally exposed message-for-reencryption, so I think it makes sense to avoid unnecessary qualifiers - previously MessageKit wasn't even instantiated anywhere.
• moves kits.py to policies/ (seems like the proper place for it, the objects there are all policy-related)
• makes encrypt_and_sign() a classmethod constructor of MessageKit (renamed to author())
• author() from above only returns MessageKit and not a tuple of the kit and the signature - the latter is already a part of the kit
• fixes Define format for re-encryption metadata #259 by removing excess parts of the data Bob sends to Ursula
• WorkOrder/PRETask/MessageKit streamlining (the existing behavior of externally distributed message kits is preserved; more changes may be made in other PRs depending on the discussion in Message kit contents #2743)
• Bob supports an external caching of cfrags (through PolicyMessageKit objects)
• New protocol objects: ReencryptionRequest and ReencryptionResponse
• encrypted_treasure_map is now required in retrieve(); label and publisher_verifying_key are not needed
• enrico parameter is removed from retrieve; policy_encrypting_key is mandatory
• reworked the availability check before retrieval (former Bob.follow_treasure_map). Fixes Make Bob.follow_treasure_map() less obscure #2791
• (TODO) support serialization of PolicyMessageKit objects? For Bob to store them somewhere.
• (TODO) naming stuff. retrieve(), retrieve_cfrags(), PolicyMessageKit, RetrievalResult are the main offenders.

The whole retrieval procedure has been significantly simplified. Bob creates a RetrievalPlan plan based on Ursulas from the treasure map, the capsules in the kits, and possibly the attached cfrags in the kits. From it work orders for Ursulas are spawned and the results saved back into the plan.

You can see the new usage in test_bob_handles_frags.py. A lot of tests there were removed (because they tested the internal workings of retrieve()) and replaced with new ones.

For reviewers:

• is retrieve() is a good name? Maybe retrieve_and_decrypt()?
• should RetrievalClient be a class? It is pretty much stateless and could probably be rewritten as a function.

[piotr-roslaniec]

I don't think there is one, not even in eth_utils or eth_typings

[piotr-roslaniec]

LGTM 🎉 Great work!

[KPrasch]

Thanks for the walkthrough in-person. This PR represents the introduction of many new internal objects and altered system metaphors, it will take me a while to get used to this version of the codebase. Overall I think we're introducing some complexity to gain immutability and reduce the number of invalid instance states (as far as I understand this is precisely your goal). 👍🏻