GetEntryOptions #932

maackle · 2019-01-29T18:09:23Z

Question:

How do we deal with the difference between storing one's own header vs. storing others' headers?

During Action::Commit, the header gets added to the chain CAS.
During Action::Hold, the entry and header get added to the DHT CAS, and metadata is added to DHT EAV

The question is do we also want to add our own header to the DHT CAS and EAV, so we can get them all from the same place? Or do we want to look in both places, adding our own header to the collection of stored headers from other agents? (This is because in general we don't Hold our own Entry)

Answer:

We look in both places: search our chain for our own header to this entry, and also check the DHT meta storage for other headers we've seen, merging the two results into the total set of headers.

I have added a summary of my changes to the changelog
Add headers to CAS and EAV when seen during Hold
Add headers to GetEntryOptions
Understand the difference between AgentState.chain.content_storage and DhtStore.content_storage

changes

header "sources" is now "provenances"
added triple provenance app spec test
- added carol
adds pretty_assertions crate as a core dep
Action::Hold now operates on EntryWithHeader
chain header no longer optional in validation package
DHT can find headers by checking meta/CAS
DHT can add headers to CAS/meta
fix flaky waiter logic in testing that made invalid assumptions about memory network

followups

impossible states: EntryWithHeader to ChainItem impossible states: EntryWithHeader to ChainItem #946
impossible states: "provenance" as real struct impossible states: "provenance" as real struct #947

…st into get-entry-options

lucksus

Cool! This looks great.

I would prefer looking also in our chain to find our own data locally, instead of either just relying on getting it from a remote DHT node (which would also be fine) or storing a copy in the local DHT store. The latter doesn't make sense this copy would be only for us since the network would not know we have this entry in our DHT..

zippy · 2019-02-03T08:59:23Z

Hey @maackle I'd love to merge this but now it's a got a conflict. Tried to merge but not clued in enough on waiter... :-)

thedavidmeister · 2019-02-03T14:27:18Z

core_types/src/eav.rs

@@ -25,7 +25,7 @@ use std::fmt::Debug;
 pub type Entity = Address;

 /// Using String for EAV attributes (not e.g. an enum) keeps it simple and open
-pub type Attribute = String;
+pub type Attribute = String; // BUT we totally could make it an enum with a Custom variant!


sure, just like Entry ;)

thedavidmeister · 2019-02-03T17:47:28Z

core/src/agent/state.rs

@@ -95,6 +95,12 @@ impl AgentState {
            .iter_type(&self.top_chain_header(), &entry.entry_type())
            .find(|h| h.entry_address() == &entry.address())
    }
+
+    pub fn get_header_for_entry_address(&self, entry_address: &Address) -> Option<ChainHeader> {


get_first_header_for_entry_address?

there can be many headers for a single entry, even within a single chain

also i think we used to have this but i removed it because it boils down to applying a find to an iter that already exists

also self.chain().iter(&self.top_chain_header()) feels wrong somehow :/

Thanks for pointing this out, it was a real bug.

I introduced iter_chain to AgentState for this purpose, which just iterates from the top header, what do you think?

thedavidmeister · 2019-02-03T17:49:37Z

core/src/dht/actions/hold.rs

-        context: context.clone(),
-        address: entry.address(),
+        context: context,
+        address: entry_wh.entry.address(),


don't collapse names of variables like this, write the whole thing out :P

How about we deal with this after the ChainItem refactor?

thedavidmeister · 2019-02-03T17:50:42Z

core/src/dht/dht_store.rs

+            // get the header addresses
+            .map(|eavi| eavi.value())
+            // fetch the header content from CAS
+            .map(|a| self.content_storage().read().unwrap().fetch(&a))


any way to not unwrap here?

@thedavidmeister Didn't we agree that it's OK to unwrap when locking a mutex or rwlock? All the unwraps are of that variety.

I'd like to defer on all the unwraps until we get our error handling story straight (#949)

thedavidmeister · 2019-02-03T17:51:04Z

core/src/dht/dht_store.rs

+            &ENTRY_HEADER_ATTRIBUTE.to_string(),
+            &header.address(),
+        )?;
+        self.content_storage().write().unwrap().add(header)?;


thedavidmeister · 2019-02-03T17:51:10Z

core/src/dht/dht_store.rs

+            &header.address(),
+        )?;
+        self.content_storage().write().unwrap().add(header)?;
+        self.meta_storage().write().unwrap().add_eavi(&eavi)?;


thedavidmeister · 2019-02-03T17:54:36Z

core_types/src/entry/mod.rs

@@ -137,6 +137,8 @@ pub struct EntryWithMeta {
    pub maybe_crud_link: Option<Address>,
 }

+pub type EntryWithMetaWithHeaders = (EntryWithMeta, Vec<ChainHeader>);


haha, luckily that never got used

…st into get-entry-options

maackle added 10 commits January 28, 2019 07:41

Remove redundant ValidationData::sources

547ba44

Couple source with signature (MISI)

0fb7201

Add unimplemented headers field

aa28546

Make include ChainHeader with Hold

ea1c9a1

Got all tests to compile

ceeafed

Merge remote-tracking branch 'origin/develop' into get-entry-options

dc53658

app_spec test for multiple headers on entry

5496150

Return ChainHeaders -- but doesn't quite work

1f43c94

Proper storage and retrieval of headers

c7374fe

Cleanup and test

eaf0cad

maackle changed the title ~~[WIP] GetEntryOptions~~ GetEntryOptions Jan 30, 2019

Merge remote-tracking branch 'origin/develop' into get-entry-options

0d34f57

maackle requested review from zippy, lucksus and thedavidmeister January 30, 2019 22:05

maackle added 4 commits January 31, 2019 04:05

Merge branch 'develop' into get-entry-options

19c830f

Better test

5ca249b

Merge branch 'get-entry-options' of github.com:holochain/holochain-ru…

bccb0eb

…st into get-entry-options

Fix wasm tests

445f336

lucksus approved these changes Jan 31, 2019

View reviewed changes

maackle added 7 commits January 31, 2019 18:11

Add failing test

9dea47e

This doesn't work

f068968

...but this does!

53e4dd2

Merge branch 'develop' into get-entry-options

9463dd5

Fix integration tests

b9d4e9b

Fix duplicate problem

5bf4ab7

Fix doc

86057db

zippy approved these changes Feb 1, 2019

View reviewed changes

willemolding approved these changes Feb 1, 2019

View reviewed changes

Merge branch 'develop' into get-entry-options

f490177

Merge branch 'develop' into get-entry-options

f18bee4

thedavidmeister and others added 3 commits February 4, 2019 00:30

Update CHANGELOG.md

e395b2e

Update CHANGELOG.md

38fedaa

Merge remote-tracking branch 'origin/develop' into get-entry-options

7fbfac2

thedavidmeister reviewed Feb 3, 2019

View reviewed changes

thedavidmeister mentioned this pull request Feb 3, 2019

impossible states: "provenance" as real struct #947

Open

Update CHANGELOG.md

cce241c

thedavidmeister suggested changes Feb 3, 2019

View reviewed changes

maackle added 5 commits February 3, 2019 19:13

Allow for multiple local headers for entry

f7d3d32

Merge branch 'get-entry-options' of github.com:holochain/holochain-ru…

cb1ac68

…st into get-entry-options

Remove type alias of doom

12b685c

Merge branch 'develop' into get-entry-options

fa07a51

chain is not really a chain. rename -> chain_store

43949f1

thedavidmeister approved these changes Feb 4, 2019

View reviewed changes

maackle added 2 commits February 5, 2019 21:39

Merge remote-tracking branch 'origin/develop' into get-entry-options

1a23190

Fix old-style block_on calls

3c103fb

lucksus merged commit 9f0e375 into develop Feb 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GetEntryOptions #932

GetEntryOptions #932

maackle commented Jan 29, 2019 •

edited by thedavidmeister

Loading

lucksus left a comment

zippy commented Feb 3, 2019

thedavidmeister Feb 3, 2019

thedavidmeister Feb 3, 2019

maackle Feb 3, 2019

thedavidmeister Feb 3, 2019

maackle Feb 3, 2019

thedavidmeister Feb 3, 2019

maackle Feb 3, 2019

maackle Feb 4, 2019

thedavidmeister Feb 3, 2019

thedavidmeister Feb 3, 2019

thedavidmeister Feb 3, 2019

maackle Feb 3, 2019

GetEntryOptions #932

GetEntryOptions #932

Conversation

maackle commented Jan 29, 2019 • edited by thedavidmeister Loading

Answer:

changes

followups

lucksus left a comment

Choose a reason for hiding this comment

zippy commented Feb 3, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maackle commented Jan 29, 2019 •

edited by thedavidmeister

Loading