Add open-proposal-ipld-prolly-trees

[RangerMauve]

Hello,

Here's a PR for a devgrant we'd like to pursue for standardizing B tree based indexes in the IPLD ecosystem.

This was put together with the help of @autonome and folks in the IPFS ecosystem like @mikeal

cc @kenlabs

[warpfork]

Holistic: YES.

We need more support for big maps in the IPLD ecosystem. This proposal will accomplish that.

The most commonly discussed "big map" ADL currently in our ecosystem is the HAMT. HAMTs are very lovely, but they have two weaknesses:

• They're not ordered! (This means they're approximately useless for indexing purposes.)
• In practice... we didn't standardize them particularly well, and we have many minor variations of them floating around in the wild. (Not a problem! But not ideal, either -- our experience with how much "extra" this "costs" vs value provided by the variation should give us good reason to encourage standardization and test vectors (!!) for future endeavors.)

The approach proposed here will support ordered big maps, which will provide value that HAMTs do not currently provide. Also, if there is success in standardizing and publishing lots of test vectors in such a way that future developments can converge better, I'll be very happy with that as a bonus win.

Equally critically, the proposed data structures still have the property of being ahistoric -- in other words, the history of insertions and deletions doesn't matter to the sharding state reached at any point in time -- meaning they converge! -- which means these structures are on approximately equal footing with HAMTs in the convergence property. (We care about converge properties for deduplication outcomes, but also out of sheer general principle in decentralization: it tends to have myriad simplifying effects that can affect higher levels of protocol in drastically positive ways.)

In total, implementing these data structures should provide new value in areas that we don't already have covered. 👍 👍 👍

---

I'm less familiar with the hyperbeedeebee part, and haven't reviewed that system deeply, but at first glance, I don't think I have any objections to it, directionally: so, sure. (I'm totally happy with how the indexing part is layered in this proposal separately from the core datastructure parts! That means I don't even have to buy in to hyperbeedeebee in order for the proposal to be good overall. At worst, hyperbeedeebee still serves as a "pull forward" sort of goal, which will help move the lower level work forward. At best, it becomes awesome on its own. Even the "at worst" outcome is good!)

[RangerMauve]

Just wondering if there was any sort of ETA on the FIL team for reviewing this. It'd be nice to get to present it at the IPFS thing next week. 😁

[FF-FOIT]

Hi @RangerMauve, this grant has been approved! We will follow up with an email to discuss next steps.

[RangerMauve]

Awesome, thank you! 🎉

[RangerMauve]

Hey! We've finished up Milestone 1.

We've got the initial version of the spec ready in ipld/ipld#254 We also have links to our research materials there.

Our MVP is here: https://github.com/kenlabs/ptree-bs/pulls as well as the initial fixtures we made.

We will be doing the ADL implementation in the next. We'll also be generating new fixtures as part of that.

[RangerMauve]

Hey, tried to send the invoice for Milestone 1 in December and haven't heard anything back.

Is there a better place to reach folks on the finance team or some sort of ETA?

[FF-FOIT]

Hi @RangerMauve, this invoice is being processed. Just sent you an email!

[RangerMauve]

Hey folks, we're ready for a review of our next milestone (milestone 3).

Our repo for the indexer is here: https://github.com/RangerMauve/ipld-prolly-indexer/

[autonome]

Discussed, and @RangerMauve and team will make a nice table for us to see which part of team is doing which milestone, which is completed, has invoice in process, etc.

[RangerMauve]

Hey there, I finally had a chance to review our two tracks (R: research, D: Development) and set up this table tracking progress:

Milestone	Description	Amount	Completion	Payment
D1	IPLD Schema and MVP implementation	96000	Done	Invoice TODO
D2	Golang implementation of Prolly Tree spec	96000	Done	Invoice TODO
D3	Golang IPLD indexer library	85800	Done	In-Review
D4	Example app to integrate with smart contracts	24000	TODO
R1	pre-study and research on Prolly tree literature & projects	39000	Done	Paid
R2	Golang IPLD indexer library	7500	Done	Sent
R3	Research needs in the (mostly web3) ecosystem	9000	TODO
R4	Research Applications and Feasibility	9000	TODO

Right now we're reviewing milestones D3 and R2

[RangerMauve]

Also we're starting to see usage of the indexer on some projects already (not yet public). Personally I'm excited to bring it to more social data and knowledgebases.

The spec itself was used in another devgrant to have a rust version in the Defluencer project which is really cool to have. A useful follow-up would be to get the indexer spec implemented in there too so we can more easily use prolly trees in FVM use cases.

[RangerMauve]

Updated ticket

Milestone	Description	Amount	Completion	Payment
D1	IPLD Schema and MVP implementation	96000	Done	Done
D2	Golang implementation of Prolly Tree spec	96000	Done	Done
D3	Golang IPLD indexer library	85800	Done	Done
D4	Example app to integrate with smart contracts	24000	Final Review	TODO
R1	pre-study and research on Prolly tree literature & projects	39000	Done	Done
R2	Golang IPLD indexer library	7500	Done	Done
R3	Research needs in the (mostly web3) ecosystem	9000	Publishing	TODO
R4	Research Applications and Feasibility	9000	Publishing	TODO

Aiming to finish R4 and R3 end of November

[RangerMauve]

Link to live deployed oracle UI: http://www.lpmyblog.cn:88/

Oracle source code: https://github.com/KenCloud-Tech/prolly-tree-oracle

[ianconsolata]

HI @RangerMauve! I would love to check this out, but I can't seem to reach the oracle UI. It might be the hotel wifi I am on blocking port 88 -- would it be possible for you to deploy the UI at port 80?

[RangerMauve]

Looking into this now!

[taoshengshi]

HI @RangerMauve! I would love to check this out, but I can't seem to reach the oracle UI. It might be the hotel wifi I am on blocking port 88 -- would it be possible for you to deploy the UI at port 80?

Hi @ianconsolata re-deploy here:
Prolly Tree Indexed Data Oracle for IPLD
https://dataoracle.gitdata.ai/

[ianconsolata]

Hi folks! Really excited to see this work progressing. I was able to connect a wallet and get to the UI for the smart contract, but I am receiving errors. Things I tried:

1. I created a few collections (dbName: ian), which seemed to succeed. However, I accidentally created the ny-baby-names table multiple times because the UI feedback wasn't clear. I probably should have received an error when I tried to submit the second and third time.

• test-names Create https://sepolia.etherscan.io/tx/0xd60d8b199aa5844b1d5ba02bda27d2969586f2836418603a0265c22d03b15e33
• ny-baby-names Create https://sepolia.etherscan.io/tx/0xb27996fd221533d1dcb3723a326d3b026c242af89ab85bf07bb9c4e583d12e20
• 2nd ny-baby-names Create https://sepolia.etherscan.io/tx/0xf8984f8dd8b077cc7d3d479461add7f1c38f427eb207ba27558f31cf8313f696
• 3rd ny-baby-names Create https://sepolia.etherscan.io/tx/0x64c3dcf670132e844ca23a57187d6d6f034467603892350e9340ae6166ceca78

2. I tried to put some data to the test-names collection using data created by the "Fill in with 1 click" button. However, when I try to submit I get Error Call Putundefined

3. I tried to import by URL into the ny-baby-names collection, using data from the following URL: https://data.cityofnewyork.us/api/views/25th-nujf/rows.csv. That import errors with the following message, which says the collection has not been created yet:

Error Call ImportError: cannot estimate gas; transaction may fail or may require manual gas limit (error={"code":-32603,"message":"execution reverted: This collection has not been created yet","data":{"originalError":{"code":3,"data":"0x08c379a0000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000285468697320636f6c6c656374696f6e20686173206e6f74206265656e206372656174656420796574000000000000000000000000000000000000000000000000","message":"execution reverted: This collection has not been created yet"}}}, method="estimateGas", transaction={"from":"0xbA4b610d92DA15ff48B3FB433CAc40D09368CDe6","to":"0xF177e3dbE1C3b1cba96EEef0E0E5511910735860","value":{"type":"BigNumber","hex":"0x03e8"},"data":"0x8f75bfd8000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000c000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000160000000000000000000000000000000000000000000000000000000000000000369616e0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000d6e792d626162792d6e616d657300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003a68747470733a2f2f646174612e636974796f666e6577796f726b2e75732f6170692f76696577732f323574682d6e756a662f726f77732e63737600000000000000000000000000000000000000000000000000000000000000000000000000036373760000000000000000000000000000000000000000000000000000000000","accessList":null}, code=UNPREDICTABLE_GAS_LIMIT, version=providers/5.4.5)

I was not able to test any of the other features yet, because I could not figure out how to get data into a collection. Could you please debug these issues, and write up a short set of instructions for which steps I need to take in which order to effectively use this smart contract / UI?

[taoshengshi]

Hi @ianconsolata Sorry that the devops engineer forgot to deploy the backend server totally as we were doing domain name migration last time 😭 Now re-deploy the backend server and please check again.

[ianconsolata]

Hi @taoshengshi I tried again, but I am still getting the same errors as before. Is https://dataoracle.gitdata.ai/ still the correct url to try? Or do I maybe need to call a different command (such as CreateIndex) between Create and Put/Import?

[RangerMauve]

@ianconsolata When you have time could you review the post for R4?

RangerMauve/blog.mauve.moe#3

[RangerMauve]

R3 published on medium.https://medium.com/@lendon.gertner/probabilistic-b-trees-in-peer-to-peer-databases-988cd4232621

[RangerMauve]

Updated ticket

Milestone	Description	Amount	Completion	Payment
D1	IPLD Schema and MVP implementation	96000	Done	Done
D2	Golang implementation of Prolly Tree spec	96000	Done	Done
D3	Golang IPLD indexer library	85800	Done	Done
D4	Example app to integrate with smart contracts	24000	Final Review	TODO
R1	pre-study and research on Prolly tree literature & projects	39000	Done	Done
R2	Golang IPLD indexer library	7500	Done	Done
R3	Research needs in the (mostly web3) ecosystem	9000	Published	TODO
R4	Research Applications and Feasibility	9000	Published	TODO