Support for returning commits from /history

[mpoffald]

Related issues

Closes #333
Closes #359
Closes #342

If merged, fluree/http-api-gateway#7 will not be required.

Overview

This PR extends the history api with the ability to return commits for a given time range.

This is done using flakes to represent commit data: we amended the flakes we were already inserting upon commit to include all the necessary parts of the commit. These are then retrieved via time-range calls and assembled into json-ld maps.

New Syntax

• optional :commit-details flag. If true, returns commit data, eg:
  - {:history :ex/alice :commit-details true :t {:from 1 :to 5}}: amend history results with commits for any ts reprsented in those results
  - {:commit-details true :t {:from 1 :to 5}} : just the commits for time range t
• New t options:
  • :latest uses the db's t:
    • {:history :ex/alice :commit-details true :t {:from 1 :to :latest}}
  • :at can be used instead of from and to (sugar for equal from and to values):
    • {:commit-details true :t {:at :latest}}
  • Note a t is now required for all history queries, with or without commit details. Previously, the caller could omit ts to range over all of history. Now you would need to pass in {:from 1} to do that.

Example commit response:

A map keyed by :f/commit :

{:f/commit
  {:f/address "fluree:memory://14efe5ef1163589f9ee6fcc5af8e2830c3300a80abbbae0aa82194083e93aebe",
    :f/v 0,
    :f/previous {:id "fluree:commit:sha256:bbr54svhy4ergg3mmed4eugzljonsl7jlfmadsbcq6b7sr2cs7yyl"},
    :f/time 720000,
    :id "fluree:commit:sha256:bb3v3rd7q6lojz5w4gsweglna6afhc6ooh5fgfdhxitdtdwquxwgv",
    :f/branch "main",
    :f/message "meow",
    :f/data  {:f/previous {:id"fluree:db:sha256:bt6d6eup2oo2icrjj7ejcmfoa3wsen45y2l2iomlnbqsuoypizgn"},
              :f/address  "fluree:memory://a505e939983b4cfa9325ebe0bbd8615865606c3f4ab9343dd00d594ee722cc10",
              :f/flakes 102,
              :f/size 9408,
              :f/t 5,
              :f/assert [{:ex/x "foo-cat", :ex/y "bar-cat", :id :ex/alice}],
              :f/retract [{:ex/x "foo-3", :ex/y "bar-3", :id :ex/alice}]},
    "https://www.w3.org/2018/credentials#issuer" {:id "did:fluree:TfCzWTrXqF16hvKGjcYiLxRoYJ1B8a6UMH6"},
    :f/alias "committest",
    :f/context "fluree:memory:///contexts/b6dcf8968183239ecc7a664025f247de5b7859ac18cdeaace89aafc421eeddee"}}

This inlines the data we usually store in a separate file into the data field of the commit map (see #333 (comment) for relevant discussion).

Implementation notes

Why add commit flakes instead of grabbing the commit from storage?

• Retrieving flakes allows us to apply permissions on the data we return
• In cases where users request multiple commits, we would have to retrieve from storage for every single commit. This could be very expensive.
• We were already adding flakes to represent some of the commit data. This PR just ensures that list of flakes is complete.

If we don’t want any commit-related flakes at all, then we should consider a future solution that allows us to retrieve commit data some other way (besides storage).

Future considerations: distinguishing fluree-generated data from user-relevant data

We had to do some workarounds to make sure the asserts/retracts only reflected the data asserted/retracted by the user, and omitted fluree-generated flakes that are just to make our implementation work. Because fluree-supplied flakes (such as commit flakes) and user-supplied changes live alongside each other in the db, we need to filter them out ourselves.

For now, we’ve just hard-coded (eg commit-metadata-flake? in the history ns). This is similar to something we already do by hand when creating commit files (see the commit/generate-commit fn)

However, if the distinction between internal metadata and "user" data continues to be important, we should consider a more principled way to sequester fluree-generated data.

[mpoffald]

You can see we get back a lot of fluree-generated commit data now if we pull the whole db after committing. This wasn't happening before because we had a caching bug that kept the commit flakes from being discoverable at all, even though they were there (see #333 (comment) for discussion).

[mpoffald]

We used to also add flakes for commit tags here. We didn't include them in this PR because we weren't clear on the status of tag-related features, but if anyone thinks they should be here, we can add them back.

[mpoffald]

To add commit details to history results, we:

• do the regular history query, which tells us which ts are relevant.
• we then go get commit details for those ts -- getting commits requires a tspo range scan

That tspo range scan only works on ranges, so if we have sparse/non-consecutive ts to consider, right now we make range scan call for each. If we need commits for ts 1-3 and 5, we scan:

• from t=1 to t=3
• then again from t=5 to t=5

Ideally, there's some solution where we don't do multiple passes, or where this is less expensive than it is now. For now, this Works (TM).

I did add this little optimization to at least chunk together consecutive ts to reduce the number of calls in some cases, but I still hold out hope to do something lower-level in the future.

This optimization could still be improved a bit though: the history-flakes->json-ld fn returns all the results in one list on the output channel. I'd like to change it to put one result at a time, so that this fn can read off the channel rather than us fully realizing the list just to go through it again.

I'm going to work on that while this is under review, but I'd welcome any pointers from people who know more about core.async than I do.

[mpoffald]

I just pushed a commit that changeshistory-flakes->json-ld to just put one result at a time on the channel: f6f4558

The rest of the above comment still applies

[mpoffald]

Apologies for the enormous diff here, I wanted to move the history-specific helpers and syntax stuff into its own ns.

[mpoffald]

This fixed a bug where we we were writing out a stringified async channel instead of the actual signed commit, eg:

#object[clojure.core.async.impl.channels.ManyToManyChannel 0x66731af8 "clojure.core.async.impl.channels.ManyToManyChannel@66731af8"]%

[zonotope]

There are a few things I think we should change before merging. There are a few places where we wrap things in go blocks when it's not necessary. Every go block creates a channel, and every put and take off of a channel has a cost, so we should try to minimize the channels we create.

Secondly, there are a lot of places where we use <? and go-try where I don't think we should. There are two ways to use core.async channels. One is like an analogue of a promise, and another is like an async analogue of a seq. The promise analogue returns a single result of an async process "later". That's what we should use <? and go-try for, because it makes sense to check the single contents of the channel for an exception.

The seq analogue is different however. Higher level sequence processing falls down hard once some element of a sequence could be an exception. Higher order functions like async/map, async/reduce, async/transduce, transducers attached to channels like what you get from partition-by, map, filter, etc, and calls to the pipeline-* functions don't mix when there could be an exception on a channel at any point. I think it's best to make that explicit by using go and <! in those cases, with the addition of an explicit error channel that we can use to "short-circuit" processing if there's any part of the asynchronous process generating the sequence that has an unrecoverable error.

Lastly, and this is probably too heavy of a lift to address in this pr, but I also think there are areas where we could improve performance here by changing the order of processing a bit. We are wasting a lot of time and space by collecting all the history flakes in a single vector in time-range, only to immediately split that vector again multiple times (at least once for history-flakes->json-ld, and then again for add-commit-details). I think we can save on this memory thrashing by processing the flakes from the tspo index lazily as they are loaded by reimplementing the logic here to use more transducers and more calls to pipeline-async. That isn't the smallest lift, so I'm fine with merging this for now after the other cleanup with plans to refactor later.

[zonotope]

Using go-try here implies that the code inside can throw exceptions, and that those exceptions would be caught and placed on the output channel. There's nothing to read that output channel created by go-try to handle any errors that might be on it though (this is wrapped in a pipeline-async call, which will only read the passed in ch argument to the async fn. Any new channels returned by the async fn will be ignored). If any of the code inside does throw errors, those errors aren't being handled.

I think we should be explicit here about exceptions. If the code inside can't throw an exception, then we should replace go-try with go.

If it can throw exceptions, then we should replace go-try with (go (try* ... (catch* ..., and put any caught errors on the error channel so they can be handled downstream.

[zonotope]

It looks like we're wrapping this code in a go-block (which has a cost) for no reason. There are no asynchronous puts or takes in the wrapped code until the very end. Instead of wrapping this code in a go-block that doesn't do any async stuff before it terminates, you could rewrite this function like this.

(defn t-flakes->json-ld
  [db compact cache fuel error-ch t-flakes]
  (let [s-flake-ch (->> t-flakes
                        (group-by flake/s)
                        vals
                        async/to-chan!)
        s-json-ch   (async/chan)]
    (async/pipeline-async 2
                          s-json-ch
                          (fn [assert-flakes ch]
                            (-> (s-flakes->json-ld db cache compact fuel error-ch assert-flakes)
                                (async/pipe ch)))
                          s-flake-ch)
    (async/into [] s-json-ch)))

That saves a channel and doesn't change the behavior of this function from the perspective of the caller.

[zonotope]

It looks like this function is only called once. I think it'll be clearer if we apply the code inside to the t-flakes directly on line 127.

[zonotope]

Depending on how you want to use the output of this function, it's usually much more flexible and performant to save the step of fully realizing an async channel and collecting its contents into a vector to the very end of your code path right before you return data to the user. That way downstream code can filter, limit, or even ignore the stuff on the channel entirely without us paying the penalty of loading the whole thing if we didn't actually need it all. It's also much easier to reason about, apply transducers to, and include in a pipeline-* chain a channel-of-x's instead of a channel-of-collections-of-x's

[zonotope]

Could we explicitly compute these keys (and possibly the takes off of these channels too) in the let binding instead of inline while creating this output map? It was a bit difficult for me to tell what was going on here, and computing this stuff in the binding gives the opportunity of implicit documentation with the binding names.

[zonotope]

If there are multiple items on the history result channel (implied by the loop below), then there should never be an exception on it. If something needs to throw an exception at some point of the process of generating history results, then that exception should be handled in a separate error channel instead of mixed in with valid results. <? should be used for channels that will only contain a single item, and that item might be an exception.

[zonotope]

I think we're compacting this iri-t constant multiple times with the same context. I think that implies that this add-commit-details function should come higher in the chain of computing history results so you can access the data you need before it's compacted.

[zonotope]

It looks like you're wrapping a channel with a channel here, which will lead to very weird results. It doesn't look like there are any async puts or takes in here, so I think you can just remove this go block all together.

[zonotope]

You're calling time-range here directly, and then further processing the flakes it returns. The problem with this is time-range collects its results into a single vector, and then you split that vector again and process it asynchronously for each t in the results. There's a relatively big cost in collecting all the results in memory only to split them up again for further processing.

Ideally we'd only collect results at the last moment before they are returned to the user. This would use both less time and less space. You can do this by implementing a function in terms of tree-chan that returns a channel that will contain a sequence of flake-chunks where there's one chunk for each t by passing the appropriate transducers to tree-chan. This function would probably look a lot like time-range without that collection step.

Then pass the channel returned by this new function with the logic in history-flakes->json-ld without the code to partition by t to process each chunk of flakes with pipeline-async to transform each flake chunk into a jsonld map.

Then, I would reimplement add-commit-details as a transducer + an async function to process each jsonld map and add commit details if necessary.

[zonotope]

It looks like this loop here does two things: 1: groups consecutive t's into chunks, and 2: processes each chunk asynchronously to add commit details. It's definitely more efficient space-wise, and probably more efficient time-wise to reimplement this function in terms of a transducer that does step 1 and a pipeline-async call that does step 2.

[zonotope]

Nice work on the cleanup

[zonotope]

It doesn't look like this into? is used anywhere.