Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for returning commits from /history #379

Merged
merged 62 commits into from
Feb 15, 2023
Merged

Commits on Jan 20, 2023

  1. Configuration menu
    Copy the full SHA
    6f0fc92 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1b1a9ca View commit details
    Browse the repository at this point in the history
  3. useful repl workflow

    mpoffald committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    cfd93f2 View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2023

  1. add history query require

    Co-authored-by: Marcela Poffald mpoffald@flur.ee
    dpetran committed Jan 23, 2023
    Configuration menu
    Copy the full SHA
    7ec1c32 View commit details
    Browse the repository at this point in the history

Commits on Jan 24, 2023

  1. update committed db with new tt-id

    This allows us to properly retrieve commit flakes from the cache.
    
    Co-authored-by: Marcela Poffald mpoffald@flur.ee
    dpetran committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    3c0c9c6 View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2023

  1. fix commit schema flakes to use correct @id for dbid

    Both the dbId subject flake and the ledger time had the @id of :f/time.
    dpetran committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    2f92e44 View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2023

  1. rename for clarity

    mpoffald committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    5e70f20 View commit details
    Browse the repository at this point in the history
  2. add commit-details api fn

    mpoffald committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    f810b83 View commit details
    Browse the repository at this point in the history
  3. rm old block api fn

    mpoffald committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    edc20d1 View commit details
    Browse the repository at this point in the history
  4. correct the commit schema flakes

    It turns out if you have a flake with a predicate of `const/$iri` (0), the datatype of the
    flake must be a `const/$xsd:string` (1). Otherwise the query select mechanism will try
    to resolve the flake object's @id, but that requires the flake object to be a subject
    id.
    
    These commit schema flakes had predicate: iri and datatype iri, which was causing
    queries that referenced them to blow up.
    dpetran committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    442a67b View commit details
    Browse the repository at this point in the history
  5. correctly create commit issuer flakes

    The problem was we were only using the `(next-sid)` result in the new issuer flake but
    not in the reference pointing to it. Now we use it in both flakes if we need both
    flakes.
    
    Co-authored-by: Marcela Poffald mpoffald@flur.ee
    dpetran committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    1d40020 View commit details
    Browse the repository at this point in the history
  6. create necessary commit flakes on commit

    We were not creating an @id for the commit, which was inserting nils for the commit
    subject.
    
    Also refactored the creation of commit flakes to make each group of flakes more clear
    and only create new sids when we actually use them.
    
    Updated the s+p+o query test to not expect nil subjects and control for the source of
    non-determinism in a commit.
    
    Co-authored-by: Marcela Poffald mpoffald@flur.ee
    dpetran committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    ce25cfa View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2023

  1. properly prime schema cache on bootstrap

    Co-authored-by: Marcela Poffald mpoffald@flur.ee
    dpetran committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    a9e93aa View commit details
    Browse the repository at this point in the history
  2. put commit schema flakes into bootstrap

    Also refactored commit to generate flakes first and the merge them into novelty.
    
    Co-authored-by: Marcela Poffald mpoffald@flur.ee
    dpetran committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    61cdfd1 View commit details
    Browse the repository at this point in the history
  3. create artisinal commit flakes

    Now we can rebuild the Commit document from a query.
    
    Co-authored-by: Marcela Poffald mpoffald@flur.ee
    dpetran committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    95024af View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2023

  1. add all commit data as flakes

    Changed the commit data to all share the same subject id: t. Added one missing
    vocabulary term for commitdata-address. Now s+p+o returns all flakes with no nils.
    
    Co-authored-by: Marcela Poffald mpoffald@flur.ee
    dpetran committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    e4d6a06 View commit details
    Browse the repository at this point in the history
  2. add missing constants

    dpetran committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    91293ac View commit details
    Browse the repository at this point in the history
  3. Merge branch 'main' into feature/commit-metadata

    # Conflicts:
    #	dev/user.clj
    #	src/fluree/db/api/query.cljc
    #	src/fluree/db/json_ld/api.cljc
    #	src/fluree/db/json_ld/vocab.cljc
    mpoffald committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    bfb1063 View commit details
    Browse the repository at this point in the history
  4. fix tests

    The `vocab/update-with` function didn't do what I expected - it takes a db and returns a
    schema, and I was expecting it to return a db with the updated schema. Instead I swapped
    it with the `vocab/update-with*` which takes a schema and some vocab flakes and returns
    a new schema cache.
    
    Also added the `test-with-files` dep to the :dev deps so I can run the test from
    api-tests from the repl.
    dpetran committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    6de7332 View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2023

  1. only create commit schema flakes when needed (upon committing), + som…

    …e cleanup
    
       We don't need flakes of the type `[s const/$rdf:type o ...]`, now that we have a `.-dt` field on flakes
    mpoffald committed Feb 1, 2023
    Configuration menu
    Copy the full SHA
    fc3cfc1 View commit details
    Browse the repository at this point in the history
  2. begin constructing json-ld commit document containing both metadata a…

    …nd asserts/retracts
    
    Co-authored-by: Daniel Petranek <dpetranek@flur.ee>
    mpoffald and dpetran committed Feb 1, 2023
    Configuration menu
    Copy the full SHA
    6886a26 View commit details
    Browse the repository at this point in the history

Commits on Feb 2, 2023

  1. Configuration menu
    Copy the full SHA
    2d8bb0b View commit details
    Browse the repository at this point in the history

Commits on Feb 3, 2023

  1. Configuration menu
    Copy the full SHA
    375335c View commit details
    Browse the repository at this point in the history
  2. fix destructuring

    mpoffald committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    e086f5f View commit details
    Browse the repository at this point in the history
  3. use existing iris for commit vocab, rather than special iris

    We don't need special iris for commit-related flakes, eg `iri-commitdata-v`. The property `v` is the same property, denoting a version, regardless of whether it appears in a commit document or elsewhere.
    mpoffald committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    6901a0e View commit details
    Browse the repository at this point in the history
  4. move db address outside of asserts list

    `asserts` should include just the actual data transacted by user
    mpoffald committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    f0c663d View commit details
    Browse the repository at this point in the history

Commits on Feb 6, 2023

  1. undo unused changes

    mpoffald committed Feb 6, 2023
    Configuration menu
    Copy the full SHA
    a3d07da View commit details
    Browse the repository at this point in the history
  2. rm troubleshooting def

    mpoffald committed Feb 6, 2023
    Configuration menu
    Copy the full SHA
    150bda0 View commit details
    Browse the repository at this point in the history
  3. Merge branch 'main' into feature/commit-metadata

    # Conflicts:
    #	src/fluree/db/api/query.cljc
    #	test/fluree/db/query/history_test.clj
    mpoffald committed Feb 6, 2023
    Configuration menu
    Copy the full SHA
    1a3f67e View commit details
    Browse the repository at this point in the history
  4. fix assert order

    mpoffald committed Feb 6, 2023
    Configuration menu
    Copy the full SHA
    633f51e View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    7e5b132 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    1e3483f View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    9a26a38 View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2023

  1. Configuration menu
    Copy the full SHA
    077837e View commit details
    Browse the repository at this point in the history
  2. wip history commit-details

    Co-authored-by: Marcela Poffald <mpoffald@flur.ee>
    dpetran and mpoffald committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    3370800 View commit details
    Browse the repository at this point in the history
  3. refactor commit-details to make t-flakes->json more general

    We're now no longer duplicating the key compaction work at several levels.
    
    Co-authored-by: Marcela Poffald <mpoffald@flur.ee>
    dpetran and mpoffald committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    e8a0b61 View commit details
    Browse the repository at this point in the history
  4. update json-ld dependency

    This should fix the cljs tests.
    dpetran committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    d0d8835 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    2b8a3ab View commit details
    Browse the repository at this point in the history
  6. add support for commit-details to history queries

    closes #339
    
    Co-authored-by: Marcela Poffald <mpoffald@flur.ee>
    dpetran and mpoffald committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    46f8e13 View commit details
    Browse the repository at this point in the history
  7. add t-range tests for commit-details

    Co-authored-by: Marcela Poffald <mpoffald@flur.ee>
    dpetran and mpoffald committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    dcde308 View commit details
    Browse the repository at this point in the history
  8. wip consolidate commit and history query api entry point

    These are so close to being the same thing that we thought we'd try them with a unified
    query syntax.
    
    Co-authored-by: Marcela Poffald <mpoffald@flur.ee>
    dpetran and mpoffald committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    ebaa76d View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2023

  1. combine history and commit-details

    Unified the query syntax for history and commit queries. Specifying a time range is now
    required, which should simplify the explanation of the query syntax.
    
    also:
    - removed some dead code
    - updated docstrings
    
    Co-authored-by: Marcela Poffald <mpoffald@flur.ee>
    dpetran and mpoffald committed Feb 8, 2023
    Configuration menu
    Copy the full SHA
    8070449 View commit details
    Browse the repository at this point in the history
  2. update json-ld dependency

    dpetran committed Feb 8, 2023
    Configuration menu
    Copy the full SHA
    9fd2a1a View commit details
    Browse the repository at this point in the history
  3. clean up docstrings

    mpoffald committed Feb 8, 2023
    Configuration menu
    Copy the full SHA
    eae7d45 View commit details
    Browse the repository at this point in the history
  4. tidy up user ns

    mpoffald committed Feb 8, 2023
    Configuration menu
    Copy the full SHA
    53db97c View commit details
    Browse the repository at this point in the history
  5. reduce the number of range calls for adding commit-details to history…

    … result
    
    This updates `add-commit-details` to loop through the commit results and chunk together those results with continuous `t`s. We then retrieve commit details for the range of that chunk, rather than for each result.
    
    In cases where history results have such continuous results, we can reduce the number of times we make `tspo` range calls to retrieve commit data.
    mpoffald committed Feb 8, 2023
    Configuration menu
    Copy the full SHA
    d2a93f8 View commit details
    Browse the repository at this point in the history
  6. remove println

    mpoffald committed Feb 8, 2023
    Configuration menu
    Copy the full SHA
    7c5765a View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    8d09f04 View commit details
    Browse the repository at this point in the history

Commits on Feb 9, 2023

  1. Merge branch 'main' into feature/commit-metadata

    # Conflicts:
    #	src/fluree/db/json_ld/commit.cljc
    #	test/fluree/db/query/misc_queries_test.clj
    mpoffald committed Feb 9, 2023
    Configuration menu
    Copy the full SHA
    9a95e56 View commit details
    Browse the repository at this point in the history
  2. rm unnecessary vocab update

    mpoffald committed Feb 9, 2023
    Configuration menu
    Copy the full SHA
    cb9f13b View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    f4b6c9b View commit details
    Browse the repository at this point in the history
  4. ensure all commit keys are present in commit query

    We had to sneak the previous commit into the commit-flake-addition process in order to
    get the previous commit and commit data ids.
    
    We also had to update how we were constructing the results to not filter the iri flakes
    out of the commit flakes because we need ids in the final map.
    
    Co-authored-by: Marcela Poffald <mpoffald@flur.ee>
    dpetran and mpoffald committed Feb 9, 2023
    Configuration menu
    Copy the full SHA
    e47f8a3 View commit details
    Browse the repository at this point in the history

Commits on Feb 10, 2023

  1. pass previous commit to add-commit-flakes

    Co-authored-by: Marcela Poffald <mpoffald@flur.ee>
    dpetran and mpoffald committed Feb 10, 2023
    Configuration menu
    Copy the full SHA
    151f39e View commit details
    Browse the repository at this point in the history
  2. fix at timestamp bug

    mpoffald committed Feb 10, 2023
    Configuration menu
    Copy the full SHA
    b0e8faa View commit details
    Browse the repository at this point in the history
  3. put one history-flakes->json-ld at a time on output channel

    This way we are not unnecessarily realizing the whole list of history results before going through them again to add commit-details.
    mpoffald committed Feb 10, 2023
    Configuration menu
    Copy the full SHA
    f6f4558 View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2023

  1. remove unused require

    mpoffald committed Feb 13, 2023
    Configuration menu
    Copy the full SHA
    78f9be5 View commit details
    Browse the repository at this point in the history
  2. refactor history query async pipelines

    We removed some unnecessary go-blocks, added more explicit error handling, tried pushing
    the realization of the final results as high up the call stack as we could.
    
    Co-authored-by: Marcela Poffald <mpoffald@flur.ee>
    dpetran and mpoffald committed Feb 13, 2023
    Configuration menu
    Copy the full SHA
    57ee698 View commit details
    Browse the repository at this point in the history
  3. refactor commit details async pipeline

    We changed the default sort order of the history query to be in ascending order by t -
    earliest change first. This matches the default tspo sort order, so we no longer have to
    sort the commit flakes. History still needs to be sorted because sometimes it uses
    non-tspo indexes to satisfy the query.
    
    Moved the async into calls as high up the call stack as we could, and removed
    unnecessary go blocks.
    
    Co-authored-by: Marcela Poffald <mpoffald@flur.ee>
    dpetran and mpoffald committed Feb 13, 2023
    Configuration menu
    Copy the full SHA
    b232caf View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2023

  1. properly treat time-range results as a stream instead of a promise

    time-range returns a stream of flake slices, where each slice is the subset of flakes
    from an index leaf. Up to now we have been treating this as a promise and only taking
    the first result off the time-range chan. Now we treat it as a proper stream.
    
    Co-authored-by: Marcela Poffald <mpoffald@flur.ee>
    dpetran and mpoffald committed Feb 14, 2023
    Configuration menu
    Copy the full SHA
    9a0ef8f View commit details
    Browse the repository at this point in the history
  2. remove comment

    mpoffald committed Feb 14, 2023
    Configuration menu
    Copy the full SHA
    6104e6f View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    102b79a View commit details
    Browse the repository at this point in the history

Commits on Feb 15, 2023

  1. remove unused refer

    dpetran committed Feb 15, 2023
    Configuration menu
    Copy the full SHA
    aec1754 View commit details
    Browse the repository at this point in the history