Daily Notes not showing up in Chronological Order

[itsrainingmani]

Problem

After creating a certain number of Daily Notes, they no longer show up in chronological order.

Expected Behavior

Daily Notes should always be in chronological order irrespective of how many daily notes have been created

To Reproduce

• Go to Daily Notes page
• Create atleast 10 new daily notes by scrolling down

Screenshots

https://www.loom.com/share/7fa0eeb7148d4da68e354d2e254e002b

[itsrainingmani]

@tangjeff0 Can you assign this to me (again) 😅 ?

[tangjeff0]

You can't assign yourself? That's why I added you all to /devs team lol.

Also, did you get to read my comment about the "Jump to Timeline" on the original issue?

[itsrainingmani]

I wasn't able to assign myself. Not sure why.

Unfortunately I didn't get to read it. Do you remember what you wrote?

[tangjeff0]

Using ascending dates to debug was a good move. I feel that is-timeline-page is hacky, but it's based on Roam's US date format and doesn't seem to be the problem here, because the re-frame vector is correct.

You should know that I would eventually like to have a "Jump to Timeline" feature, mentioned in #66. This would allow a user to jump to any date in the past and scroll up or down to go to neighboring dates. Right now in Roam, you can only scroll down starting from today.

Basically, if you are on a date page, you would see a little button like < Timeline near the title to take you to a scrollable date page. This may inform your solution here (you would have to append OR prepend the vector).

Also, the vector should be cleared when you navigate away from daily pages. Don't think that's the problem, just something to keep in mind.

[itsrainingmani]

Awesome! Thanks Jeff.

[itsrainingmani]

The issue is beyond just daily notes. Even using Athena to create 9 pages in a row messes up the internal order of the pages.

[nthd3gr33]

Documenting some things from convo with @itsrainingmani:

if you fire up a clojure repl and you make a map and conj stuff into it, you'll notice that first ~8 elements have insertion order preserved as well. and the 9th insertion re-arranges the order

;; small maps stay as 
(def mapA {:a 1 :b 2})
mapA   ; => {:a 1, :b 2}

;; somewhere around 9 elements leads to rearranging
(def mapB {:a 1 :b 2 :c 3 :d 4 :e 5 :f 6 :g 7 :h 8 :i 9})
mapB   ; => {:e 5, :g 7, :c 3, :h 8, :b 2, :d 4, :f 6, :i 9, :a 1}

(see this clojuredocs.org link for reference)

My initial thought was to attempt to override this behavior somehow (naive thinking), but @itsrainingmani informed me that

datascript doesn't use a plain ol clojure map internally. it uses a custom implementation of a B+ Tree Structure

See this excerpt from a relevant post about Datascript internals from Tonsky's blog.

We are looking into B+ tree structures starting with the wiki. Will update as we make more progress.

[itsrainingmani]

I think the issue here is in the order that DataScript returns the results of a query rather than an issue with daily-notes since the all_pages table order also seems to be altered after 8 or so page creations.

The number 8 is interesting because the insertion order seems to be "preserved" until a 9th element is added which then changes the order. However, the DataScript indexes (EAVT, AVET, AEVT) don't seem to change.

It seems like there is no order guarantee for DataScript queries as far as I can see.

Now why does DataScript preserve order for some number of items? My hypothesis is that this is a function of the B+ Tree structure.

After a certain number of elements, (in this case 8), the tree rebalances it's buckets.

Solutions -

• Sort the result of the query. The tradeoff is that for large result sets, this could be expensive.
• Use the datoms function to query the DataScript indices directly for ordered data. https://cljdoc.org/d/datascript/datascript/0.18.13/api/datascript.core#datoms

[tangjeff0]

Great research gang.

1. q is set-based, so we can't expect any ordering.

      (let [eids (q '[:find [?e ...]
                      :in $ [?uid ...]
                      :where [?e :block/uid ?uid]]
                    db/dsdb
                    @note-refs)]

2. I do think pull-many will return sorted results, but it might be worth double-checking.

(let [notes (pull-many db/dsdb '[*] @eids)]

We should look into places wherever we make larger queries such as Athena and linked references and see how we want to sort results if we want to at all. See #20 (comment)

---

Regarding your solutions, even pulling all pages without sort is expensive. See Roam's. @jelmerderonde also has paginated results in datascript devtools. I don't think having limits in q makes a difference during the pull, but it does make rendering the view faster.

[sid597]

Went back ~20 notes, everything is correct and can't reproduce it.