feat(PE-2256): define chunk data cache interface and implementation #12

dtfiedler · 2022-09-20T17:24:23Z

Details

Creates new class FsChunkCache that implements ChunkSource and ChunkDataCache interfaces
Checks data/chunks/<data_root>/data/<relativeOffset> for cached chunk data
Falls back to retrieving chunk data from ChunkSource if chunk data is not found in the cache, then saves them to cache as unencoded Buffer's for future retrieval

src/cache/fs-chunk-cache.ts

…rweave client

…ache

src/arweave/composite-client.ts

djwhitt · 2022-09-21T14:56:11Z

src/cache/fs-chunk-cache.ts

+}
+
+function chunkCachePath(dataRoot: Buffer, relativeOffset: number) {
+  return `${chunkCacheDir(dataRoot, relativeOffset)}/${relativeOffset}.msgpack`;


This ticket is intended to only address data caching (see the "Note: ..." at the bottom of the ticket description). We should be storing raw data bytes without any MessagePack encoding.

djwhitt · 2022-09-21T14:58:39Z

src/types.d.ts

@@ -132,6 +138,16 @@ export interface PartialJsonTransactionCache {
  set(tx: PartialJsonTransaction): Promise<void>;
 }

+export interface JsonChunkCache {


Same comment as above, and we'll likely want a different interface interface for metadata when the time comes for that. I think a combo of my insufficient explanation + taking queues from header caching is leading you astray. 🙂

changed this here f917a07

djwhitt · 2022-09-21T15:09:31Z

src/lib/encoding.ts

@@ -194,6 +196,24 @@ export function msgpackTxToJsonTx(
  };
 }

+export function msgpackChunkToJsonChunk(


These aren't technically needed since this ticket is only about data rather than metadata caching, but we may need something similar for upcoming tickets. However, for the metadata use case, they shouldn't include the actual chunk data.

going to remove for now, will add back with that ticket

src/types.d.ts

djwhitt · 2022-09-21T15:14:47Z

src/cache/fs-chunk-cache.ts

+    relativeOffset: number,
+  ): Promise<JsonChunk> {
+    try {
+      const chunkPromise = this.get(dataRoot, relativeOffset).then(


Remove the caching logic from this and let it always fall through to the network. Metadata caching will be addressed in a future ticket.

modified in c1de268

djwhitt · 2022-09-21T15:15:27Z

src/cache/fs-chunk-cache.ts

+    dataRoot: Buffer,
+    relativeOffset: number,
+  ): Promise<Readable> {
+    const { chunk } = await this.getChunkByRelativeOrAbsoluteOffset(


Add logic here to cache just the chunk data.

now caching only chunk data as unencoded buffer c1de268

djwhitt

Yeah, this is what I was trying to describe in the ticket. Thanks for the update!

feat(PE-2256): define chunk cache interface and implementation

a1e9bb6

dtfiedler commented Sep 20, 2022

View reviewed changes

src/cache/fs-chunk-cache.ts Outdated Show resolved Hide resolved

dtfiedler added 2 commits September 20, 2022 13:29

feat(2256): implmenet chunk cache on arweave composite client

46a41b9

chore: linting

8cbe7eb

dtfiedler force-pushed the PE-2256-chunk_cache branch from f08e06a to 8cbe7eb Compare September 20, 2022 19:33

dtfiedler marked this pull request as ready for review September 20, 2022 19:35

dtfiedler requested a review from djwhitt September 20, 2022 19:36

dtfiedler marked this pull request as draft September 20, 2022 19:40

dtfiedler added 5 commits September 20, 2022 13:58

refactor(PE-2256): implement ChunkSource in FsChunkCache and revert a…

42a223c

…rweave client

chore: pass down buffers for data root when storing/retrieving from c…

ce60089

…ache

chore: improve logging of data_roots

26c2963

chore: remove redundant validation on chunks when fetched from cache'

817419b

chore: linting

61b969d

dtfiedler force-pushed the PE-2256-chunk_cache branch from bf9824d to 61b969d Compare September 20, 2022 20:17

fix: cleanup and reset ttls

38a7e3f

dtfiedler force-pushed the PE-2256-chunk_cache branch from 221bbe9 to 38a7e3f Compare September 20, 2022 20:23

dtfiedler marked this pull request as ready for review September 20, 2022 20:25

ar-io deleted a comment from ario-bot Sep 20, 2022

dtfiedler changed the base branch from main to develop September 20, 2022 21:25

djwhitt requested changes Sep 21, 2022

View reviewed changes

dtfiedler added 4 commits September 21, 2022 13:15

refactor: revert composite client node cache

b4dd076

refactor: remove msgpack encoding

ab39b4c

refactor: modify chunk data cache interface, remove partial for now

f917a07

refactor: only cache chunk data, fetch metadata from nodes

c1de268

dtfiedler changed the title ~~feat(PE-2256): define chunk cache interface and implementation~~ feat(PE-2256): define chunk data cache interface and implementation Sep 21, 2022

refactor: more cleanup

4a704f7

dtfiedler force-pushed the PE-2256-chunk_cache branch from 5086f07 to 4a704f7 Compare September 21, 2022 19:57

chore: minor logging change

0f01ed2

dtfiedler requested a review from djwhitt September 21, 2022 20:01

feat: add error handling and logging

f8b7d80

djwhitt approved these changes Sep 22, 2022

View reviewed changes

dtfiedler force-pushed the PE-2256-chunk_cache branch from eca4c3c to c24330f Compare September 22, 2022 04:47

feat: add tests to validate cache vs. network retrieval

9caa07a

dtfiedler force-pushed the PE-2256-chunk_cache branch from c24330f to 9caa07a Compare September 22, 2022 13:50

dtfiedler merged commit 151bd15 into develop Sep 22, 2022

dtfiedler deleted the PE-2256-chunk_cache branch September 22, 2022 14:20

github-actions bot mentioned this pull request Oct 18, 2023

chore(main): release 1.0.0 #57

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(PE-2256): define chunk data cache interface and implementation #12

feat(PE-2256): define chunk data cache interface and implementation #12

dtfiedler commented Sep 20, 2022 •

edited

Loading

djwhitt Sep 21, 2022

djwhitt Sep 21, 2022

dtfiedler Sep 21, 2022

djwhitt Sep 21, 2022

dtfiedler Sep 21, 2022

djwhitt Sep 21, 2022

dtfiedler Sep 21, 2022

djwhitt Sep 21, 2022

dtfiedler Sep 21, 2022

djwhitt left a comment

feat(PE-2256): define chunk data cache interface and implementation #12

feat(PE-2256): define chunk data cache interface and implementation #12

Conversation

dtfiedler commented Sep 20, 2022 • edited Loading

Details

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djwhitt left a comment

Choose a reason for hiding this comment

dtfiedler commented Sep 20, 2022 •

edited

Loading