SDK V2 Prerelease - (Slot based storage) #2901
Conversation
…or hashing now, simpified how the slot file is choosen
🦋 Changeset detectedLatest commit: d7bc246 The changes in this PR will be included in the next version bump. This PR includes changesets to release 27 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
…o transient slotfiles over exisistng files, cleanup, bugfixes and serialization of persistence
…ent-to-martins-test-app feat: add bundle component to martins test app
# Conflicts: # inlang/source-code/message-bundle-component/src/stories/inlang-variant.ts # inlang/source-code/sdk/src/v2/helper.ts
notes from initial slotfile storage explorationSlotfiles are an on-disk text file format designed to play nicely with git. They persist inlang message data into discreet slots in files, thereby avoiding collisions and minimizing merge conflicts. High level design
Questions
Nits
|
|
Thanks for the feedback @jldec
👍
👍
👍
👍
It checks for a slot in the transient file / files if one excists first (transient files are about to be created - and slots used in those can only be occupied by the current process so conflicts can not occur). If no transient file exists yet (the normal case for non batch operations) it will check the files existing files by the nearest files by name first for a free slot. If all slots in existing files are occupied it will create a transient file.
👍
To understand this the following properties are important:
During a saving to disc routine, the transient records tries to find a slot in existing files (to reduce file creation to a minimum) if no free slot is found a transient file is created (for the lifetime of a save operation). A transientFile is also taken into account to find free slots for other transient records (compare slot search from comment above)
This is an extra roundtrip to collect changes on the disc that may happen right before the save. This is not a guarantee to catch all possible changes from other processes but since we will (TODO in place) block incoming event processing during the save an extra load was placed. Definitely a need to clearify and Optimize with the introduction of the lock.
it should be „ASDasd“ but I guess my fingers gone crazy when I needed a log I can look for during debugging 🤣
🕵️♂️ don’t know without looking into the code Will check tomorrow
🕵️♂️ Will check - as well
Sure - I started writing some todos in the .test.ts file please add everything you would like to see
Disagree - I commit WIP and build only what I need in my focus, everything that kicks me out of the flow - but now it’s time to clean up - on it!
👍
👍
🤓👍 |
@marting-lysk lintCOnfig need the installedLintRules to show UI with good UX
…omponent-is-not-reacting-on-updates fix: example lintConfig -> added installedLintRules
| const slotStorage = await createSlotWriter<DocumentExample>({ | ||
| fs: memoryFs, | ||
| path: "/slot/", | ||
| fileNameCharacters: 3, |
There was a problem hiding this comment.
use a sane default, i expect this will rarely ever need to be touched.
inlang/source-code/sdk/src/persistence/slotfiles/createSlotWriter.test.ts
Outdated
Show resolved
Hide resolved
inlang/source-code/sdk/src/persistence/slotfiles/createSlotWriter.test.ts
Outdated
Show resolved
Hide resolved
| // expect(Object.isFrozen(docs1[0])).eq(true, "Records returned by slotStorage should be frozen") | ||
| // eslint-disable-next-line @typescript-eslint/no-non-null-assertion -- we check the length before | ||
| expect(docs1[0]!.data.content).eq(insertedDocument.content) | ||
| const updateDocument = structuredClone(insertedDocument) |
There was a problem hiding this comment.
why is clone needed? if storage can break without cloning, this has to move into the library
| expect(slotStorage._writerInternals.transientSlotEntries.size).eq(1) | ||
|
|
||
| await slotStorage.save() | ||
| expect(slotStorage._writerInternals.transientSlotEntries.size).eq(0) |
There was a problem hiding this comment.
aahhhhh so this is really a transient storage, not just a bad name for a memory layer on top of the file?! this is the first time i understand it is really working like that., maybe i misread this before? if you just updated the memory representation of the file directly an then reconcile when saving you could reuse the same conflict logic as for writes by multiple threads and the whole impl would be simpler. i dont fully understand the need for this 3 layer design.
inlang/source-code/sdk/src/persistence/slotfiles/createSlotWriter.test.ts
Outdated
Show resolved
Hide resolved
|
|
||
| await slotStorage1.save() | ||
| await slotStorage2.save() | ||
| const conflictingRecord = slotStorage2.findDocumentsById([insertedDocument.id])[0] |
There was a problem hiding this comment.
isnt locking supposed to prevent this case? we need locking in any case to now destroy the file write on concurrent threads, so it should be simple to just prevent any writes from other threads concurrently.
…-and-languagetag-in-the-settings-component Nilsjacobsen/inlmc 104 use locale and languagetag in the settings component | INLMC 🧩
…ntation-for-message-bundle-component feat: add docs for the message bundle component | INLMC 🧩
Add RPC-FS to SDKv2 branch
|
Closing because outdated. Inlang SDK v2 is rebuild on sqlite |
Iteration 1 (Prerelease 15.7.)
Meta TODO's:
TODO's
@nils.jacobsen
MessageBundle Editor
Settings Component
@loris.sigrist
@martin.lysk1
Iteration 2 (Experimental release 30.7.)
@loris.sigrist collects requirements (utility functions for upgrades)
...
Iteration 3