Persistent storage data for plugins that also get synced

[danielb2]

overview

currently there's no way for a plugin to persist data that will also get synched.

use-case

I've written a plugin to publish notes to gist. because I don't have persistent storage for the plugin across devices, I make a new gist for each note. It would be nice to be able to link a noteid to a gistid and store it across

implementation suggestion

I'd imagine something like (for osx)

$HOME/.config/joplin-desktop/plugins/io.github.jackgruber.backup/ where the plugin is sandboxed to only be able to write there, and then maybe a joplin.fs.read/write function to access the files there.

Spec

• Data must be synchronised across all devices
• Plugins only have access to key/values of their namspaces
• Include the data in the RAW, JEX, and MD exports
• Set a limit to how much data can be stored because storing too much either due to a bug or on purpose could cause issues with sync
• Each key must have a timestamp associated with it, and that will be used to resolve conflict
  • A simple conflict resolution would be to keep the value with the latest timestamp

references

1. https://discourse.joplinapp.org/t/best-way-to-persist-plugin-specific-data/14248/3

[JackGruber]

Some links to discourse where this topic has already been mentioned

https://discourse.joplinapp.org/t/feature-request-discussion-custom-api-metadata/2474/9

https://discourse.joplinapp.org/t/lets-add-the-possibility-to-plan-todo/20781/5

https://discourse.joplinapp.org/t/feature-request-custom-properties-for-notes/22574

[laurent22]

I've added a requirement "Set a limit to how much data can be stored because storing too much either due to a bug or on purpose could cause issues with sync"

Any suggestions on what that limit could be?

[Daeraxa]

I guess it depends if it should be entirely open or just designed for some plain text stuff. Do we want to allow BLOBs in there for example? Or should it all be human readable data?

Would there be any real reason to go above 100kb of plaintext for example? That is already quite a lot of data if it was only text.

[roman-r-m]

Would there be any real reason to go above 100kb of plaintext for example?

Indexing resources for full-text search for instance.

[danielb2]

Is there an API which returns the syncdir location. Because I think that would be enough, at least for those options that use the local filesystem to store files.

[roman-r-m]

Is there an API which returns the syncdir location. Because I think that would be enough, at least for those options that use the local filesystem to store files.

You could try something like joplin.settings.globalValue("sync.2.path")
It's not documented but I think it should work.

[danielb2]

😍 thanks!

[danielb2]

@laurent22 I've run npm run update and I'm getting Property 'userDataGet' does not exist on type 'JoplinData'.

I don't see the function defined in the api/ folder

P.S I worked around it by manually adding this in JoplinData.d.ts for now, but I think the yeoman generator is missing this

[laurent22]

Oh that's right, the generator needs to be released

[laurent22]

Ok the generator should be up-to-date now with the latest types

[danielb2]

Ok the generator should be up-to-date now with the latest types

[~/w/o/joplin-publish]$ yo joplin --update
Error joplin --update

Cannot use 'in' operator to search for 'defaults' in npm

[laurent22]

Hmm, that's not good, I'll check.

Which version of Yeoman do you have? (yo --version) And could you check with 3.1.1? (npm install -g yo@3.1.1)

[danielb2]

4.3.1 and using 3.1.1 I get

yo --version
3.1.1
[~/w/o/joplin-publish]$ yo joplin --update

Error joplin --update

This generator (joplin:app) requires yeoman-environment at least 3.0.0, current version is 2.10.3, try reinstalling latest version of 'yo' or use '--ignore-version-check' option

[~/w/o/joplin-publish]$ yo joplin --update --ignore-version-check
Current yeoman-environment is not compatible with current generator, min required: 3.0.0 current version: 2.10.3. Some features may be missing, try updating reinstalling 'yo'.
Current yeoman-environment is not compatible with current generator, min required: 3.2.0 current version: 2.10.3. Some features may be missing, try updating reinstalling 'yo'.
Error joplin --update --ignore-version-check

Cannot use 'in' operator to search for 'defaults' in npm

[laurent22]

Maybe try to uninstall and reinstall the yo package?

npm uninstall -g yo && npm i -g yo

And if it doesn't work either, try to remove yeoman-environment and reinstall

npm uninstall -g yeoman-environment && npm uninstall -g yo && npm i -g yo

[danielb2]

the issue persists. I take it you're not seeing it. nodejs version? v16.20.0 for me

[danielb2]

just tried 18.16.0 and 20.2.0 with the same results

[danielb2]

#7877

Looks like someone else had this error before

[laurent22]

Yes i think the generator needs to be updated to support the latest version of Yeoman

[laurent22]

@danielb2, the generator should be fixed now, please try updating it again and see if it works

[laurent22]

I've also added a function npm run updateVersion to update and keep the package.json and manifest.json versions in sync

[danielb2]

I'm getting this error now

$ yo joplin --update

Error joplin --update

Cannot set properties of undefined (setting 'force')

I tried this too, but still persists npm uninstall -g yeoman-environment && npm uninstall -g yo && npm i -g yo

I poked around reading a bit about yeoman etc, but I can't find anything that explains this error

[laurent22]

That's annoying, and unfortunately Yeoman's error reports are terrible. Could you try to enable the debug flag to see if it prints more errors? DEBUG=yeoman:generator yo joplin --update

[danielb2]

that was one of the things I tried after reading the docs (macos has a different debug command) but it availed me no info at all. I reviewed the generator and docs and I didn't find anything suspect at all that would cause this error.

short of looking at the yeoman source code, I wouldn't know where to go from there. I can take a look later after work perhaps

[laurent22]

Yes they have some onError handler that pretty much silences all errors. Typical JS project I guess.

For the previous error I indeed fixed it by going through the Yeoman source code, which thankfully is all in one file and not minified. If you can add a few console statements here and there, you should find at what point it stops. Or maybe a search and replace to change all onError to console.error would work to see where the error is.