FR: Centralized Sidecar directories #9265
Comments
|
Try this: In the koreader directory create a directory called sdr. local DATA_SDR_DIR = DataStorage:getDataDir() .. "/sdr/"
function DocSettings:getSidecarDir(doc_path)
if doc_path == nil or doc_path == '' then return '' end
local file_name = doc_path:match(".*%/(.*)%.")
if file_name then
return DATA_SDR_DIR .. file_name .. ".sdr"
end
file_name = doc_path:match(".*%/(.*)")
if file_name then
return DATA_SDR_DIR .. file_name .. ".sdr"
end
return doc_path .. ".sdr"
end |
|
However you will need to do this after every update |
|
I'm not planning to do this as a one-time thing. My intention is to implement this in the KOReader. |
Definitely not a win-win. Imagine trying to sync between two devices. |
Syncing via plugin? That only syncs progress now, right? Or, syncing via third-party methods? |
|
By syncing I mean any action that copies the file and its metadata, no specific method. You can do it by hand. (But no, not the progress sync.) |
|
It also just means organizing your own files using your computer. |
|
I see. Can't we make a setting to handle this? People can choose where they want there sidecars to be. |
|
That setting already exists, doesn't it? |
|
Or something like #8577 |
Okay. Couldn't find it though. |
|
See #4831 (comment) |
|
Even more links in #8281 (comment) There is no setting yet. Everybody gets .sdr/ folders alongside the books. It should be a setting, and I think it should default to what we have currently. Having subdirectories making the full path of the file is obviously better than the somehashortruncatedbutsinglefile idea, which is not reversible (you can't guess the fullpath from this somehashortruncatedbutsinglefile) - and you would still have all the .old backup and ffiutil.fsyncOpenedFile() stuff working as-is. |
|
That's a patch, not a setting. And a very bad patch I guess. I'll loose data, right? |
Well, I didn't remember the exact details about this thing I don't care for one iota. ;-) |
Or, instead of centralizing, which means much work, we can hide the folders (at least in unix-like systems) by prepending a dot like another FR suggested. |
|
What's the current situation on Kindle @NiLuJe? |
Here's a confusion, and I'm responsible for that. In the original post I wanted to move the directory, not making it a Currently the sidecar is: I wanted it to be: |
|
^ That's how I understood it - so my bad if you thought I wasn't on track :) |
Not necessarily. You can use #9104 to apply user patches.. Put you code into n=0 means to execute very early on startup (once after an update). n=1 means to execute very early on every startup. n=2 means a bit later on startup xxx may be choosen as you like. I think you can use You can find an example for patching in https://gist.github.com/zwim/1edcd34ef8a59166f203d5ee8c08f7e3 |
Same as everywhere (that isn't Android ;p) else (in fact, the whole sidecar folder thing was designed after how the stock reader does this) ;). |
|
FWIW, I am completely unbothered by the current situation, but at least what @uroybd proposes is much less fugly than the current ro fallback ;). |
My bad, I didn't express myself clearly. I meant, does the stock reader still do that? :-) |
@hius07 did answer a similar question of mine at #8281 (comment) |
|
TL;DR: Yes, and there's even more stuff in there than it used to ;p. |
|
Now... Should we centralize sidecars or make them hidden with dots? Of course configurably. |
|
The fact is that KOReader will have to find any sidecar dir/file in any past, present or future possible locations.
So, if you want to keep this nice idea which wouldn't need any migration, you could just add new locations:
And if you can do that all abstracted in docsettings.lua (+ may be some adjustments in filemanager.lua for the move/copy/delete a book), it wouldn't (shouldn't) need any changes anywhere else, except for using possible added docsettings.lua API. And may be, if we add support for C, we can get rid of proposing/creating D (although we should still scan/read D) as D is the most ugly system as we can't go back from a setting to the book (and it doesn't handle .old and using .old when the current is corrupted). (Proposing this choice to the user is not really a feature we would need, but the system as it is currently and as it can still be, could make it easy - as we need the various locations for technical reasons.) |
|
I have no strong opinion on this topic, or in most topics :p. But the So 👍 to include it as an option if it is useful for somebody. -1000 to make it default. I guess I would be ok even if it is the new default if there's a way to switch back to current behaviour.
That would be awesome. Even |
That's also my opinion for similar reasons in case I hadn't said so yet. It's basically the clutter without the positive aspects. I think the better form of that would probably look like a .sdr folder that simply has the relevant filename.lua metadata inside for all the files. To be clear, I'm not suggesting we do that. ;-) |
|
I had said this in another issue and was told this was the active discussion on it. I have the same problem/annoyance with this. It's be far better if it used the koreader folder as a base, so if I have a book at say That seems like it'd be the best option to avoid filesystem clutter while keeping code changes to the minimum. |
|
If anyone needs this, I've implemented a solution similar to the one posted above as a patch, thus it should survive an update. I have this in -- move sidecar directories, they mess up the DropBox sync
local DocSettings = require("docsettings")
local DataStorage = require("datastorage")
local lfs = require("libs/libkoreader-lfs")
local function getSidecarDir()
local d = DataStorage:getDataDir() .. "/sidecars"
if lfs.attributes(d, "mode") ~= "directory" then
lfs.mkdir(d)
end
return d
end
local SIDECAR_DIR = getSidecarDir()
function DocSettings:getSidecarDir(doc_path)
if doc_path == nil or doc_path == '' then
return ''
end
local file_without_suffix = doc_path:match("(.*)%.")
if not file_without_suffix then
file_without_suffix = doc_path
end
return SIDECAR_DIR .. "/[" .. file_without_suffix:gsub("(.*/)([^/]+)", "%1] %2"):gsub("/", "#") .. ".sdr"
end(The The reason this is needed is there is a weird bug in PocketBook's DropBox client. Every single time it syncs, it duplicates the metadata.epub.lua and metadata.epub.lua.old files, so after a couple days of reading, my DropBox is full of |
|
Please note the centralized sidecar folder is not supported in copy/move filebrowser operations. |
@poire-z, can you please elaborate on this. I'm afraid I got it wrong. |
|
Well, I don't really remember what I had in mind :/ May be when I wrote this, I had wrongly in mind that it was saved as So, well, that's all I can think of today :/ Either I was confused that day, or I was more clever than I am today :) (I'm also not sure why it would be important to go back from the setting to the book, except when "rebuilding the history", which feels non-essential.) |
|
Good, thanks. |
|
Yes, probably. I'm not sure, but I guess that with C, I was thinking about replicating the whole directory tree under sidecar/, ie: So, dunno which of C or D is easier. But given you would still have to handle D (or migrate them to C), if C does not bring anything new to D, we could go with just D. Also beware with D: on FAT32, the max filename length is 255, so users with deeply organized book trees could meet this limit. (In the past, we had also thought about storing other things in the sidecar directory, like book cover image - even if we never ended up doing anything like that - C would allow it, while D would not.) |
|
Another bad thing with D (doc-settings files in koreader/history) is that removing a book from History kills its history-file (that contains doc-settings). |
|
So, it looks like with #10074, you went with C - and not with D that seemed to have your preference above. |
|
The reasons of chioice are yours: limit of 256 chars in the full name and future impossibility of having more files related to a book in its sdr folder. koreader/frontend/readhistory.lua Lines 265 to 272 in 50b2267 While removing an item from history, its (legacy) history file is deleted. We cannot store anything in it. If we keep it, the book will reappear in History. |
Temporary quasi-solution I am using is color filters (in WinSCP, Total Commander on PC): |
|
I see a lot of mentions of $SIDECAR_DIR as the alternative centralized root directory instead of storing .sdr folders alongside the original files. But on the 2023.04 update, it looks like the new I can make a new FR issue for this, assuming I'm not just missing something. I can't see many downsides to this - perhaps it would be relatively easy to add, given the work is already there for this? |
|
We have no existing settings to relocate the similar-in-design settings and cache folders (and others), so I'm not sure I understand the interest/use-case? |
|
@NiLuJe Thanks, I didn't realize that $SIDECAR_DIR was also used for other things besides storing the .sdr files. I was thinking it was a new directory for storing just the .sdr files and thus could reside anywhere. My use case is being able to place a '.sdr folder only' version of this folder in a place where I can easily sync it using Syncthing so that I can sync reading progress, annotations, and reading statistics (if they're stored in there) between all devices using KOReader. This already seems to work pretty well keeping the .sdr files in their original places (next to their corresponding document files) and I was thinking it wouldn't be too hard to keep them in a local folder instead and just point to that folder in the settings. But if you use that folder for more things too, it's a different story. On the other hand, it would be nice to even sync app settings and other things in this same way (for example, obsidian.md puts ALL app settings in a |
It's not. That wasn't my point. My point was other "user data" folders exist, and they're not relocatable either. My actual question being: what's stopping you from pointing your sync stuff to the existing (or not, since it's opt-in) folder? |
|
Settings are another kettle of fish, as they're not guaranteed to be cross-device compatible (e.g., when something is platform-specific, it's checked in the UI, not in the place the setting is read [mostly]). So, while you can simply two-way sync 'em (again, it's a fixed location, but a stable one), it's probably a terrible idea, especially if you cross a platform-gap (e.g., Android to anything !Android; Android <-> Android would be mostly fine, probably. But still not guaranteed to not blow up in fun and interesting ways). |
|
Re settings sync, I agree, you may want settings to be different for different devices with different properties and use cases. This is somewhat of a negative for Obsidian. VS Code has a neat solution of this with some defaults that can be overridden for specific devices and profiles, but that's far too complicated for something like this or most apps.
That is an option - it's just that this default absolute path changes from OS type (on Linux it's a different root path). I have a consistent folder setup for Syncthing across devices and OSes and would prefer to keep everything under them. I could always try symlinking or rsync, but that sometimes causes random issues. You have a good point, but it also makes me wonder why if the Screenshots folder path can currently be specified, why it wouldn't make sense to allow (and reuse) the same customizability here (maybe I'm missing a key reason). I think there are a lot of unforeseen reasons why people may want path customizability which is why so many apps allow it. |
|
It would feel like gratuitous complexity, and a very fine way of wondering why shit suddenly doesn't work as expected ;). Also, on Android, we have to deal with the sandboxing fs access shenanigans, so the less hoop jumping we have to go through, the better. TL;DR: Unlikely to change, ever. |
|
Unrelated but could I ask how 'stable' the .sdr folder contents are across different versions of KOReader (I presume they are the same across OSes)? I'm thinking about ways to use them as a basic format for storing highlights that can also be written and read in other applications. For example, Zotero offers a built-in PDF reader and stores its own note/highlight annotations in its own database outside of the PDF file - I'm thinking about building a sync translation layer that would look at those annotations and write an equivalent version into the .sdr file Lua data structure so the highlights show up in KOReader. And vice versa when making annotation changes in KOReader so the Zotero PDF highlights database is updated. Yeah, there's a ton of issues with sync conflicts that I'd have to figure out - I'm just wondering if I should invest time at all if the .sdr structure changes so frequently that it would break every other update. Thanks. |
|
Here's a link to KOHighlights if you haven't come across it yet: https://www.mobileread.com/forums/showthread.php?t=265433 Stable depends a bit on what you mean by the word. There are some changes occasionally but on average that's (almost) as annoying for us as it'd be for another app. |
|
The main pain point to deal with it externally is probably going to be that it's actually Lua code. You'll notice KoHighlights uses a Python module that basically massages Lua tables into Python dictionaries ;). I haven't really looked all that much, but that may very well be the only example of that sort of cross-language compatbility. |
|
Maybe KoHighlights has already done a lot of the translation then and it could be a good thing to contribute to or build off of. Thanks. |
Currently, KOReader creates a sidecar folder whenever we open a book in the same directory the book is in. The best thing about the sidecar-based approach is its easy discoverability and portability. On the other hand, It can create a lot of directories and kinda look bad in file explorers.
I'm proposing it to centralize in a way which I believe will be a win-win.
Firstly, there will be a sidecar directory in KOReader's config directory, similar to the clipboard directory. Let's call it
$SIDECAR_DIRfor now.Instead of creating a sidecar in the same directory of the book, it will be created at:
$SIDECAR_DIR/absolute/path/of/the/file.sdr/In this way, sidecars will be easily discoverable, and the directories outside will be clean too.
The text was updated successfully, but these errors were encountered: