Invalid server connection and replication notifications

[gitmaker032]

Thank you for the wonderful plugin! But there is one very inconvenient point. Notifications about the status of connection/replication. I use own DB server. In the plugin settings, you can enter ANY valid IP and it does not matter if CouchDB is installed, whether the authorization data are correct, the plugin always notifies - "CONNECTED TO IP". Moreover, synchronization is always supposedly successful. Replication always ends successfully. In reality, NOTHING happens with the remote storage, the data is not sent to the server correctly. Please make correct informative notifications about connection/actions status. For example: cannot connect to CouchDB, wrong login/password for authorization, cannot create DB (if not enough rights), etc. If there is no direct access to the server, it is very helpful to understand why it does not work.

Also missing is a tooltip for icons in the status bar (zzz, stop icon, etc). To better understand what exactly is going on.

[vrtmrz]

Thank you for asking. Did you tried 'Test' button in the setting dialog?
If the plugin couldn't connect, plugin should say 'Could not connected to host'
Or perhaps, did you set the remote uri without schema? (i.e. https://) It's needed.
In next update, I'll improve checking method.
Please check it again.

Tooltip icon in status bar also be improved!

[gitmaker032]

Did you tried 'Test' button in the setting dialog?

Yes, of course. And I've already dealt with this issue. It might be worth adding an input mask to the field and refining the notifications text to make it clear that there's a problem with the database connection or replication.

Right now you can enter any IP (no http, yes) and run replication/drop history/reset remote storage. And the plugin will tell you that everything is ok, the remote storage is updated/created, but it's not, of course, and it's confusing.

I figured out how to connect to my database, but I have another problem. I get an error every time I replicate with no details ("Verbose log" is enabled):

I've tried dropping the history or re-initializing the DB.
On the CouchDB server I see the database in which only one document "obsydian_livesync_version" appears and nothing else. There are no errors in the CouchDB logs.

When I execute "Drop history" another error appears, already with details:

I am using my own Ubuntu server, in the authorization data I enter the admin/password pair which is set when installing CouchDB. What else am I missing or is this some kind of bug?

P.S: In the sync log window I can not select and copy text, also the window cannot be resized, so I put a few screenshots :)

[vrtmrz]

Thank you for trying hard. I'm happy to heard 'connected'.
But "Invalid UTF8-JSON" is little mysterious.
'obsydian_livesync_version' has been put, so your credentials must be ok.
I attach my CouchDB's configurations(It's v3.1.1 on Ubuntu ). please check set settings?
config_sample.json.txt

And you right, sync log should be selectable. I'll fix it too.

[gitmaker032]

Thank you for your help. However, the clue turned out to be quite... unusual and funny :)

Replication ends with an error if you rename the vault directory to test (exactly with a lowercase letter) or create a new one with that name. And my test vault was called exactly that! Any other directory (vault) name works fine and does not cause errors.

On your test server (https://olstaste.vrtmrz.net/), I have not been able to test this behavior. Unable to connect with the authentication data I get on the site.

Synchronization now works... But only on the desktop version 🙄. On android I can't connect to the server. Test connection ends with an error ("could not connect to..."). Of course, the authorization data is exactly the same as on the desktop. Perhaps it is related to the way the android version of the plugin handles the DB remote URI? I have it in the following format: http://0.0.0.0:5984/bdname (I use IP and port instead of domain name).

[vrtmrz]

Wow, vault name.. I'll test after.
And sorry for test server too.

and, I'm relived to answer this, Android or iOS may couldn't connect to non-secure URI.
Could you test ngrok like solution, or may be install self-signed certificate to your device and to your CouchDB.
docs are here

[gitmaker032]

I already have a self-signed certificate configured and I tried to enter the address with https. The result is the same, unfortunately 🥲 On the desktop everything is fine in both cases.

[vrtmrz]

Oh my.. I have the Rpi in the local. I'll try to reproduce, and search what is behind!

[vrtmrz]

I'm very sad, Reproduced.
It looks like Obsidian's platform(capacitorjs) doesn't trust for local CA. (I didn't check so deeply yet)

Do you have any domain? If you have one and with some technics, You can use Let's encrypt's certificate to local server(with local dns)

[gitmaker032]

Do you have any domain? If you have one and with some technics, You can use Let's encrypt's certificate to local server(with >local dns)

Done. I set up a letscrypt certificate for the domain, https works fine. It didn't make any difference on the Android. The error is the same: "Сouldn't connect".

[vrtmrz]

Thank you for your try. but I'm really sorry for this.
I'll check it again. It's the really matter.

[vrtmrz]

I've fixed most of what we're talking about, but Local CA is still there.
I asked in forum and discord, but I don't get the answer yet.
keep in touch, but could you check other issues?

[gitmaker032]

Thanks for the plugin update!

I also had an update of Obsidian itself yesterday (v 1.0.5). And after that on Android I have no problem connecting to the database! Finally. Still in the process of active testing (v 0.1.11). In general, so far the synchronization is not too stable and reliable. Quite often in the process errors of various kinds fly out (I will try to describe them later), which including spoil the structure of the storage. It is easy to catch them if you actively make changes on both devices and loading files (pictures, audio-records). At some point something breaks. Several times I had to use "Drop History". Sometimes I just have to close Obsidian, because replication hangs, it's impossible to start the next one ("Another replication in progress"). Also often errors occur when merging data from two devices.

A few ideas on how to improve the plugin:

1. Add a description to the settings of the "Garbage collection" function. What it does and when to run it. By the way, the "Corrupted Data" section is always empty too, there is only a header and it is unclear what to expect there.
2. Settings of automatic replication launching after a certain configurable interval (1 min - 1 hour). For example, function "real time sync" can consume a lot of traffic and not everyone needs "real time". It is not very convenient to run replication every time manually or specially edit some file.

[vrtmrz]

Thank you for testing and giving me ideas.
Your investigation is really awesome and makes the point! The plugin had performance issues and related some more to large or non-text files. Now, it should become better.

And replication also had a few problems with past rough improvements. I made it simpler (But still be complicated..).
Also, I implemented new features, "Periodic Sync" and "Sync on File Open". It should be useful that in a limited bandwidth.

I added a note into the "Corrupted data" section. but forgot about "Garbage collection". I'll fix it next version up.

FYI:
When the note was edited. the note is split into some chunks and the differential with a header is just synced. Old chunks are left there.
So, the plugin should delete old unused chunks.
But, the chunk is represented as the hash of their contents, sometimes old chunks would be reused. We don't have to haste.

[gitmaker032]

Thanks for the frequent updates to the plugin! The encryption feature is especially useful for me.
I think synchronization is much more stable now. However, I still encounter some strange things sometimes.

Here is one example (happens quite often on plugin version 0.1.16):
I made edits to the notes on my computer. Replication completed correctly.
Opened Obsidian on my Android. Replication causes conflicts for every file previously edited on the PC!
There can be dozens such files. I have to solve the conflict in each case. In fact, there should be no conflicts at all.

To clarify: after each plugin update I do a full history/DB drop to avoid bugs. And I use encryption.

Feature ideas:

1. Default setting for file version conflicts. With options: "Always select newer version of file" and "Always merge files".
2. A button to pause all replications in the Obsidian bottom bar. Sometimes you need to pause any replication for some time (e.g. during active files editing) without editing plugin settings. You can restore replication according to the settings using the same button or by starting replication manually.

But, the chunk is represented as the hash of their contents, sometimes old chunks would be reused. We don't have to haste.

Thanks for the clarification. However, this mechanism causes the remote database to become incredibly bloated in size. Those files that take up 10mb locally in the database take up 45mb (especially if you edit files frequently or activate the live editing feature). And there is no way to clean these deleted documents (only manual database deletion helps). So far I had to use only manual replication to avoid clogging the database with deleted docs.

[vrtmrz]

Thank you too! I really appreciate it to you.
I'm glad to hear that it works for you.
And, I'm relieved to hear that it'd to get better even if a little steadily.

May I ask what's the conflict? newlines?
Auto resolving is a very good idea.

And the growing database is normal. All docs are preserved with their old 100 revisions. so if decrease this value.
Compaction sometimes affects well, But compactions are also don't purge deleted docs. It makes deleted docs to "tombstone" that contains only key.
I test that an idea that flashed in head.
Anyway, the database growing slowly, The best way is Drop history.
If remote dabase's size is the matter, Try

curl -H "Content-Type: application/json" -X POST http://localhost:5984/my_db/_compact

I'll check this later, and it works effectively and if works fine, it wil be implemented

Replication canel buttons is reasonable.
but there're many bottons, Maybe I implement as ToggleSync Options(Ctrl+p)

[vrtmrz]

@gitmaker032

Default setting for file version conflicts. With options: "Always select newer version of file" and "Always merge files".

It is almost implemented! wait for testing.

A button to pause all replications in the Obsidian bottom bar. Sometimes you need to pause any replication for some time (e.g. during active files editing) without editing plugin settings. You can restore replication according to the settings using the same button or by starting replication manually.

I implemented this as Command (Toggle All Sync), please try it.
And also implemented "Batch database update" that completely delays reflecting file changes to the database till it's needed.
It may work well for saving expanding the database.

[vrtmrz]

@gitmaker032
Hi! I released new configurations: "Use newer file if conflicted"
Could you try this?