Error when importing notebook with sub-notebooks #597

pynting · 2018-06-08T21:59:54Z

Operating system

Linux

Application

Desktop
Terminal

Hi, I am experiencing a really troublesome bug when syncing to filesystem in 1.0.69.
I am trying to sync between my laptop and android phone using syncthing. I first experienced this when I was doing my first tests with joplin, but after the initial difficulties it started working and I just presumed I had done something wrong with the setup initially. So I moved all my notes over to Joplin from my previous system (a bunch of txt files in different folders), convinced Joplin was the way to go. It worked and I got my notes synced on my phone as well.
A couple of hours ago I was doing some changes to my notes, and when I hit sync, ALL my notes disappeared! I quickly went offline with my phone and managed to save it from being auto synced. I got an adb backup of the database from my phone and pasted it to my .config/joplin-desktop folder and changed the name. Got my notes back in the correct folder and sub folder structure (but the colours of my Joplin install are all inverted).

I can replicate the removal of all my notes by setting up a new sync folder in settings, press sync, then change sync folder and hit sync again.

I am trying to export/import the notes to start over fresh with a new database file, but the export or import functions does not keep all my notes (they are all plain text, nothing fancy), and the notebook/sub-notebook structure is gone, also some of the notebooks go missing.
The terminal app simply crashes and stays broken after trying to import a jex file. I have to delete the .config/joplin folder (probably just the database, though) to get it to run again.

This is my first post on GitHub, so pardon if I am not doing things right... Should I post new issue tickers for every bug/issue I encounter?

this is the log from the desktop app:
log.txt

this is a printout from terminal after attempting to import jex file:
cli-output.txt

laurent22 · 2018-06-09T09:58:55Z

I can replicate the removal of all my notes by setting up a new sync folder in settings, press sync, then change sync folder and hit sync again.

That's most likely the issue. If you change the sync folder without copying the notes files to it (eg. all the .md files and the .resources dir), Joplin does what's logical: since there's no data in the new folder, it assumes that everything has been deleted and proceeds to also delete everything locally.

So when you change the folder, make sure to also move all the files in it.

If it's a different issues though, please describe steps by steps how you got this bug.

This is my first post on GitHub, so pardon if I am not doing things right... Should I post new issue tickers for every bug/issue I encounter?

Here it's fine if it's a bug, otherwise for general user support it might be better to post on the forum https://discourse.joplin.cozic.net (you can login with your GitHub account)

pynting · 2018-06-09T18:03:00Z

Hmm, I don't think i did any changes to the sync folder when it happened... I was mocking about after it happened to try and find what was going on and that was when I found that by changing the sync folder it deleted my notes. I will try to find a way to replicate it somehow.

How Joplin treats an empty sync folder sounds really dangerous to me. The mechanics of detecting if a note is really deleted or not should really be stated explicitly and not assumed by an empty sync folder! It is a very fragile system, the way it works now it sounds like.

Also, it should be a waste bin in the database, so that deleted notes can be recovered if deleted by mistake, either by the system or user. It could perhaps have some sort of janitor routine that removes deleted notes after x days.

Another problem i described was the exporting/importing of a database not managing to pass over sub folders. I have attached two images, to illustrate before and after. The same goes for both JEX and RAW export.

Thanks for your prompt responses. I am committed to using Joplin now, and am exited to see it develop further!

laurent22 · 2018-06-09T18:17:37Z

How Joplin treats an empty sync folder sounds really dangerous to me.

The behaviour is correct actually, for example if you sync with an empty dir with rsync along with the --delete flag it will also delete all your files, this is what sync does.

I agree however that it might be unexpected and, as a general rule, the filesystem sync is more for advanced users (originally it was just a tool for testing/debugging but since some people found it useful I've made it available) so it needs to be used with care.

Also, it should be a waste bin in the database, so that deleted notes can be recovered if deleted by mistake, either by the system or user. It could perhaps have some sort of janitor routine that removes deleted notes after x days.

Yes that's been suggested a few times, I think there's an open issue about it. In the meantime, it's recommended to keep a backup of your notes. You can automate this with a cron and the joplin CLI. eg joplin export --format raw /path/to/backup

pynting · 2018-06-09T19:50:46Z

Ok, thanks for the clarification.
In the future one might think to make it more robust against people or software making changes in their file structure. I will keep a backup from now on.
Only problem is that it does not export everything, like in the images I attached in the last post, a lot of my notes that was residing in sub-notebooks does not get exported or imported, not sure where the problem resides...

pynting · 2018-06-09T20:02:30Z

btw: I wrote a small shell script for joplin CLI to speed up note taking in terminal, not sure if it is something people would appreciate or not, anyway, here is the link
I am hesitant to post it in your forum as it seems a bit off topic perhaps...

laurent22 · 2018-06-09T20:03:50Z

It looks like there's indeed an issue when exporting notebook with sub-notebooks. I've renamed the issue and tagged as bug to keep track of it.

laurent22 · 2018-06-09T20:07:21Z

btw: I wrote a small shell script for joplin CLI to speed up note taking in terminal, not sure if it is something people would appreciate or not, anyway, here is the link

Seems nice! If the user doesn't specify a notebook, to which notebook will the note go? Maybe something to mention in the help page.

Posting it on the forum is fine. @foxmask has also been developing a tool called Jong to create notes from RSS feeds.

pynting · 2018-06-09T20:17:31Z

Thanks!
Good catch, I copy/pasted from the scripts help text, and forgot to update the variable name with its default value in the readme. Fixed now

pynting · 2018-06-14T21:16:58Z

Hey Laurent, don't forget to update the CLI app!
I have tried to build from source as well as normal install method, it still crashes when trying to import from raw format with sub-notebooks (desktop application works perfectly though!).
Both cloned from github and built, and via normal install method says version 1.0.107 (dev and prod respectively)

Unhandled promise rejection Promise {
  <rejected> TypeError: Cannot read property 'parent_id' of null
    at FolderListWidget.folderDepth (/home/username/.joplin-bin/lib/node_modules/joplin/gui/FolderListWidget.js:46:16)
    at FolderListWidget.itemRenderer [as itemRenderer_] (/home/username/.joplin-bin/lib/node_modules/joplin/gui/FolderListWidget.js:28:33)
    at FolderListWidget.render (/home/username/.joplin-bin/lib/node_modules/joplin/node_modules/tkwidgets/ListWidget.js:285:46)
    at FolderListWidget.render (/home/username/.joplin-bin/lib/node_modules/joplin/gui/FolderListWidget.js:152:9)
    at Renderer.renderWidget (/home/username/.joplin-bin/lib/node_modules/joplin/node_modules/tkwidgets/framework/Renderer.js:53:12)
    at Renderer.renderWidget (/home/username/.joplin-bin/lib/node_modules/joplin/node_modules/tkwidgets/framework/Renderer.js:74:15)
    at Renderer.renderWidget (/home/username/.joplin-bin/lib/node_modules/joplin/node_modules/tkwidgets/framework/Renderer.js:74:15)
    at Renderer.renderWidget (/home/username/.joplin-bin/lib/node_modules/joplin/node_modules/tkwidgets/framework/Renderer.js:74:15)
    at Renderer.renderWidget (/home/username/.joplin-bin/lib/node_modules/joplin/node_modules/tkwidgets/framework/Renderer.js:74:15)
    at Renderer.renderRoot (/home/username/.joplin-bin/lib/node_modules/joplin/node_modules/tkwidgets/framework/Renderer.js:84:14) } reason: TypeError: Cannot read property 'parent_id' of null
    at FolderListWidget.folderDepth (/home/username/.joplin-bin/lib/node_modules/joplin/gui/FolderListWidget.js:46:16)
    at FolderListWidget.itemRenderer [as itemRenderer_] (/home/username/.joplin-bin/lib/node_modules/joplin/gui/FolderListWidget.js:28:33)
    at FolderListWidget.render (/home/username/.joplin-bin/lib/node_modules/joplin/node_modules/tkwidgets/ListWidget.js:285:46)
    at FolderListWidget.render (/home/username/.joplin-bin/lib/node_modules/joplin/gui/FolderListWidget.js:152:9)
    at Renderer.renderWidget (/home/username/.joplin-bin/lib/node_modules/joplin/node_modules/tkwidgets/framework/Renderer.js:53:12)
    at Renderer.renderWidget (/home/username/.joplin-bin/lib/node_modules/joplin/node_modules/tkwidgets/framework/Renderer.js:74:15)
    at Renderer.renderWidget (/home/username/.joplin-bin/lib/node_modules/joplin/node_modules/tkwidgets/framework/Renderer.js:74:15)
    at Renderer.renderWidget (/home/username/.joplin-bin/lib/node_modules/joplin/node_modules/tkwidgets/framework/Renderer.js:74:15)
    at Renderer.renderWidget (/home/username/.joplin-bin/lib/node_modules/joplin/node_modules/tkwidgets/framework/Renderer.js:74:15)
    at Renderer.renderRoot (/home/username/.joplin-bin/lib/node_modules/joplin/node_modules/tkwidgets/framework/Renderer.js:84:14)

laurent22 · 2018-06-14T21:50:10Z

Indeed I completely forgot to update and just noticed that my backup script has been broken for a month because of this. That should now be fixed so please give it another try .

pynting · 2018-06-15T09:58:26Z

Hi, works now when importing my "test notebooks", but when trying to import my actual notes I get:

  <rejected> TypeError: Cannot read property 'parent_id' of null
    at FolderListWidget.folderDepth (/home/username/Apps/joplin/CliClient/build/gui/FolderListWidget.js:46:16)
    at FolderListWidget.itemRenderer [as itemRenderer_] (/home/username/Apps/joplin/CliClient/build/gui/FolderListWidget.js:28:33)
    at FolderListWidget.render (/home/username/Apps/joplin/CliClient/node_modules/tkwidgets/ListWidget.js:285:46)
    at FolderListWidget.render (/home/username/Apps/joplin/CliClient/build/gui/FolderListWidget.js:152:9)
    at Renderer.renderWidget (/home/username/Apps/joplin/CliClient/node_modules/tkwidgets/framework/Renderer.js:53:12)
    at Renderer.renderWidget (/home/username/Apps/joplin/CliClient/node_modules/tkwidgets/framework/Renderer.js:74:15)
    at Renderer.renderWidget (/home/username/Apps/joplin/CliClient/node_modules/tkwidgets/framework/Renderer.js:74:15)
    at Renderer.renderWidget (/home/username/Apps/joplin/CliClient/node_modules/tkwidgets/framework/Renderer.js:74:15)
    at Renderer.renderWidget (/home/username/Apps/joplin/CliClient/node_modules/tkwidgets/framework/Renderer.js:74:15)
    at Renderer.renderRoot (/home/username/Apps/joplin/CliClient/node_modules/tkwidgets/framework/Renderer.js:84:14) } reason: TypeError: Cannot read property 'parent_id' of null
    at FolderListWidget.folderDepth (/home/username/Apps/joplin/CliClient/build/gui/FolderListWidget.js:46:16)
    at FolderListWidget.itemRenderer [as itemRenderer_] (/home/username/Apps/joplin/CliClient/build/gui/FolderListWidget.js:28:33)
    at FolderListWidget.render (/home/username/Apps/joplin/CliClient/node_modules/tkwidgets/ListWidget.js:285:46)
    at FolderListWidget.render (/home/username/Apps/joplin/CliClient/build/gui/FolderListWidget.js:152:9)
    at Renderer.renderWidget (/home/username/Apps/joplin/CliClient/node_modules/tkwidgets/framework/Renderer.js:53:12)
    at Renderer.renderWidget (/home/username/Apps/joplin/CliClient/node_modules/tkwidgets/framework/Renderer.js:74:15)
    at Renderer.renderWidget (/home/username/Apps/joplin/CliClient/node_modules/tkwidgets/framework/Renderer.js:74:15)
    at Renderer.renderWidget (/home/username/Apps/joplin/CliClient/node_modules/tkwidgets/framework/Renderer.js:74:15)
    at Renderer.renderWidget (/home/username/Apps/joplin/CliClient/node_modules/tkwidgets/framework/Renderer.js:74:15)
    at Renderer.renderRoot (/home/username/Apps/joplin/CliClient/node_modules/tkwidgets/framework/Renderer.js:84:14)

laurent22 · 2018-06-17T16:07:07Z

Do you have a JEX file that allows replicating this issue?

pynting · 2018-06-23T00:01:14Z

Hi Laurent, I did another update of both the desktop and cli programs (I think I might have been to early on the ball after your last update when updating through "npm install"), and now it actually imports the notebooks without crashing (hurrah! ), however, the sub-folders have swapped places in some instances... I will post a file in not to many days. If it is the database that is corrupt, it seems weird to me that it is not "washed clean" after doing an export/import in the desktop app, then exporting and trying to import in the cli app...

Another thing regarding the reason for starting this thread in the first place: When I updated to the lates appimage and ran it, the database came up just fine, but then, when it synced with my local sync folder, all the notes disappeared. The files were in the sync folder still (and due to the testing I am doing, I always back up my work anyway, so I had backups) I ran an "import raw folder" from the sync folder, and my notes all came back in their right places. Is this not weird behavior? The two releases was 1.0.99 and 1.0.103.

pynting · 2018-07-07T22:48:42Z

Sorry that it took so long to get to it, had to try to remove my personal stuff from the (growing) database as well. After replacing almost all the entries with numbers and letters one can also more easily see that there is something fishy going on. Attached the jex file exported from latest joplin appimage, imported in latest cli build (installed with npm). Hope it makes sense to you what might be happening

anonExportTest.zip

laurent22 · 2018-09-21T18:41:42Z

Closed as duplicate

laurent22 added the bug It's a bug label Jun 9, 2018

laurent22 changed the title ~~Sync to filesystem deletes my whole database!~~ When exporting folders, sub-folders are not exported Jun 9, 2018

laurent22 closed this as completed in ab96755 Jun 10, 2018

laurent22 reopened this Jun 15, 2018

laurent22 changed the title ~~When exporting folders, sub-folders are not exported~~ Error when importing notebook with sub-notebooks Jun 15, 2018

laurent22 closed this as completed Sep 21, 2018

lock bot locked and limited conversation to collaborators Oct 16, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error when importing notebook with sub-notebooks #597

Error when importing notebook with sub-notebooks #597

pynting commented Jun 8, 2018 •

edited

laurent22 commented Jun 9, 2018 •

edited

pynting commented Jun 9, 2018

laurent22 commented Jun 9, 2018

pynting commented Jun 9, 2018

pynting commented Jun 9, 2018

laurent22 commented Jun 9, 2018

laurent22 commented Jun 9, 2018

pynting commented Jun 9, 2018

pynting commented Jun 14, 2018

laurent22 commented Jun 14, 2018

pynting commented Jun 15, 2018

laurent22 commented Jun 17, 2018

pynting commented Jun 23, 2018

pynting commented Jul 7, 2018

laurent22 commented Sep 21, 2018

Error when importing notebook with sub-notebooks #597

Error when importing notebook with sub-notebooks #597

Comments

pynting commented Jun 8, 2018 • edited

Operating system

Application

laurent22 commented Jun 9, 2018 • edited

pynting commented Jun 9, 2018

laurent22 commented Jun 9, 2018

pynting commented Jun 9, 2018

pynting commented Jun 9, 2018

laurent22 commented Jun 9, 2018

laurent22 commented Jun 9, 2018

pynting commented Jun 9, 2018

pynting commented Jun 14, 2018

laurent22 commented Jun 14, 2018

pynting commented Jun 15, 2018

laurent22 commented Jun 17, 2018

pynting commented Jun 23, 2018

pynting commented Jul 7, 2018

laurent22 commented Sep 21, 2018

pynting commented Jun 8, 2018 •

edited

laurent22 commented Jun 9, 2018 •

edited