New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to import all Evernote notes. #55

Closed
leveled opened this Issue Dec 1, 2017 · 21 comments

Comments

Projects
None yet
4 participants
@leveled
Copy link

leveled commented Dec 1, 2017

Operating system

  • Windows
  • macOS
  • Linux
  • Android
  • iOS

Application

  • Desktop
  • Mobile
  • Terminal

If it is a bug if possible please provide a screenshot showing the problem

evernote-export

evernote-export-2

joplin-import

If there is an error, please copy and paste the full error message


I'm running into an issue while trying to import my Evernote notebook. For whatever reason Joplin seems to be capping out at importing 412 notes and then stopping.

@RupGautam

This comment has been minimized.

Copy link

RupGautam commented Dec 2, 2017

Hi, you can run the below command to allow application to open the dev-tool.
And then you can re-run your imports
echo "--open-dev-tools" > ~/.config/joplin-desktop/flags.txt

And post the logs here.

@laurent22 laurent22 added the bug label Dec 2, 2017

@leveled

This comment has been minimized.

Copy link

leveled commented Dec 2, 2017

Is there a way to export logs from the console? I can't seem to right click -> export in the app. In any case once the import process reached 412 notes the warnings in the screenshot below started appearing.

image

@laurent22

This comment has been minimized.

Copy link
Owner

laurent22 commented Dec 2, 2017

Thanks, I see your file has some tags that aren't currently supported by Joplin, I will add support for them as soon as possible. Yes you can actually export the console content by right-clicking it and selecting "Save as". Could you then please post the content of this log here?

@leveled

This comment has been minimized.

Copy link

leveled commented Dec 2, 2017

@RupGautam

This comment has been minimized.

Copy link

RupGautam commented Dec 2, 2017

Yeap, tags unsupported or you don't have tag for some of the attached files.
Simple hack(might not work) but you might want to re-tag them and try again.

This snippet will give you an idea.
All attached resources needs an "objID" tag

@laurent22

This comment has been minimized.

Copy link
Owner

laurent22 commented Dec 2, 2017

Yes the missing tags are easy to fix but the hash with no associated resource is a bit stranger. It's possible that the note simply references a resource that has been deleted, or maybe the XML is in a weird layout I don't know about.

Actually @leveled, is there any chance you could search for this ID "8588acc02b2142971de08b7f282c9933" in the ENEX file and copy and paste the surrounding code? In particular any en-media tag (if any) and any resource tag (if any)?

@laurent22

This comment has been minimized.

Copy link
Owner

laurent22 commented Dec 2, 2017

Hi @leveled, I've created a new release which should fix the sup and blockquote issue although it's still strange that the import stop as it should have gone past the warnings. If you can give it another try and let me know if it works that would be great. The app is at https://github.com/laurent22/joplin/releases/download/v0.10.33/Joplin-0.10.33.dmg

@leveled

This comment has been minimized.

Copy link

leveled commented Dec 2, 2017

Hey @laurent22, I'm still having issues importing the full set of notes. The sup/blockquote issues are not showing up in the console but the hash issue still pops up.

Log: https://pastebin.com/KQgi8A2u
ENEX Code: https://pastebin.com/SPhFAvcX

@RupGautam

This comment has been minimized.

Copy link

RupGautam commented Dec 2, 2017

Yes, those enex log shows that there are no object id for attached files.
Did you try re-tagging them on Evernote? You can try exporting it after re-tag and run the import again?

@leveled

This comment has been minimized.

Copy link

leveled commented Dec 2, 2017

I deleted the note in question and tried reimporting it. This time there were no warnings/error thrown in the console but it still stopped importing (at 411 notes).

@martinpengellyphillips

This comment has been minimized.

Copy link

martinpengellyphillips commented Dec 2, 2017

For reference, I'm also having this issue on Windows 10 Joplin version 0.10.35. It stops at 20 notes out of 5231 for me. Enex file is 3.24 GB in size.

Console log attached - seems to be 'img' tag unsupported along with a bunch of the "Hash with no associated resource".

joplin_evernote_import_error.log

@laurent22

This comment has been minimized.

Copy link
Owner

laurent22 commented Dec 2, 2017

The img tag can be fixed and maybe the hash tag too (unless it's really referencing a non-existing resource) but It's really weird that the import simply gets stuck instead of throwing an error.

Would any of you by any chance has an enex file or even just a note that can be shared and that shows this issue? In my tests initially I've imported up to 2000 notes but didn't hit this problem.

@martinpengellyphillips

This comment has been minimized.

Copy link

martinpengellyphillips commented Dec 2, 2017

I selected a subset (all hacker news articles) which can be downloaded from https://www.dropbox.com/s/hvofvwx7xd89cuj/joplin_test.enex?dl=0

It stops at 10 out of 318 having created 0. Different error though so might be a different issue.

C:\Users\martin\AppData\Local\Programs\Joplin\resources\app.asar\lib\import-enex-md-gen.js:302 Uncaught Error: Found a <tr> tag outside of a table
    at SAXStream.<anonymous> (C:\Users\martin\AppData\Local\Programs\Joplin\resources\app.asar\lib\import-enex-md-gen.js:302:40)
    at emitOne (events.js:96:13)
    at SAXStream.emit (events.js:191:7)
    at SAXParser.me._parser.(anonymous function) [as onopentag] (C:\Users\martin\AppData\Local\Programs\Joplin\resources\app.asar\node_modules\sax\lib\sax.js:258:17)
    at emit (C:\Users\martin\AppData\Local\Programs\Joplin\resources\app.asar\node_modules\sax\lib\sax.js:624:35)
    at emitNode (C:\Users\martin\AppData\Local\Programs\Joplin\resources\app.asar\node_modules\sax\lib\sax.js:629:5)
    at openTag (C:\Users\martin\AppData\Local\Programs\Joplin\resources\app.asar\node_modules\sax\lib\sax.js:825:5)
    at SAXParser.write (C:\Users\martin\AppData\Local\Programs\Joplin\resources\app.asar\node_modules\sax\lib\sax.js:1278:15)
    at SAXStream.write (C:\Users\martin\AppData\Local\Programs\Joplin\resources\app.asar\node_modules\sax\lib\sax.js:239:18)
    at StringStream.ondata (C:\Users\martin\AppData\Local\Programs\Joplin\resources\app.asar\node_modules\readable-stream\lib\_stream_readable.js:612:20)
@laurent22

This comment has been minimized.

Copy link
Owner

laurent22 commented Dec 2, 2017

Great, thanks for sharing the sample file. I see it might be related to parsing tables, so looks like I either need to fix the logic or make the parser more tolerant to invalid html.

@martinpengellyphillips

This comment has been minimized.

Copy link

martinpengellyphillips commented Dec 2, 2017

make the parser more tolerant to invalid html.

This sounds like it might be needed as I can see another parse error in another batch: joplin_test_2.zip

Uncaught (in promise) Error: Non-whitespace before first tag.
Line: 0
Column: 2
Char: h
    at error (C:\Users\martin\AppData\Local\Programs\Joplin\resources\app.asar\node_modules\sax\lib\sax.js:651)
    at strictFail (C:\Users\martin\AppData\Local\Programs\Joplin\resources\app.asar\node_modules\sax\lib\sax.js:677)
    at beginWhiteSpace (C:\Users\martin\AppData\Local\Programs\Joplin\resources\app.asar\node_modules\sax\lib\sax.js:951)
    at SAXParser.write (C:\Users\martin\AppData\Local\Programs\Joplin\resources\app.asar\node_modules\sax\lib\sax.js:1010)
    at SAXStream.write (C:\Users\martin\AppData\Local\Programs\Joplin\resources\app.asar\node_modules\sax\lib\sax.js:239)
    at StringStream.ondata (C:\Users\martin\AppData\Local\Programs\Joplin\resources\app.asar\node_modules\readable-stream\lib\_…:612)
    at emitOne (events.js:96)
    at StringStream.emit (events.js:191)
    at addChunk (C:\Users\martin\AppData\Local\Programs\Joplin\resources\app.asar\node_modules\readable-stream\lib\_…:284)
    at readableAddChunk (C:\Users\martin\AppData\Local\Programs\Joplin\resources\app.asar\node_modules\readable-stream\lib\_…:271)

Thanks!

@laurent22

This comment has been minimized.

Copy link
Owner

laurent22 commented Dec 7, 2017

Hi, a new version is available now with improved Enex support. In particular:

  • Better handling of notes containing entire web pages (such as when imported via Web Clipper)
  • Support for <img> tags
  • Support for various other tags
  • Improved importing web pages that contain tables within tables. In which case the outer tables (which are usually the website layout) are rendered as regular text block and only the inner tables are actually rendered as tables.
  • Fixed many other minor warnings and errors

I think most errors should be gone now and the resulting Markdown, when rendered to HTML, should be much better. Please give it a try and let me know if you're still having any issue importing notes:

https://github.com/laurent22/joplin/releases/tag/v0.10.37

@martinpengellyphillips

This comment has been minimized.

Copy link

martinpengellyphillips commented Dec 7, 2017

New notebook "evernote_2017_12_02" will be created and file "evernote_2017_12_02.enex" will be imported into it
Found: 5231. Created: 5180. Resources: 14512. Tagged: 1084.
The notes have been imported: Found: 5231. Created: 5180. Resources: 14512. Tagged: 1084.

Worked! Thank you :)

@leveled

This comment has been minimized.

Copy link

leveled commented Dec 8, 2017

Hi @laurent22,

The fix worked for me as well. I noticed whilst importing my Evernote notes there were still a lot of tag and type errors although they did not affect the actual import process itself. Would it be useful information for you to get some of this debug information?

@laurent22

This comment has been minimized.

Copy link
Owner

laurent22 commented Dec 8, 2017

Hi @leveled, good to know it's getting better. Yes please if you could post the warnings and other errors that would be useful.

@leveled

This comment has been minimized.

Copy link

leveled commented Dec 8, 2017

The first section of the log is debug information from when I was deleting my older notebook.

Import Log: https://pastebin.com/8dcuX7BF

@laurent22

This comment has been minimized.

Copy link
Owner

laurent22 commented Dec 8, 2017

Thanks @leveled, the next version will handle these tags that were missing (it won't make a big difference except for "kbd" which will be rendered as code).

For the tr/td in the wrong place, this is probably fine and due to invalid HTML. The text content is still imported even when this warning shows up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment