Fix: Evernote .enex image import #7785
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
|
TODO: Import 390M .enex on HTML wiki is smooth, but on NodeJS it quickly become 100% usage on server side, I need to figure out what is consuming CPU later. If open in chrome, it will cause chrome 100%, so it is actually a frontend issue, maybe the import table is too long? But this only happen on my wiki, can't reproduce on Official wiki. But this PR is OK to merge. |
There was a problem hiding this comment.
Hi @linonetwo this looks good, apart from one minor issue. The use of a random number in the title seems clumsy. I'm not sure what options there might be, but a simple one would be if we could use a meaningful string (perhaps the truncated and possibly slugified text field), with a hash of the text field as a disambiguator.
I forgot I can use the hash provided by Evernote. |
It's probably a Node.js file IO problem that causes problems. I did create 20000+ tiddlers from the debug console with a node-server active and it caused problems after about 10-20 seconds on the filesystem side. The tiddlers itself where created immediately. May be there is a problem in the node.js server backend if many files are written at the same time. may be memory leak or something similar. |
|
I don't have more things to add to this PR. I've done using it importing all my Evernote content. Now I have 26k tiddlers, with size 2GB+ (including .git backup, hope tiddlywiki nodejs won't look into .git ...) |
No, I find it's smooth with or without It is just on my wiki. Maybe is cause by some plugin, but I don't have time to bi-sect it by hand. ENEX import can be a good reproducible example for this performance issue. I will look into it one day maybe. |
|
Thanks @linonetwo |
fixes #3176
Tested with 200M+ .enex file full of website clipped by Evernote WebClipper...All images in website are preserved.
A .enex that has wikihow clipped for test:
wikihow.enex.zip
Only thing to note is that you need to use vzhd1701/evernote-backup#54 to export .enex file, otherwise .enex that resources don't have hash can't be processed properly.