#204 - Evernote bug bounty #207

akosbalasko · 2024-02-02T18:07:34Z

Evernote improvements and fixes: #204

#190 - Fixed
#200 - Implemented
#26 - Fixed
#201 - Fixed
#24 - Implemented, "@@@" used as delimiter between notebookstack's and notebook's name in the enex file
#187 - Cannot be implemented phisically, possible solution would be to ask the users to flat nested tags by '/' before the conversion within Evernote
#194 - Cannot reproduce, possible reason patched
#184 - Cannot reproduce, reporter asked to close
#185 - Cannot reproduce, no hints from reporter

+1 - Title removed from the converted note's body

kepano · 2024-02-02T19:45:57Z

@akosbalasko thanks! Will review shortly. Two more things:

Can you submit a PR to the obsidian-help repo with additional detail on the Evernote import instructions for handling tags/folders? Those will go up on the Help site once this PR is merged.
Can you reply here stating I have read the CLA agreement and I hereby sign the CLA. per our contribution guidelines?

akosbalasko · 2024-02-02T20:21:21Z

@kepano
Sure, I'll open a PR soon.

I have read the CLA agreement and I hereby sign the CLA.

akosbalasko · 2024-02-02T21:04:40Z

doc PR is here: obsidianmd/obsidian-help#670

src/formats/yarle/utils/filename-utils.ts

src/formats/yarle/utils/templates/default-template.ts

tgrosinger · 2024-02-02T21:26:24Z

src/formats/yarle/utils/templates/checker-functions.ts

@@ -40,7 +41,8 @@ export const hasSourceURLInTemplate = (templateContent: string): boolean => {
 };
 export const hasAnyTagsInTemplate = (templateContent: string): boolean => {
 	return (hasItemInTemplate(TAGS, templateContent)
-		|| hasItemInTemplate(YAMLARRAYTAGS, templateContent));


Should we remove the various bits of code for array tags now? It looks like they are unused?

well, yeah, I was thinking on it, as you wish, I think the whole array tags in yaml together with its templates is currently not used. @lishid any suggestions?

I think removing it would be best, thanks!

@lishid removed

src/formats/yarle/utils/templates/default-template.ts

tgrosinger · 2024-02-02T21:30:33Z

src/formats/yarle/process-resources.ts

@@ -77,9 +77,13 @@ const processResource = (workDir: string, resource: any): any => {
 	// Skip unknown type as we don't know how to handle
 	// Source: https://dev.evernote.com/doc/articles/data_structure.php
 	// "The default type "application/octet-stream" should be used if a more specific type is not known."
-	if (resource.mime === 'application/octet-stream') {
+	// Update: 


@lishid or @ericaxu, does this seem reasonable to you? I'm not too familiar with Yarle.

tgrosinger · 2024-02-02T21:55:07Z

src/formats/yarle/utils/folder-utils.ts

+	fs.mkdirSync(path.join(options.outputDir, ...notebookFolderNames), { recursive: true });
+	enex.fullpath = enex.fullpath.replace(enex.basename, notebookName || enex.basename);
+	enex.name = enex.name.replace(enex.basename, notebookName || enex.basename);
+	options.outputDir = [options.outputDir, ...notebookFolderNames].join(options.pathSeparator);


Will this behave as expected if there are multiple files being imported?

It looks like if we modify the outputDir here it will affect subsequent files being imported.

Nope, because I save the original outputDir at line 187 in yarle.ts and restore after the conversion (line 201).

I believe that's just saving a reference to the object, but that's the same object that will be modified here.

@tgrosinger I think that would be true in the case if an object would have been stored in the variable, but in this case, it is just a copy of a primitive value. Here is an example:

Ah, sorry I misread and thought you were storing the full options object.
Thank you for clarifying for me!

While this is technically correct, it is really a bad design pattern because other parts of the code may not know or understand why the outputDir was changed. It gives the same variable two meanings at different times of the code execution that would be confusing.

In this case it's best if we can return a value and have it passed along to the next functions/

@lishid ok, let me try to improve it.

src/formats/yarle/utils/filename-utils.ts

tgrosinger

Looks good to me, pending the remaining comments for @lishid.

akosbalasko · 2024-02-07T08:12:37Z

Any updates on this @kepano @lishid @ericaxu ?

lishid · 2024-02-07T14:44:37Z

src/formats/yarle/utils/filename-utils.ts

@@ -37,7 +37,10 @@ export const getResourceFileProperties = (workDir: string, resource: any): Resou

 	if (resource['resource-attributes'] && resource['resource-attributes']['file-name']) {
 		const fileNamePrefix = resource['resource-attributes']['file-name'].substr(0, 50);
-		fileName = fileNamePrefix.split('.')[0];
+		const lastIdx = fileNamePrefix.lastIndexOf('.');


I would suggest using the existing filename parser at https://github.com/obsidianmd/obsidian-importer/blob/master/src/filesystem.ts#L210

lishid · 2024-02-07T14:53:48Z

src/formats/yarle/utils/folder-utils.ts

+	fs.mkdirSync(path.join(options.outputDir, ...notebookFolderNames), { recursive: true });
+	enex.fullpath = enex.fullpath.replace(enex.basename, notebookName || enex.basename);
+	enex.name = enex.name.replace(enex.basename, notebookName || enex.basename);
+	options.outputDir = [options.outputDir, ...notebookFolderNames].join(options.pathSeparator);


While this is technically correct, it is really a bad design pattern because other parts of the code may not know or understand why the outputDir was changed. It gives the same variable two meanings at different times of the code execution that would be confusing.

In this case it's best if we can return a value and have it passed along to the next functions/

lishid · 2024-02-07T14:57:47Z

src/formats/yarle/utils/folder-utils.ts

+		notebookName = enex.basename;
+	}
+	fs.mkdirSync(path.join(options.outputDir, ...notebookFolderNames), { recursive: true });
+	enex.fullpath = enex.fullpath.replace(enex.basename, notebookName || enex.basename);


These aren't meant to be modified. PickedFile represents a file picked from the operating system. If we need to work on a different set of names over the original file then we should use a proper data structure to make this clear.

Also, this wouldn't work if the full path contains a piece that is identical to the enex basename. For example, C:/Evernote/Files/Evernote.enex would only get the first "Evernote" replaced instead of the intended second.

These aren't meant to be modified. PickedFile represents a file picked from the operating system. If we need to work on a different set of names over the original file then we should use a proper data structure to make this clear.

@lishid But in this case, it would mean just to clone the current PickedFile, changing its props and use it instead of that we got from outside, or not?

Also, this wouldn't work if the full path contains a piece that is identical to the enex basename. For example, C:/Evernote/Files/Evernote.enex would only get the first "Evernote" replaced instead of the intended second.

Good point I'll resolve it.

@lishid But in this case, it would mean just to clone the current PickedFile, changing its props and use it instead of that we got from outside, or not?

Can you explain the use case here? If I'm understanding it correctly you'd like to specify a different title/output path for the same input file, right? In that case I would suggest passing this additional information over to the code that uses it (through an object or function parameters) to clearly indicate that it's the supposed output path.

@lishid yeah, that would be the use case, and I used to use options object would to store these kind of info. Now I cleaned a code a bit around notebook stacks, hope it looks better for you too.

lishid · 2024-02-07T15:00:55Z

src/formats/yarle/utils/templates/checker-functions.ts

@@ -40,7 +41,8 @@ export const hasSourceURLInTemplate = (templateContent: string): boolean => {
 };
 export const hasAnyTagsInTemplate = (templateContent: string): boolean => {
 	return (hasItemInTemplate(TAGS, templateContent)
-		|| hasItemInTemplate(YAMLARRAYTAGS, templateContent));


I think removing it would be best, thanks!

lishid · 2024-02-07T15:02:28Z

Sorry I am late!

lishid · 2024-02-07T21:28:28Z

src/formats/yarle/utils/folder-utils.ts

+	const { notebookName } =  getNotebookNameAndFolderNames(baseEnex.basename);
+
+	const stackedPickedFile = new NodePickedFile((baseEnex as NodePickedFile).filepath);
+	stackedPickedFile.fullpath = replaceLastOccurrenceInString(baseEnex.fullpath, baseEnex.basename, notebookName || baseEnex.basename);


Sorry I didn't make this clear; we should not be modifying internal data structures of NodePickedFile. I have made an update to the API on master, if you can rebase you will understand what I meant.

@lishid Ohh, okay, now I get it, thanks for the clarification! Changes made to make it work with untouchable NodePickedFiles. :)

As I understand it, what's happening here is that we are finding some code down the line that uses these fields, and we'd like to change their behavior to use a different names/paths. Doing it this way is a "shortcut"; while it results in minimal amount of changes, it also makes the code much more confusing. Similar to what I said in the other comment, when you have a field that is defined to mean one thing, and you use it for a slightly different purpose (and thus modify the contents to fit your alternative needs), it creates a chance of confusion, or at the very least, forces the reader to "keep in mind" these extra concerns. This is often where bugs can slip in, especially if there are many different little things one needed to keep in mind when working with the code.

Usually the best way to do this is being more explicit with the code to say what you intend with the variables and fields. Passing additional vars with names that indicate what they do (or is intended for) helps a lot.

Yup the new changes works for me.

@lishid , sorry, I had a bad feeling about this, so I tried to improve the code further, introduced a couple of new functions to decrease the level of the possible confusions. I hope you agree, but if not, feel free to ask to revert.

Thanks. I would say don't worry too much about it. Yarle is unfortunately written in a very different paradigm of programming than the rest of the importer. It uses global singleton objects and naked functions with context being passed sometimes, set in the global object at other times.

I decided not to invest too much effort to rewrite yarle as it would not be worth the time for a functionally equivalent package.

@lishid Okay, I see. Thanks for the review! Then we can move forward with it, right?

lishid

Code looks ok to me. Might be worth doing one final pass of the auto formatter.

tgrosinger · 2024-02-09T19:22:09Z

Looks good to me.

akosbalasko added 7 commits January 29, 2024 21:15

obsidianmd#190 - multiple dots in filenames

45e93a6

obsidianmd#200 - source url in properties

c70d86d

obsidianmd#26 correct tag list in frontmatter

df2cda6

title removed as H1

7c1e0fd

obsidianmd#194 patched

8cd12ef

obsidianmd#201 fixed

bb21863

obsidianmd#24 notebookstacks supported

b7b3ae0

kepano requested review from tgrosinger and lishid February 2, 2024 19:38

tgrosinger requested changes Feb 2, 2024

View reviewed changes

filename calculation simplified

cf90e45

akosbalasko commented Feb 2, 2024

View reviewed changes

src/formats/yarle/utils/filename-utils.ts Outdated Show resolved Hide resolved

akosbalasko requested a review from tgrosinger February 5, 2024 14:52

tgrosinger approved these changes Feb 5, 2024

View reviewed changes

lishid reviewed Feb 7, 2024

View reviewed changes

akosbalasko added 4 commits February 7, 2024 16:46

tags-array-placeholders and its functions removed

9125727

using the existing filename parser

3e6fe4c

tags array fix

775b58b

refactor around notebookstacks

6ad7ba1

akosbalasko requested a review from lishid February 7, 2024 20:32

lishid reviewed Feb 7, 2024

View reviewed changes

akosbalasko added 2 commits February 7, 2024 22:45

Merge branch 'obsidianmd:master' into evernote-bounty-204

200b0a4

NodePickedFiles untouched

15d3145

lishid approved these changes Feb 7, 2024

View reviewed changes

further improvements around notebookstacks

7ecf43d

akosbalasko requested a review from lishid February 7, 2024 23:25

lishid merged commit 7dd3152 into obsidianmd:master Feb 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#204 - Evernote bug bounty #207

#204 - Evernote bug bounty #207

akosbalasko commented Feb 2, 2024 •

edited

Loading

kepano commented Feb 2, 2024 •

edited

Loading

akosbalasko commented Feb 2, 2024

akosbalasko commented Feb 2, 2024 •

edited

Loading

tgrosinger Feb 2, 2024

akosbalasko Feb 2, 2024 •

edited

Loading

lishid Feb 7, 2024

akosbalasko Feb 7, 2024

tgrosinger Feb 2, 2024

tgrosinger Feb 2, 2024

akosbalasko Feb 2, 2024

tgrosinger Feb 5, 2024

akosbalasko Feb 5, 2024 •

edited

Loading

tgrosinger Feb 5, 2024

lishid Feb 7, 2024

akosbalasko Feb 7, 2024

tgrosinger left a comment

akosbalasko commented Feb 7, 2024

lishid Feb 7, 2024

lishid Feb 7, 2024

lishid Feb 7, 2024

akosbalasko Feb 7, 2024

lishid Feb 7, 2024

akosbalasko Feb 7, 2024

lishid Feb 7, 2024

lishid commented Feb 7, 2024

lishid Feb 7, 2024

akosbalasko Feb 7, 2024

lishid Feb 7, 2024

lishid Feb 7, 2024

akosbalasko Feb 7, 2024 •

edited

Loading

lishid Feb 7, 2024

akosbalasko Feb 8, 2024 •

edited

Loading

lishid left a comment

tgrosinger commented Feb 9, 2024

#204 - Evernote bug bounty #207

#204 - Evernote bug bounty #207

Conversation

akosbalasko commented Feb 2, 2024 • edited Loading

kepano commented Feb 2, 2024 • edited Loading

akosbalasko commented Feb 2, 2024

akosbalasko commented Feb 2, 2024 • edited Loading

Choose a reason for hiding this comment

akosbalasko Feb 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akosbalasko Feb 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tgrosinger left a comment

Choose a reason for hiding this comment

akosbalasko commented Feb 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lishid commented Feb 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akosbalasko Feb 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akosbalasko Feb 8, 2024 • edited Loading

Choose a reason for hiding this comment

lishid left a comment

Choose a reason for hiding this comment

tgrosinger commented Feb 9, 2024

akosbalasko commented Feb 2, 2024 •

edited

Loading

kepano commented Feb 2, 2024 •

edited

Loading

akosbalasko commented Feb 2, 2024 •

edited

Loading

akosbalasko Feb 2, 2024 •

edited

Loading

akosbalasko Feb 5, 2024 •

edited

Loading

akosbalasko Feb 7, 2024 •

edited

Loading

akosbalasko Feb 8, 2024 •

edited

Loading