New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parse Tags from the Note itself #530

Closed
cspann opened this Issue Mar 2, 2017 · 129 comments

Comments

Projects
None yet
6 participants
@cspann

cspann commented Mar 2, 2017

Expected

Wouldn't it be better to indicate Tags by e.g "@tag" in the notes itself to make notes really self contained. Otherwise all tags will be lost when migrating to another Tool.

Actual behaviour

Tags ar not contained in the documents but in a separate sqlite db

Steps to reproduce

none required

Output from the debug settings

Relevant log output (you have to enable enable the log widget)

@pbek

This comment has been minimized.

Owner

pbek commented Mar 2, 2017

Thank you for your suggestion, but this is about the 4th or 5th time such a request was made and every time an other "format" to store tags was suggested.

Problems are:

  • there is no standard to include tags in a markdown file
    • how to handle nested tags?
  • it would require that QON parses and modifies notes by itself when a tag is added to a note (which isn't a very good idea in my opinion and might lead to a lot of troubles)
  • it needs to work for several 1000 notes in multiple note subfolders
  • the note needs to be reparsed every time the user types something
@cspann

This comment has been minimized.

cspann commented Mar 2, 2017

there is no standard to include tags in a markdown file

Good point, but like gitlab/github where they have enhanced markdown I see no special problem here.

    how to handle nested tags?

What are nested tags?

it would require to that QON parses and modifies notes by itself when a tag is added to a note (which isn't a very good idea in my opinion and might lead to a lot of troubles)

I wouldn't handle it that way. To add a Tag the user has to write "@tag" somewhere in the document and you then index the file for this tag in your database

it needs to work for several 1000 notes in multiple note subfolders

Parsing is done only on save or when you import a new set of notes. The problem is not that you store the notes in a separate database, but that you store them only there.

Does this make my request more clear?

@pbek

This comment has been minimized.

Owner

pbek commented Mar 2, 2017

Good point, but like gitlab/github where they have enhanced markdown I see no special problem here.

there still is no standard and every feature request wanted something else

What are nested tags?

tags in tags

I wouldn't handle it that way. To add a Tag the user has to write "@tag" somewhere in the document and you then index the file for this tag in your database

In the current interface user can add tags with a button, or tag multiple notes, or move tags to other tags, or remove tags from multiple notes and you can rename or remove tags (remove a tag from a note and removing a tag altogether)...
The "new way" to handle tags would be needed to be a 2nd, alternate way with a different user interface.

Parsing is done only on save or when you import a new set of notes. The problem is not that you store the notes in a separate database, but that you store them only there.

Parsing would be done every time the current note gets saves (there are people who do that every second) while typing and when all notes need to get reloaded (for example when external changes in a note folder happen).

@cspann

This comment has been minimized.

cspann commented Mar 2, 2017

there still is no standard and every feature request wanted something else

I see no problem in the lack of a standard as long as the tag escape char does not interfere with markdown. When I would have to migrate I could easily replace your tagging symbol with the new one with a simple sed call on all files. The marker should just not be part of markdown. Laverna has this problem because it does exactly what I suggest but it uses the # as tag marker and instantly creates a tag. This is not good as # also prefixes h3 and higher indexed headers.

What are nested tags?

tags in tags

I would say nesting is against the very purpos of tags and would not support it. I would even think about not allowing spaces in Tags

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Mar 2, 2017

I've used the app that does something very close to request - Wikidpad. It's made around parsing notes for meta-data. I really love the concept, but I ditched it in favor of QOwnNotes without any regret. It's simply not worth the hustle.

Want to save tags outside of SQL? There's an in-app script to put it to a file name.

Want to make it portable? Well, it's not really portable, since there's no standard, and no other apps parse in-text tags. To get tags for Epsilon notes, which I use on my phone, I've ended up getting them from SQL and making an index note with links to all notes sorted by tags.

Getting tags and putting them as the first line of every note in "@" format is not a big thing. But what to do with that next?

@Losiara

This comment has been minimized.

Losiara commented Mar 11, 2017

Markdown is an html. Tags my be placed in <tag> tag as attrs or values. This will be ok in exporting to html/pdf and easy for parsing.

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Mar 11, 2017

Could you clarify? The app renders <tag>Test</tag> as text.
I personally prefer [tag]: (Test) for hidden text, but it has a problem with spaces inside brackets.

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Mar 11, 2017

I've put some script prototype:

import QtQml 2.0
import com.qownnotes.noteapi 1.0

QtObject {
        function handleNoteTextFileNameHook(note) { 
        if (note.noteText.substring(0,6) == "[tag]:") {
            note.noteText = "[tag]: (" + note.tagNames() + ")\n" + substring(note.noteText.indexOf("\n") + 1)
            script.log( "[tag]: (" + note.tagNames() + ")\n" + note.noteText.split("\n").slice(1).join("\n") )
        } 
        else {
            note.noteText = "[tag]: (" + note.tagNames() + ")\n" + note.noteText
            script.log( "[tag]: (" + note.tagNames() + ")\n" + note.noteText )
        }
    }
}

But it won't work because note.noteText is a one-way thing, I can't write to it from script. You can see how it is supposed to work in the log, try use smaller notes. Tags syntax can be altered.

As far as I understand, it can be implemented only:

  • if @pbek will make some function to freely edit note text from a script or a hook like handleNoteTextFileNameHook but for text.
  • via running external shell commands which I can write only for Linux.
    But it's easier to rip all data directly from SQL for all notes, and I already have a shell script for that.

Also It would be good to be able get nested tags like nested/tag.

@Losiara

This comment has been minimized.

Losiara commented Mar 11, 2017

For example <t tag="a, b, c"><tag> or <tag>a, b, c</tag> and so on.
I thought that it should been rendered as invisible in preview without any additional efforts. Because of unknown tags.

IIANM there is a function convertHtmlToText. It should omit this html in rendering.

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Mar 11, 2017

I use markdown because it's portable and "future-proof", so I'm trying not to put anything tied up to specific app. So I don't think using convertHtmlToText is a convenient solution.

<t tag="a, b, c"><tag> is hidden in QOwnNote, <tag>a, b, c</tag> isn't. But both are visible in Epsilon Notes I use on my phone.

Anyway, if there'd be a script function to edit text, it will be easy to change format.

@Losiara

This comment has been minimized.

Losiara commented Mar 11, 2017

But both are visible in Epsilon Notes

Thats because Epsilon notes is out of standart. Usually Markdown - is "wrapped and rendered as html". So "unknown tags" should be ok. But I realised that webengine is not an option for this project. So your option seems right.

@fairplay

This comment has been minimized.

fairplay commented Apr 11, 2017

Any inline tagging will be easy to implement using scripting, but API lacks "removeTag" method, is it possible to add it?

@pbek

This comment has been minimized.

Owner

pbek commented Apr 11, 2017

@fairplay can you please explain in more detail what you are trying to do?

@fairplay

This comment has been minimized.

fairplay commented Apr 11, 2017

I have a little script which parse third line for tags in org-mode format (":tag1:tag2:tag3:")

import QtQml 2.0

QtObject {
    function onNoteStored(note) {
        var lines = note.noteText.split(/\n/);
        var tags = lines[2].split(':');
        for (var i = 0; i < tags.length; i++) {
            var tag = tags[i];
            if (tag.length == 0) {
                continue;
            }
            script.tagCurrentNote(tag);
        }
    }
}

But I need to clean up existing tags before setting new ones. Did I miss smth?

@pbek

This comment has been minimized.

Owner

pbek commented Apr 11, 2017

Does that script already work? It's a bit dangerous, because script.tagCurrentNote tags the current note, and not the note that is provided with onNoteStored. script.tagCurrentNote also takes care of updating the user interface of the current note.

And your script only would take care of the current note (when you modify it in QOwnNotes), it doesn't take care of all other notes.

@fairplay

This comment has been minimized.

fairplay commented Apr 11, 2017

Ye, thanks, it was mistake, but usually it works because it seems that note stored is current note usually.
Nevertheless, it causes the error "UNIQUE constraint failed" on the second call, obviously.
Can I manipulate note's tags directly somehow, without "tagCurrentNote" usage?

@pbek

This comment has been minimized.

Owner

pbek commented Apr 11, 2017

The links between tags and notes are stored in notes.sqlite in your note folder, if that is what you mean.

@fairplay

This comment has been minimized.

fairplay commented Apr 11, 2017

I understood that, I mean "directly" using scripting API.

@pbek

This comment has been minimized.

Owner

pbek commented Apr 11, 2017

No there isn't. If you would want to play around with the code, https://github.com/pbek/QOwnNotes/blob/develop/src/api/noteapi.cpp is a good point to start.
You would need Tag::removeLinkToNote to remove the tag from a note. See: https://github.com/pbek/QOwnNotes/blob/develop/src/entities/tag.cpp#L539

@pbek

This comment has been minimized.

Owner

pbek commented Sep 18, 2017

@fairplay, @Maboroshy something to play around with

17.09.5

  • added two new scripting-methods addTag and removeTag to NoteApi to
    add and remove tags from notes (for Issue #530)
    • custom tagging could now be implemented by parsing the note text for
      custom tag handlers and adding/removing tags from the note
    • for more information please take a look at the scripting documentation
@pbek

This comment has been minimized.

Owner

pbek commented Sep 18, 2017

And I'm working on a QQmlListProperty<NoteApi> NoteApi::fetchAll(int limit, int offset) to fetch all notes, but I haven't tested it yet.

@pbek

This comment has been minimized.

Owner

pbek commented Sep 18, 2017

There now is a new release, could you please test it and report if it works for you?

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Sep 18, 2017

Good stuff. I'll start looking into tags "parser". But it would still require note text modification, that can be done only externally now. When user drops tag by UI the tag should also disappear from the note text. So more Python job again. Not that I'm having anything against it though.

fetchAll thing is interesting. I'm still hoping to resolve the issues I had with externally creating and tagging note in a single script run.

@pbek

This comment has been minimized.

Owner

pbek commented Sep 18, 2017

Yes, the new methods were only meant for "one way". Parsing the note text and adding/removing tags. You really also want to modify the note text?

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Sep 18, 2017

I think it's a must. User would still be able to modify tags by UI, and he will expect the script reacts to it accordingly.

Also, I think that important part of in-text tagging is that such tags have a position. So selecting a tag should bring cursor to it. But I leave it to some second or third version if one ever will be. It would be quite painful.

@pbek

This comment has been minimized.

Owner

pbek commented Sep 19, 2017

There are so many operations with tags... Tags can also be removed altogether (I guess than all notes have to be checked and modified) and notes can be tagged in bulk. There must be a hook for everything.

The script would have to decide what to do if a tag should be added to or removed from a note.

@pbek

This comment has been minimized.

Owner

pbek commented Sep 19, 2017

And all notes would need to be scanned for tags constantly (every time when a note tree reload would be requested).

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Oct 27, 2017

Renaming and deletion can be painful though possible. But I'm mostly targeting predefined "top" tags. Like @!20:00 adds 20:00 to Reminder tag, knowing that ! = Reminder

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Oct 27, 2017

Maybe I should have put up example syntax as @top/ @tag. It would be closer to current logic.

@pbek

This comment has been minimized.

Owner

pbek commented Oct 27, 2017

I can't use slashes in tagName or in the result tags because that could be part of a tag name...

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Oct 27, 2017

My idea is that / is definable via script settings. It can have some other default, like >.

It's like _ for multi-word tagging, it can be part of a tag name too, but there's a convention it's translated to spaces for multi-word tags.

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Oct 27, 2017

Also / can be between tags, like @top / @tag. It's the matter of tuning second regex that will parse top as a parent of tag.

@pbek

This comment has been minimized.

Owner

pbek commented Oct 27, 2017

I guess script settings will not work at that time...
And of course changing every scripting part in QOwnNotes to sub-tags would break all current tag assignments, because currently the sub-tag will be fetched automatically by its name.

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Oct 27, 2017

I guess script settings will not work at that time...

You want to hardcode that? But why?

And of course changing every scripting part in QOwnNotes to sub-tags would break all current tag assignments, because currently the sub-tag will be fetched automatically by its name.

Could you clarify. I must be missing something. The app currently would parse @top / @tag without any issues. Why should that change? We just miss the part where QML script runs a function to put tag into top on a condition, which is _/_ between them.

@pbek

This comment has been minimized.

Owner

pbek commented Oct 28, 2017

You want to hardcode that? But why?

  • I'm not sure QOwnNotes can access the script settings at the right time
  • why does the user care about the API? (I'm not talking about how the script handles tags internally, I'm talking about how the script talks to the application!)
  • I can't force the script to have an internal setting

The app currently would parse @top / @tag without any issues.

Currently if QOwnNotes finds a tag named garden deep in the tag tree (like in my/deep/structure it uses that tag for the assignment. I'm not talking about parsing, I'm talking about the API with QOwnNotes.

@pbek

This comment has been minimized.

Owner

pbek commented Oct 28, 2017

Currently the tagging for the script engine works like you enter a tag for the current note with the Add tag button. If you enter garden then my/deep/structure/garden is assigned to the current note. No new tag will be created in the tag-root.
e.g. search for void MainWindow::linkTagNameToCurrentNote(QString tagName) in the code.

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Oct 28, 2017

I'm not sure QOwnNotes can access the script settings at the right time

I thought it can be part of noteTaggingHook which is perfectly getting tag marker from script settings. The hook will do all the parsing in the scripting engine. The function would look like script.nestTag('tag', 'top') and will be run during hook's "list" part when child tag is encountered. I'm not talking about outputting top/tag as a return of the hook's "add" part and then making the C++ part handle that.

If you enter garden then my/deep/structure/garden is assigned to the current note. No new tag will be created in the tag-root.

And that is the desired behaviour. I don't yet understand why won't that allow to put one tag into another the same way UI does.

@pbek

This comment has been minimized.

Owner

pbek commented Oct 28, 2017

Do you want scripts to create tags in noteTaggingHook? That is currently done by QOwnNotes itself, not by the script. The script just returns the name of the tag as string or receives the name of a tag that should be created. What do you want to be the behavior in the future?

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Oct 30, 2017

It was a busy weekend. But at least I had some time to think it all through.

My target is quite simple. I want to turn @!tag into reminder / tag in the tag tree. To make ! = reminder I need tag parsing to be completely in the QML script without hard-coding it into C++. Potentially for general tagging there can be a dictionary of any such "shortcuts" in the script settings, as an addition to proper parsing.

The tag can be either

  1. moved into other tag after it is created;
  2. created as nested.

The first, though looks simpler to implement and was my favourite, has a major flaw. You need to create a tag before putting it into other tag. So, if I'd put some nestTag function into noteTaggingHook's "list", it would work ok after UI runs "add", since it runs "list" right after. But, if I'd input tags straight to text, it would require two "list" passes - to add tag and to arrange it. Not very elegant.

Second one, which is creating nested tags, needs some changes to the handling of noteTaggingHook's return and input values. I had not come up with anything better than to leave the old behaviour for string input/output, and to use a list (or a special object) as output for nested tags. C++ part will check the output type and act accordingly.

It can even be possible to have both 'tag' and ['top', 'tag'] work at the same time, if 'tag' would first match root level tags and then go level after level.

Renaming and deletions can be handled by having list as input for nested tags. Tag tree structure changes can all be "renaming", with tagName and newTagName being two input lists.

@pbek

This comment has been minimized.

Owner

pbek commented Oct 30, 2017

That doesn't sound anything like it's currently done. :D But I don't think I completely grasp everything you are talking about.

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Oct 31, 2017

That's what happens when tinkering with in-text tag scripting for too long.
I think it will be more effective if I'll start coding the back-end of a reminder app. You'll see how it works, and decide if it's worth to implement nested in-text tagging for better integration to QON. Maybe you'll see some another better way.

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Dec 12, 2017

After a month of extensive and mostly unnecessary hacking I've made a working prototype of my Plain text reminder.

It turns @!10:00 Hello (and many others) in plain text notes into reminder. Only at CLI for now.
I would be very grateful for any feedback on it.

As I've stated earlier, it would be great to integrate it into QON by making specific in-text tags, like the one above, act as a note tag, which is nested into some specific "Reminders" tag.

@pbek

This comment has been minimized.

Owner

pbek commented Dec 12, 2017

Nice!
I naively ran sh run.py, that blocked my mouse and wouldn't quit... :) I'll play around with it when I've more time. I guess it needs to run on a system that is on 24/7 to work, right? And works just locally on that system....

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Dec 12, 2017

Right now I'm heavily studying the distribution of Python apps. Not an easy feat considering I target all 3 desktop platforms. There will be some major improvements in distribution, but not too soon. Documenting code, tests coverage and GUI are my priorities for now.

It doesn't need any "server" to work on desktop. After you start the app it will resume where you left off: reparse all text files, show you all missed reminders and start waiting for next ones.

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Dec 12, 2017

The app has module architecture and is event-driven. Model, scheduler and view are independent. So, if scheduler works on 24/7 server, and there's, say, Google Calendar, Telegram Bot or e-mail as a view, it can work on phones without dedicated GUI for them. There can be a web socket as a view and remote GUI on that socket. Models, schedulers and views theoretically can be linked in various ways.

@pbek

This comment has been minimized.

Owner

pbek commented Dec 12, 2017

Cool, seems you did a great job!

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Dec 13, 2017

Your test will show if its true.

@pbek

This comment has been minimized.

Owner

pbek commented Dec 14, 2017

Strange, even after a sudo pip install watchdog python-dateutil I then got the error message ImportError: No module named 'watchdog' when I run python3 run.py -d.

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Dec 14, 2017

Weird. Is it macOS?
Please, try pip3 install watchdog python-dateutil.

python3 -c 'import watchdog, sys; print(sys.modules.keys())' will try to import watchdog and print all imported modules. If it will fails, then there's no watchdog for python3 in the system.

@pbek

This comment has been minimized.

Owner

pbek commented Dec 14, 2017

I already tried with pip3. Said that the modules were installed. It was Ubuntu 14.04, I'll try again.

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Dec 14, 2017

14.04 can be pain, it's quite outdated.

There are packages for watchdog and dateutil for 16.04. But no proper watchdog package for 14.04, so it can only be installed from pip.

Looks like my "Installation" section is too optimistic.

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Dec 15, 2017

I've checked fresh 14.04 in VM. Installed pip3, got modules by pip3 install watchdog python-dateutil and successfully ran python3 run.py -d.

I'll make environment independent bundles soon. But they will look like a scary binaries.

@pbek

This comment has been minimized.

Owner

pbek commented Dec 15, 2017

I made it run, I really hadn't installed python3-pip. Thanks!
run.py immediately changes an entered date @!8:30 to @!17-12-15/08:30 while you are typing. I guess that was your intention. If the time was in the past, the next day will be set, nice.
Reminder notifications work too, only once I had to dismiss a notification multiple times, I'm not sure why.
If you say +2d the date in the text-file will be changed, awesome!
Great work, @Maboroshy!

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Dec 15, 2017

Changing entered time is intended and is a must. The app doesn't store reminders anywhere but your text files. So@!10:00 should become a single defined reminder you want - tomorrow at 10. Not the next 10:00 every parse, which is recurring, is @!10:00* and not yet implemented.

There's an option for time to delay parsing modified files, so it won't interrupt your work. Each modification resets this timer.

There are still some bugs I hope to resolve with test coverage.

@pbek

This comment has been minimized.

Owner

pbek commented Apr 6, 2018

I will close this issue until there is more information.

@pbek pbek closed this Apr 6, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment