feature: ZkGrep / grep through note bodies

[Joelius300]

Thank you for zk-nvim, it's a great extension to zk.

I've been using :ZkNotes and live-grep in the zk_root to find but neither are as ergonomic as zk list -i. Is there a way to emulate this?

I would love a zk list -i-like picker in nvim that

• searches in titles (and prioritizes those matches)
• searches in contents using the available fts-database
• displays the note's title instead of the path
• previews the file
• updates search results with input (possibly with debounce)

AFAIK this is all functionality available in zk list -i

One additional feature that I would love is to be able to write #ADML and it automatically filters for the tag ADML.

I think this is achievable by using something like telescope.nvim, sending the input to zk match in the appropriate format (after debounce), collecting the results, transforming them into something that works for telescope's listing and preview and done.
This would allow to prompt with all of the available filtering options.

Unfortunately, I don't know how to do that but I imagine I'm not the first to think of this. Is this a desirable feature for zk-nvim and/or does anyone have any pointer on how to implement this?

Relevant issues: #73 #68

[mickael-menu]

Overall I agree that this would be interesting to have. Unfortunately zk-nvim doesn't have an official lua maintainer and I'm not versed in lua myself. But if someone is interested in contributing this feature, please chime in in this discussion.

[shfattig]

@Joelius300 Is this something you're still looking for? I too would appreciate this functionality, but likely something more like zk edit --interactive so that I could create a new note if search result is empty. I'm a lua novice, but I'd be willing to try my hand at an implementation!

[vicrdguez]

Hi! I'm playing with something similar as well.

for 2. Overall I don't see much value into ZkNotes also searching in note contents. It will very likely hurt performance, and we can have probably more powerful full text search by just using any Fuzzy finder grep/live_grep commands and achieve the same.

for 1. and 3. I think it is already the case if your note has a title? (either an H1 with the title or a title: property in the yaml frontmatter

about 4. I would say also is implemented if you are using telescope.
Not sure about what you mean in 5.

What I'm looking to see how to implement is to include note aliases as entries on the note picker. Something like:

> important note
  alias (important note)

which will let you open notes by aliases as well, showing the original title just after probably in a different color

[Joelius300]

Thanks for the comment :)

Obviously full text search is slower than just the titles but I would like to be able to do exactly that with the additional information on titles etc. for displaying instead of filepaths. Also the zk database is made for full-text search from what I understand so using that is probably faster than always calling ripgrep, no?

1 and 3 are implemented in ZkNotes, yes, but that does not do any full-text search which is a must. Same goes for 4.

With 5 I just mean that you have a live grep, unlike with the zk match command where you type the full query, press enter, and get one set of results for just that query.

Filtering for tags would also only be possible using the database and an amazing feature. Respecting aliases too.

A simple live grep has some of the things required, namely full text search, preview and being live but it's missing titles in search results, respecting aliases (I think it does that) and title match prioritization that zk edit -i provides. Filtering for tags is available in zk match but that isn't live (also the zk commands are in terminal). ZkNotes is currently the least useful one to me as it does not even do full text search.

[mickael-menu]

Technically the FTS feature is already exposed through the zk.list LSP command (see the match property in https://github.com/mickael-menu/zk/blob/main/docs/editors-integration.md#zklist). The problem is that this is fired once when using ZkNotes, then typing more in the fzf/Telescope pop-up will filter the results further but without FTS.

If I understand fully, what you want is triggering the zk.list command with match property every time you type in the picker input? I don't know if this is possible with fzf or Telescope, maybe with a custom source. In any case that could be a new picker with a dedicated UI.

Now we just need a Lua dev that is interested and capable of tackling this 😉

[vicrdguez]

Given the behavior of zk list/edit -i I would say that what the command does is just fetch the the full raw content of each note and add it to fzf as a searchable target along the title. @mickael-menu would be able to confirm. If that's the case then that's much different than a "live query/live grep" scenario, since there is no fetch to the DB on each keystroke.

If that is actually not the case, and zk queries the DB (sqlite I think?) on each keystroke, the problem I see with translating that into zk-nvim is that there are more steps than just that to get the result into the picker:

• call the api that reaches the lsp
• the lsp will then fetch the DB
• parse the resulting json into lua tables, where performance I'd say will depend on the size of the note
• iteratate over results and build picker entries
• refresh picker entry list with the new set of results

This will repeat on each character typed on the picker prompt. I would say that the picker would not be the bottleneck in any case (at least for fzf/fzf_lua, can't speak that much about telescope in this matter). This is where I see the potential performance issues, more than the fact of using the DB or ripgrep in this case. I would say ripgrep would be faster, but can't confirm.

That said, I think it would be doable at least with fzf-lua which is what I'm familiar with, by using the fzf_live, that gives you tools for this, that someone might want to pick up to implement it. I have no clue in the case of Telescope though.

Personally, note titles are "my APIs" for finding what I need from my mass of notes. I think heavily on how should I name them, thinking on what I would search my self if I were say, googling for answers on something. Then I provide aliases that are in essence other potential ways I think I would be searching for the particular thing in the future. That's why I was looking forward having aliases as entries in the pickers, which I'm close to finish:

In this case I can look for kafka log4j or reload4j kafka and still get the exact note I wanted since the original title is also a searchable target, letting you search for multiple combinations, which is something I was missing on the zk cli as well

[mickael-menu]

Your analysis is correct @vicrdguez. Reproducing exactly the same behavior as zk list -i would require sending the whole note content of all notes through the LSP API and adding them to the fzf/Telescope lines. The LSP response in particular might be quite slow on a large notebook.

[shfattig]

I have an initial pass on this working! Since the operation is very similar to Telescope's live_grep I took that code as inspiration. I was able to figure out how to execute zk list -m <prompt> for every keypress.

• Unfortunately I'm noticing differences in the results from zk list -i and zk list -m ... The latter sometimes will not show results until you type a complete word?
• Also, I could not find a way to output line numbers for the matches, so right now we're just limited to finding the entire filename. The --format=oneline option is the closest I could find, but no line numbers!
• I don't know what speed will be like. We can run some tests later, but I'm hoping this is close enough to suit our needs, at least as an initial pass!
• I'm still missing the option to create a new note if search is empty, but I know that's possible!

Do you want me to make a pull request for this?

[shfattig]

oh but I do have Telescope previews working for the files, so you can see contents as you search!

[Joelius300]

My attempt is this. It just opens zk list -i in a terminal, tries to get the output of it and extract the path, then open it. As you can see there's tons of debugging prints etc. in there because it does not work and I have no idea why. I have very little experience with lua and nvim scripting but it feels very close; often times the output is there but fails to be parsed, although it's well formed and complete when copying it out from the printed debug log. I'm at a loss, maybe someone can help me figure this out.

The utils is imported from astronvim (local utils = require "astronvim.utils") but it's just sending a toast-like popup in the top right, I only used it as debug print. The syntax with the table entry ["<leader>zi"] is also from astronvim but as far as I know it's pretty simple to just take the function and add it as a command the conventional way.

Without all the comments and prints:

["<leader>zi"] = {
  function()
    local Terminal  = require('toggleterm.terminal').Terminal
    local output = 'NO OUTPUT'
    local termOptions = {
      cmd = "zk list -i -q -P -f json",
      hidden = true,
      close_on_exit = true,
      -- store stdout from process
      on_stdout = function(t, job, data, name)
        if (data ~= nil) and (data[1] ~= '') then
          output = data[1]
        end
      end,
      -- grab path from json output after terminal closes (can then be used to open the file)
      on_close = function(t)
        local worked, parsed = pcall(vim.json.decode, output)
        if worked == false then
          utils.notify("COULD NOT PARSE: " .. parsed)
        else
          utils.notify(parsed[1].absPath)
        end
      end
    }
    -- create and open terminal, callbacks are set up
    local zkList = Terminal:new(termOptions)
    zkList:toggle()
  end, desc = "Use `zk list -i`" },

Raw:

["<leader>zi"] = {
  function()
    local Terminal  = require('toggleterm.terminal').Terminal
    local output = 'NO OUTPUT'
    local termOptions = {
      cmd = "zk list -i -q -P -f json",
      hidden = true,
      close_on_exit = true,
      on_stdout = function(t, job, data, name)
        -- if #data[1] > 100 and #data[1] < 1500 then
          -- utils.notify("very top" .. data[1])
        -- end
        if (data ~= nil) and (data[1] ~= '') then -- and (data[1].sub(1,2) == "[{") then
          output = data[1]
          -- utils.notify("just before sub" .. output)
          -- if output:sub(1, 1) == '[' then
            -- local len = string.len(output .. '')
            -- utils.notify((output .. ''):sub(1, 20))
            -- utils.notify(string.sub(output .. '', len-20, len))
            -- utils.notify("after sub" .. output)
          -- end
        end
        -- vim.cmd('e ' .. response.absPath)
        -- utils.notify(response.absPath, vim.log.levels.WARN)
      end,
      --on_exit = function(t, job, exit_code, name)
        --response = vim.json.decode(output)
        --print(response)
        --print('a' .. response.absPath)
        --utils.notify(response.absPath, vim.log.levels.WARN)
      --end,
      on_close = function(t)
        -- utils.notify("on_close")
        -- local response = vim.json.decode(output)
        -- utils.notify(response.absPath)
        -- utils.notify(output:sub(1, 20)) # note that it appeared that sub modifies the string
        -- but online it says it doesnt..
        -- Current issue is that not the entirety of the output appears to be grabbed, for Named Entities
        -- the initial bracket is often missing (1 in like 5 times it was there), others like LU-Zerlegung
        -- aren't working at all?? maybe some cache issue but wtf
        local worked, parsed = pcall(vim.json.decode, output)
        if worked == false then
          utils.notify("COULD NOT PARSE: " .. parsed)
        else
          utils.notify(parsed[1].absPath)
        end
        -- print(response)
        -- print(output)
      end
    }
    local zkList = Terminal:new(termOptions)
    zkList:toggle()
  end, desc = "Use `zk list -i`" },

[tjex]

this is effectively the latest functionality of :ZkNotes (except it only searches through the title, and not the body). Closing for now.

[Joelius300]

Thanks, but unfortunately searching in the body is by far the most important part for me :/
Currently, I am just using live-grep but that doesn't display the note titles (only paths) and it doesn't make use of the already populated database by zk (I guess that's optional but it would make sense, no?).

Please consider reopening.

Ps. I do understand that good naming and aliases can help (is this now handled by :ZkNotes?), but there are times I am searching for very specific phrases or sometimes even parts of formulas, which don't make sense to put in the aliases IMO.

[tjex]

Understood. I'll reopen and change the title so that it matches the current need of feature (grepping). This will help people find it, who are also keen for this feature (I am personally keen for it). There have been some other spots in discussions/issues that call for a Telescope grep picker.

[shfattig]

Hey guys, I think my attempt is pretty close to what we're looking for. I'm going to try getting a PR up in the next couple weeks so y'all can test it out. The outstanding issues mentioned previously (I implemented this almost a year ago!) I think may need to be addressed on the zk side, but we can discuss that when we get there
#127 (comment)

[tjex]

awesome!

[shfattig]

draft PR is up at #171