This is the ekg manual, which describes the operation and customization of the ekg package. All information here is relevant to the released version only.
The README is also informative, has screenshots, and can be found in the source and in the git repository, which is at https://github.com/ahyatt/ekg.
The ekg module is a simple but opinionated note taking application, for emacs. It is a substitute for such other emacs applications such as org-roam or denote. ekg stands for emacs knowledge graph.
Data is completely stored in a sqlite database. Notes are organized around tags, and you can view many notes by looking at one or more tags. This provides a read-only view of various notes, which you can navigate between and choose to edit in a separate buffer.
The editing of notes combines the editing of the text of the note with various properties stored in the database such as the tags of the note.
If using Emacs version 29, you likely have sqlite as a library compiled into
Emacs. Prior versions use emacsql for access to sqlite. If you are using a
pre-Emacs 29 version, you need to have sqlite installed on your system.
This is the easiest and recommended way to install ekg. If you have not set up Melpa yet, follow the instructions at https://melpa.org/#/getting-started.
There is no need to use Melpa Stable, because development happens in a branch and is not integrated until it is deemed stable. When bugs are discovered, fixes are pushed to the main branch soon, so it is usually a good idea to keep your version up to date.
There are many ways to use Melpa to download ekg, but using use-package is the
easiest way, and recommended because it allows you to get the pacakge and
configure it in one place. An example use is below.
(use-package ekg
:bind (([f11] . ekg-capture)))If you wish to install by hand, you need to make sure to install the triples library, found in GNU ELPA, and at https://github.com/ahyatt/triples.
Clone the ekg library, from whatever branch you would like to use (main
corresponds to the release version, and develop is where development of the
next version happens). Add your source directory and require ekg. The
following is an example assuming you cloned ekg into ~/src/ekg, and the
triples library is already installed.
(add-to-list 'load-path "~/src/ekg")
(require 'ekg)- Add export to denote (by Jay Rajput)
- Run tag hooks when a valid tag is typed, instead of relying on completion, which often doesn’t work.
- Fix issue in
markdown-modewhere the beginning of the tag values was incorrectly read-only. - Allow emojis as tags.
- Add tag and similar notes to the context when having LLMs add to or replace
notes.
ekg-llm-default-instructionsreplaces the previous variable,ekg-llm-default-prompt(because the prompt now contains the instructions). - Add inline links for Markdown, as wiki links.
- Improve regex for inline tag linkification so we linkify more valid inline tags.
- Improve logseq import to import notes without org structure, and use filetags.
- Fix inline tag bug that test and real failures.
- Simplified trashed tags from using tag prefixes to using a simple trash tag (
ekg-trash-tag). - Added
ekg-show-notes-with-tag-prefix. - Add tags while writing the body of notes by completing with “#”. See Note text section for details.
- Added functionality so that org-mode commands, and potentially others, narrow automatically so the metadata line and the read-only nature don’t mess things up. See Note text section for details.
ekg-auto-save, a new module by Qingshui Zheng.- Added canceling a note as an actual command, also by Qingshui Zheng.
- Fix issue with trashing notes with tags that are partially trashed.
- Saving while editing a note will no longer add the draft tag. This means that if you are editing a draft, it will not keep it as a draft, so it will become a normal note.
- Fix issue with using ekg-llm with vertex, which doesn’t understand system prompts.
- Fix metadata face which didn’t work well with all themes - now the metadata section is just bolded.
- Improve display regenerating embeddings via
ekg-embedding-generate-all. - Change multi-title note display from using newlines to commas
- Made deleting titles and resources possible, and properly skipped empty properties for multi-line propeties.
- Handle backup errors better, warning and proceding if not a forced backup, erroring out with a better message if forced.
- Don’t show trashed notes in drafts.
- Added autoloads
- Fixed issue that could occur when saving with malformed buffers.
- Switch LLM chat output to streaming when available.
- Fix inclusion of title-based transclusion “>t”, which included the “t” as part of the completion.
- Fix tag renaming possibly causing duplication.
- Ensure renamed tags are normalized.
- Support metadata where fields are specified via multiple property lines and make “title” such a field, so now titles can have commas.
- Fix issues using default emacs in-buffer completion, and allowing completion in places we shouldn’t.
- Add
ekg-embedding-generate-on-saveandekg-embedding-disable-generate-on-saveto turn off generating embeddings for notes.
- Added ability to save in-progress notes.
- Added “magic tags”, tags that cause elisp to be executed. See the magic tags section for more detail.
- Added
ekg-llm, a separate module, so LLMs can append to or rewrite notes, using other notes as prompts. As with ekg-embeddings, a Open API key is required. - Added
ekg-capture-fileto save notes associated with a file, or go that same note. - Depend on the llm package for embeddings and llm functionality, so the user can choose different providers.
- Improved
ekg-clean-dbto fix bugs and change empty-note deletion logic. - Improved
ekg-embeddingpackage to make it more robust to missing embeddings (like what can happen if you save notes without loading the package first). - Added commas to the tag header.
- Made separator in
ekg-capture-modeandekg-edit-modecustomizable. - Fixed display of org notes to have properly formatted links and images. Links can be opened with
[C-c o]. - Fixed bug where in embedding search and buffer similarity, the highest match was discarded.
- Fixed bug in title transclusion for company users.
- Changed template behavior to also use parent tags (so templates on “foo” tag, will work if the note tag is “foo/bar”).
Thanks to contributors Gleek and Qingshui Zheng.
- Fix native compilation errors. Upgrade to triples version 0.3.5, which contains more important bug fixes.
- Upgrade to triples version 0.3.2, which contains important bug fixes.
- Upgrade to triples module 0.3, which changes how integers are stored in the built-in sqlite (for users of Emacs 29+). Users of sqlite will have their database automatically upgraded. A backup will always be made beforehand - you may want to find it (alongside your normal emacs backups), and make sure to keep it around in case the upgrade went wrong in some way. Important: if you created your database before this version on the built-in sqlite, and afterwards switched to emacsql, you must switch back to the built-in sqlite for the upgrade.
- Store the ekg version in the database so we know when we need to do updates in the future.
- Remove older database updates that should no longer be needed.
- Ensure we always are connected to the database before any call to the database happens.
- Make ekg-close interactive.
- Fixes to ignore bad embeddings which otherwise would cause errors.
- Added the ability to kill notes in a notes view, which does not change the database, it only alters the view. Thanks to Jay Rajput for the contribution!
- Fix for tag cleanups, which were sometimes not cleaned up if the tag had other data (such as embeddings).
- Added variable
ekg-embedding-text-selectorwith a default function so that large notes can have their embeddings taken. - Added
ekg-get-notes-with-title, which is offered as a useful function for clients.
- Added inline commands, see the inlines section for more detail.
- Added customization of note display, using inline commands.
- Added logseq import / export in its own module, and removed it from the
ekg-org-roammodule. See the logseq section for more details. - Improve window management, now we play nicer with customized window configuration, and now opening a list of notes will also switch to that window.
- Improved metadata overlay look and function, it now looks just like
message-mode, which hopefully will help with people’s intuitions on how it works. - Added a blank line between notes in notes list buffers, for a cleaner look.
- Renamed
ekg-rename-tagtoekg-global-rename-tagto clarify this isn’t for changing an individual tag in a note. - Added arg prefix behavior to
ekg-notes-deleteto allow deleting notes without a warning. - Made
ekg-capturehave unique buffer names, so the user can capture multiple notes at the same time.
Thanks especially to users and contributors: Jay Rajput, Qingshui Zheng, And cuprum.
- Removed
ekg-notes-remove, which removed one or more tags from a note from a note list buffer. - New keybinding, “q” in the notes buffer, which kills the buffer (thanks to Jay Rajput for the idea).
- Added hooks
ekg-add-schema-hook,ekg-note-pre-save-hook,ekg-note-save-hook,ekg-note-delete-hook, andekg-note-add-tag-hookto enable customization. - New commands
ekg-show-notes-latest-captured,ekg-show-notes-latest-modified, for showing notes created or modified recently. - Introduced variable
ekg-notes-sizeto control the default page size for limited views such asekg-show-notes-latest-captured. - Added templating.
- Added embedding as an optional add-on, to enable note similarity and note search; requires an account at OpenAI or similar embedding provider.
- Added new function
ekg-active-notesto easily get all non-trashed notes. - Improved ability to have note list buffers that have flexible titles and operation, notably improving the
ekg-show-notes-in-trashcommand. - Standardized buffer names for
ekg-notes-modebuffers, which all are prepended with “ekg” and surrounded by asterisks, to denote that they are non-file-based. - Added the documentation you are reading right now.
- Fixed bug interfering with completion at the beginning of the tag property line.
- Fix for
ekg-notes-refreshincorrectly callingekg--show-notes.
By default, ekg uses the default triple database, which is set in the variable
triples-default-database-filename. The default value of this is
~/.emacs.d/triples.db. You can specify a different name if you want the ekg
database to not be shared with any other user of the triple package, by
customizing the variable ekg-db-file. When this is nil, it leaves the
filename up to the triples package.
The ekg package is built on a flexible database scheme called “triples”, where everything is stored as a graph structure: a subject, a predicate, and an object. The implication for the ekg package is that new kinds of data are easy to add, and live alongside other data. Values of properties, stored as “objects”, can themselves have values by adding data where the same value is the “subject”. If you plan to do extensive integration work in elisp, it will help to understand these concepts, and the best way to do so is reading the triples package README.
For notes, we can think of the subject of the triples as an ID. Notes are created, and have the following types by default, with the type having properties.
tagged: Tagstext: Text, its major mode, and any inline commands.time-tracked: Creation time and modification timetitled: Title
The ID for notes is by default an integer UUID. However, you can have notes about anything. In EKG an ID can be a resource identifier as well, such as a URL. When this happens, the ID is interesting data in its own right.
Tags may have spaces, but cannot have commas, which are used to separate them when showing them to the user and parsing them back out into properties to store. Tags also may not contain uppercase letters.
Because of the triples model, there is data about the tags for each note. Tags themselves just have type markers indicating they are tags, and can dynamically query for all notes with their tag, so tags always have a current list of notes with their tag.
When capturing or creating a note, the note buffer has two areas important to understand. The first is the area for note properties, which has a different background color. The second is the area for the note text.
The properties shown in the note property area come from the data stored in the database for the entity. At a minimum, there will be tags.
A property is displayed with a label, and the value, such as
Tags: emacs, ekg
Changing these values, when saving the note, will change the values that will be stored in the database.
New properties can be added manually, so if you wanted a title, you can add it to the property list.
Title: This is my title
It’s important to note that everything in the section with the different background color is a property and will be treated as such. Text that doesn’t look like a property there will cause problems, and properties outside this area will instead be treated as note text. The end of the property section ends with an uneditable “–text follows this line–”, below which the text of the note starts.
Not every property has a representation in the property list, only the properties which users may want to change manually.
Tag properties have completion to tags built-in, so when adding tags to a note you can choose from available tags. Or you may add a new tag that has not yet been used.
Ekg makes some effort to make sure that the user doesn’t accidentally extend the property section without adding actual properties, since this will likely result in a confusing experience for the user.
Below is the property section is the note section. The text could be anything (or nothing). This is the body text of the note, where you write down whatever you want to note about, that is relevant to the tags for the note.There are three modes for the note text: text-mode, markdown-mode, and org-mode.
More can be added by customizing the variable ekg-acceptable-modes, just make
sure its a mode that makes sense for notes. The default mode is configured in
ekg-capture-default-mode, but can be changed when capturing with the command
ekg-change-mode.
The note text provides various options for completions. The most common
completion is typing tags inline while writing notes. These tags will be added
to the note automatically upon saving, regardless of whether completion is used
or not. The tag completion is triggered by the “#” symbol. In org-mode, if
ekg-linkify-inline-tags is set to non-nil (which is the default), the tag will
be turned into an org link to the tag. It is necessary to enclose the tag in
square brackets to be detected as an inline tag. In markdown-mode, the tag will
be turned into a wiki link (denoted by double square brackets). It is
acceptable to finish the completion with a tag that is not currently defined, as
the tag will be added when the note is saved. The detection of plaintext tags of
various types can be turned off by setting ekg-inline-populate-inline-text-tags
to nil. There are other customizable tag symbols available, indicating different
prefixes. By default, in addition to the “#” symbol (representing a tag with
that name), there is also the “@” symbol for tags prefixed with “person/”, and
the “!” symbol for tags prefixed with “idea/”. These other symbols and the
prefixes they mean are controlled by
ekg-inline-custom-tag-completion-symbols. By default, we have “@” which will
denote a tag with the “person/” prefix, and “!” which will denote a tag with the
“idea/” prefix. So if, in either org-mode or markdown-mode, the text has the
following:
Everything related to #[emacs] should be colored #ffa500. This is the opinion of @[rms].
the tags that will be detected are “emacs” and, because of the special “@” prefix that indicates a tag prefix, “person/rms”. The color will not be picked up as a tag, because it is not enclosed in brackets. This helps us avoid false positives.
There is no functionality to remove inline tags that are deleted in the tag section. If an inline tag is not deleted in the note text, the tag will be re-added.
Another type of completion is for inline commands, the “>t” completion, mentioned in the Inline commands.
The metadata section above the notes often can be problematic for some commands,
especially org commands. Because of this, before commands are executed, we check
the command name against the ekg-command-regex-for-narrowing, and if there is a
match to one of the regexes, we narrow to the note section just when the command
is running. It defaults to all org-insert commands and org-meta-return, but if
there are any weird behaviors caused by the metadata section, consider
customizing this variable. Right now this just works for keybindings, and not
using execute-extended-command.
Notes can be saved midway through editing, both for capturing and editing notes.
The normal buffer-save keybinding (typically C-x C-s) will save a draft. A
draft is like a normal note, but has a special tag, by default “draft”. (This
can be customized in ekg-draft-tag.) Having this tag means the note doesn’t
show up in most views, much like the notes in “trash”. Once a note is saved
normally, it loses the draft tag.
See also the section on the auto save, to see how to turn on and set up auto save, which can automatically save drafts for new notes.
Org-mode notes are primarily to use org-mode formatting on. Org-mode has a lot of funtionality, but much of it depends on the assumption that the buffer is all for use by org-mode (not true in this case, because of the properties portion), and the assumption that the buffer is visiting a file, which is also not true. In particular, attachments will not work, and ekg-notes cannot be added to the agenda.
ekg-capture is the command to capture a note. In ekg this is probably the most
frequently used command. It will create a new buffer called *EKG Capture*. By
default, it will have the current date tag, such as “date/2023-02-21”.
ekg-capture-url will capture a note associated with a URL resource, and with
a given title as the title of the page. The idea is that the note is annotating
the reference itself as a “literature note”. The title also appears as a tag,
so other notes can reference this if needed. For example, if the URL is
http://example.com, and the title is “An example URL”, the properties buffer
will have the following:
Resource: http://example.com Tags: doc/an example url, date/2023-02-25 Title: An example URL
Capturing URLs is a bit clunky as is. If you can wrap it in a function to supply the name and url of the active browser tab, then you can create a much easier experience. The following is an example for users of Google Chrome on Mac OS X.
(defun my/ekg-capture-url ()
(interactive)
(ekg-capture-url
(do-applescript "tell application \"Google Chrome\" to return URL of active tab of front window")
(do-applescript "tell application \"Google Chrome\" to return Title of active tab of front window")))URL can also point to local files, which will be browsed using find-file by
default. The idea is that you can tag files and folders to make them easier to
find. Here is an example note similar to web address URL:
Resource: file:~/notes/20230510T162600__emacs_init-file.org Tags: doc/emacs config, date/2023-05-13, emacs/init Title: Emacs Config
You can use the function ekg-capture-file to either capture or edit a note
associated with a file from a buffer visiting that file. (If there is already a
note associated with the buffer’s file, the note will be opened.) You can use
this to store TODOs and other notes about a file.
ekg opens web addresses in a browser using browse-url and everything else in
Emacs using find-file.
A final way to capture notes comes from a buffer that is viewing a list of
notes, in ekg-notes-mode. You can call ekg-notes-create, which will capture
a new note with whatever tags (if any) are associated with the notes buffer.
To save any note that is being captured, press C-c C-c or call
ekg-capture-finalize. To cancel, just kill the buffer. You can also abort with C-c C-k, or ekg-capture-abort, which will not only kill the buffer but delete any draft saved.
For example, if there is a note with tags “daily reflection” and “template”, with the text “What did you learn today?”, adding the tag “daily reflection” to a note will cause the text “What did you learn today?” to appear. The parents of tags are also searched, so the same thing will happen if the tag you add is “daily reflection/morning” - it will get the template for “daily reflection” as long as it exists.
The adding of templates happens whether intially when setting up the capture buffer, or later when the user types a tag that is a valid tag. Because of this, it’s best to avoid adding templates to tags that are prefixes of other tags you’d like to use, but don’t want the template on, because as soon as ekg sees the prefix that’s a valid tag being typed, it will trigger that tag’s templates.
You can choose a tag other than “template” as the trigger for this templating
behavior, by customizing ekg-template-tag.
This functionality is enabled through the function
ekg-on-add-tag-insert-template in the variable ekg-note-add-tag-hook, and
can be turned off by removing it from that hook.
(remove-hook 'ekg-note-add-tag-hook #'ekg-on-add-tag-insert-template)The variable ekg-capture-auto-tag-funcs has a list of functions to call to add
tags. Each function is called, and returns a list of tags (or nil, the empty
list), which are all added to a new note. By default, this variable has the
function ekg-date-tag, which returns the tag of today’s date. If you do not
want this, you can remove this function. You can also add your own functions to
add the year, the week number, or any tag you feel is appropriate.
There are two kinds of inline commands: a normal command, and a note command. A
normal command can do anything, and takes the form “%(<command> <arg 1> <arg 2>
… <arg n>)”. In other words, this is just like an elisp function call, except
with a “%” in front. When executing one of these we look for a function
starting with ekg-inline-command-. So, for example, we have the following
commands available for use:
%(transclude-note id <numwords>): Include the contents of another note.numwordsis optional, and controls the maximum number of words to include. If not included, there is no limit.%(transclude-file filename <numwords>): Include the contents of a file.numwordsfunctions the same here as intransclude-note.%(transclude-website url <numwords>): Include the contents of a website. As of now, no attempt is made to only include the “main content”, so this is best suited to simple text sites that have content without any navigational elements.
These are defined in ekg-inline-command-transclude-note, and so on. A user
can define new commands just by creating new functions that fit this pattern.
All of these will be executed and content calculated every time the note
containing them is re-displayed. Note that there is currently no automatic
refresh when the content being transcluded changes.
The other kind of inline command is a note command. These function similarly to
normal inline commands, with the key difference that the form is now
“%n(<command> <arg 1> <arg 2> … <arg n>)”; note the “n” in front. The
difference here is that there is an implicit first argument that is the note
that is being displayed in the current context. After that note argument “<arg
1>” and so on will be added. These are used primarily for controlling the
read-only display of notes in notes lists. The note commands are primarily
driven by types, with the idea that a note can have many types, and each type
has a note command that displays information related to that type. Note
commands are defined in functions with the prefix ekg-display-note-. The
following note commands exist:
%n(id <force>): Shows the ID of the note, if it is interesting. Interesting mainly means it isn’t a random-seeming ID that we normally generate for notes, and is instead some sort of resource. Ifforceis true, then show it whether it is interesting or not.%n(text <numwords>): The text of a note (with any inline commands calculated and their results displayed inline).numwordsfunctions as noted above.%n(tagged): The tags of a note.%n(time-tracked <format-str>): The created and modified time of a note.format-str, if passed, controls how the times are formatted (see documentation forformat-time-string, default is%Y-%m-%d).%n(titled): The title of a note.%n(other): A special note command that will substitute itself with all type-relevant note commands that haven’t already appeared. So, for example, if there is a type such asperson, and a note has information with this type, that information will be shown in theothercommand, as if it was substituted by%n(person). However, if%n(person)already appears as a command, it will not add it again in theothercommand.
The %n(id <force>) is implemented in ekg-display-note-id, %n(text
<numwords>) is implemented in ekg-display-note-text, and so on. All these
are designed to be useful for customizing the note display (see Customizing note
display in ekg-notes-mode). Because we want to have these possibly not insert
anything, each function must end with a newline if the content is likely to be
needing a line to itself. The functions must always return a string. Although
the default note commands are all based around types, a note command could be
anything that needs a note.
Inlines can be added by simply typing them, or a few special commands.
ekg-edit-add-inline will add an inline note or file. For notes, it will
prompt to select a note by title or tag and then text. For files, it will
prompt for the file name. The other way is to use completion at point, by
typing “>t” and completing by notes with titles. After completion, the “>t”
will be replaced with the correct transclude-note command that refers to the
titled note selected. This is only useful for notes with titles, since they are
more easily selected by completion.
There are several functions to view notes in various ways. All of these show a
list of notes in read-only view, that can be navigated and interacted with.
This is a ekg-notes-mode buffer.
ekg-show-notes-with-tag will show all notes tagged with the given tag.
ekg-show-notes-with-any-tags will show all notes that have any of the tags given.
ekg-show-notes-with-all-tags will show all notes that have all of the tags given.
ekg-show-notes-for-today will show the notes taken today.
ekg-show-notes-latest-captured will show a number of notes from newest to
oldest. The number is 20 by default, but can be changed by customizing
ekg-notes-size.
ekg-show-notes-latest-modified will show a number of notes from newest to
oldest, but by modification time, not by creation time. The number is also 20
by default and can be changed by customizing ekg-notes-size.
ekg-show-notes-in-trash will show the notes in the trash (see the trash
section for details on how this works).
The notes buffer is navigated via the following commands (the default binding is also given):
ekg-notes-tag (t), open another notes buffer showing notes with one of the
tags of current note.
ekg-notes-open (o), edit the currently selected note.
ekg-notes-delete (d), trash the current note (or, if this is the trash list,
truly delete it).
ekg-notes-browse (b), open the resource attached to the current note, if it
exists, otherwise do nothing.
ekg-notes-select-and-browse-url (B), select from all the titles of URL
resources in the any of the notes, and browse the URL.
ekg-notes-refresh (g), refresh the list of notes in the current buffer, to
make sure any new notes or removed notes are updated in the list.
ekg-notes-create (c), capture a new note with all the tags associated with
the list.
ekg-notes-next (n), move selection to the next note.
ekg-notes-previous (p), move selection to the previous node.
ekg-notes-any-note-tags (a), open a new notes list showing any of the tags
that appear in the selected note.
ekg-notes-any-tags (A), open a new notes list showing any of the tags that
appear in any of the notes in the note list. For example, if the buffer was
displaying notes with tag emacs, and there are two notes displayed, one with
tags emacs and org-mode, and the other with emacs and ekg, a new buffer
displaying notes with any of the tags emacs, org-mode, or ekg is created.
ekg-notes-kill (k), kill a note from the current view. This only removes
the note from the current buffer; the database is not changed. If the view is
refreshed, the note will come back.
q will kill the notes buffer.
Many of these commands use the notion that notes lists have associated lists of
tags. That is the case for many commands, but not all. For example,
ekg-show-notes-latest-captured, ekg-show-notes-latest-modified, and
ekg-show-notes-in-trash have no associated tags.
ekg-display-note-template, which is a string that has inline commands in it
(normally inline note commands). See the inlines section for more details on
these commands. Through changing this, the ordering or inclusion of various
type-related information can be configured, or extra text added, or anything,
really.
The variable ekg-format-funcs has functions to run to format what ekg displays
to the user. Each format function runs in turn on a temporary buffer with the
note text in it, and can make whatever changes it needs to before the buffer’s
contents are displayed in a note list.
This works in a similar manner to templates, except that a template tag only takes effect when you add it, while a magic tag takes effect both when first adding it and when editing a note with the tag. But they also share the same shortcoming: if the tag is a prefix, it will trigger as soon as typed, even if you wanted to use a different tag that is prefixed with the tag.
Creating magic tags is also like creating templates. You create a note and use
a special tag that indicates this tag is a magic tag. That special tag is
“tag-defun” (but the name can be changed by customizing the variable
ekg-function-tag). This tag is itself a “magic tag”, and once you add it to a
note, the note will change to be in emacs-lisp-mode. Notes co-tagged with
this will take effect for any notes with those co-tags (again, just like
templates). For this reason, it’s probably best to avoid having any date tags
co-tagged, since users probably don’t want them to be magic tags. To illustrate
the example that in this section, you could have a note with tags “chinese” and
“tag-defun”. This note could have the following content:
(set-input-method 'chinese-b5-quick)In this example, once a note is added with “chinese”, this function will be run, and all subsequent editing of the note will have this function run. Note that there can be only one elisp expression in the note; if you have multiple, only the first will be used. It is not advised to have complicated elisp here, since it is not amenable to debugging. The code is run in the context of the note buffer, after the text has been inserted.
For tags that are a hierarchy, each level in the hierarchy is tried in order, from least specific to most specific. So, for example, if the tag was “chinese/writing practice”, first we would try “chinese”, apply any functions found there, then try “chinese/writing practice”, and apply any functions found there.
Notes deleted from note lists (ekg-notes-mode) buffers are not deleted outright,
but rather put in the trash, which is done by adding the ekg-trash-tag, by
default, “trash”, to the note. Any note with the “trash” tag will not be shown
in normal tag buffers.
Trashed notes can be seen by calling ekg-show-notes-in-trash. If notes are
deleted from this list via ekg-notes-delete again, they are deleted permanently.
The function ekg-notes-delete will always delete a note if the note is in the
trash, and trash it otherwise. If you want to un-trash the note, you can remove
the trash tag.
Both notes in ekg and certain note list buffers can be stored and linked to in
org-mode. To store a link to a note, you have to edit that note and call
org-store-link. That function can also be called in a ekg-notes-mode buffer
created by ekg-show-notes-with-any-tags. Other list types currently will just
store their tags assuming the user wants a link to a list with any of the tags
in the list.
You can import your notes from org-roam. This will turn all titles into tags,
and all links will become tags as well, as well as any tags org-roam thought
were in the document. At the moment, the import is started via executing elisp,
since importing can be fairly idiosyncratic, and ekg and org-roam have different
ways of expressing the same thing that you may want to change. It’s best if you
looked over ekg-org-roam.el and see what is going on, but at least read the
following description before manually executing (ekg-org-roam-import).
The import is idempotent, so it always will import to the same entities,
overwriting older data with new data. If you want to update what is in ekg, you
can just rerun the import. In the import, titles and links will be normalized
to ekg’s tag format (they will be downcased and have any commas removed). If
you have tags you want to turn into prefixes (which is a good idea for tags
widely applied, which essentially act as a categorization), you can add those
tags to the list at ekg-org-roam-import-tag-to-prefix. For example,
(setq ekg-org-roam-import-tag-to-prefix (append ekg-org-roam-import-tag-to-prefix '("idea" "person")))Then, when a note is found that is tagged with “idea”, but with title “emacs is a powerful tool”, then the title in org-roam will be turned into the ekg tag “idea/emacs is a powerful tool”, and anything linked to it will also get the same prefix.
By default, the ekg package will back up its database, using the backup
functionality built into the triples library. The backup behavior is controlled
by ekg-default-num-backups, set to 5 by default, and
ekg-default-backups-strategy, set to daily. These are, on first use of ekg,
stored in the database itself, but it can be set again at any time by running:
(triples-backups-setup ekg-db ekg-default-num-backups ekg-default-backups-strategy)The strategy can be one of the defaults of daily, weekly, every-change, or
never, and new methods can be defined as well. See the implementation in
triples-backups.el for more information.
You may occasionally notice that certain tags are obsolete and have no notes, or
notes exist that are empty, or various other annoyances. You can call
ekg-clean-db, which will:
- First, force a backup.
- Remove all tags with no uses.
- Remove notes with no text, or just a “*”, which is something that often happens with org-mode buffers.
Tags may need to be renamed because the concept has changed in some way. The
command ekg-global-rename-tag can quickly rename one tag to another globally
across the database, so all tags with the old tag now have the new tag. (Note
that the new tag may already exist, in which case this operation cannot be
easily undone.)
You can customize the behavior of ekg in a number of ways.
First, you can create your own schema to store your own data. The hook
ekg-add-schema-hook is called whenver the database is connected to. At that
point, ekg adds all of its schema, and runs the hooks in this variable. Adding
schema is idempotent, so it can be called any number of times without causing
problems. Adding schema can be done by calling the triples library. For
details on how to create schema, you can either look at the ekg implementation
for example, or the triples library README for an overview of how it works.
The ekg-note-pre-save-hook is called before saving a note, and
ekg-note-save-hook is called after saving, but in the same database
transaction as the save.
The ekg-note-delete-hook is called when deleting a note.
The ekg-note-add-tag-hook is called when adding a tag, either via the initial
tags added to a new note, or tags added after completing a new tag in the note’s
property list.
The ekg package is designed to be easy to integrate with. For example, if you want to create a note automatically in one of your functions, you can write:
(defun my/log-to-ekg (text)
"Log TEXT as a note to EKG's date"
(ekg-save-note (ekg-note-create :text text :mode 'text-mode :tags `(,(ekg-tag-for-date) "log"))))If you wanted to re-use an existing note and append to it, you can do that as well.
(defun my/log-to-ekg (text)
"Log TEXT as a note to EKG's date, appending if possible."
(let ((notes (ekg-get-notes-with-tags (list (ekg-tag-for-date) "log"))))
(if notes
(progn
(setf (ekg-note-text (car notes)) (concat (ekg-note-text (car notes)) "\n" text))
(ekg-save-note (car notes)))
(ekg-save-note (ekg-note-create :text text :mode 'text-mode :tags `(,(ekg-tag-for-date) "log"))))))There isn’t a special API, but the basic defuns such as ekg-save-note,
ekg-note-create-text, ekg-get-notes-with-tags, ekg-get-note-with-id, along
with the struct ekg-note are good starting points. Capturing notes in
different ways can be done by wrapping ekg-capture, and is how functions such
as ekg-capture-url work.
If you add schema and you want the user to be able to modify it, you should
supply new alist entries to ekg-metadata-parsers and ekg-metadata-labels.
Because inline commands exist, the complete text of a note should be retrieved
with ekg-display-note. The function ekg-note-text, will only get the text
as stored, which is missing mode related text properties and any text generated
from inline commands.
The ekg module can have any number of functionality additions. These may appear as other packages with other maintainers, but some are included as part of this package.
The embeddings functionality, for integration with an LLM, can be turned on by requiring the ekg-embeddings file and enabling it, such as:
(require 'ekg-embedding)
(ekg-embedding-generate-on-save)This module contains functionality to explore similar notes and search using techniques associated with large language models. Embeddings let you do searches at a semantic level, based on an understood meaning that is separate from the words used. For example, if I have a note with a recipe for linguini, embeddings will let me see that it is similar to notes about spaghetti, and not similar to notes about cold fusion. Because the search is not based on words, but meaning derived from those words, notes that describe the same thing in two different languages should be very similar. In ekg these let you find notes similar to a current note, or in fact any buffer. You can also do a query via embeddings.
The idea behind an embedding is that it is an abstract representation of text, represented as a multi-dimensional vector. Because it is just a vector, you can compare the distance between different embeddings, and embedding vectors that are similar should represent similar concepts. This can be used to find similar notes, but also to search, where the search string is transformed into an embedding.
The embedding interfaces with your preferred LLM provider via the llm package.
This package allows the user to define their preferred llm backends, which will
be stored in ekg-llm-provider. Please see the LLM module project page for a
complete description on how to do this, but an example would be the following:
(use-package ekg
:init
(require 'llm-openai) ;; the specific LLM provider must be required
(require 'ekg-embedding)
(ekg-embedding-generate-on-save)
(let ((my-provider (make-llm-openai :key "my-openai-api-key")))
(setq ekg-llm-provider my-provider
ekg-embedding-provider my-provider)))The embedding provider must be the same for all notes. If you want to switch to
a new provider, you will need to call ekg-embedding-generate-all with a prefix
argument (C-u M-x ekg-embedding-generate-all), which will regenerate all
embeddings asynchronously. The embedding provider does not have to be the same
as the LLM provider (if you also use the LLM add-on.) Also note that the
provider will get the text of all your notes, so if that bothers you, do not use
any provider on a server.
Once you have this set up, and you have already called (require
'ekg-embedding) and (ekg-embedding-generate-on-save) you can call M-x
ekg-embedding-generate-all. This may take a long time as each embedding has to
be generated separately with its own API call. Once you’ve done this, you can
call, in ekg-notes-mode, ekg-embedding-show-similar to get a list of similar
notes. You can also call ekg-embedding-search to perform a search over your
notes using embeddings. In any buffer, you can call
ekg-embedding-show-similar-to-current-buffer to show notes similar to whatever
the text is in the current buffer.
The variable ekg-embedding-text-selector has a function that will pre-process
all text that is sent for embeddings. The default value is
ekg-embedding-text-selector-initial, which will estimate the size of the
tokens sent and limit the text to the first 8k tokens. Right now the function
is tuned to the limits of Open AI’s embedding framework, and a different
function may be needed for other embedding APIs.
If you would like to stop generating embeddings for notes in a session, you can
call (ekg-embedding-disable-generate-on-save).
There are two ways to use logseq with ekg. One is maintaining logseq as an export-only copy of ekg data, where you don’t plan to modify anything in logseq, just using it to access your notes on other platforms. Exporting from ekg is destructive, though, so without an initial import, exporting will overwrite logseq files with data from ekg, so it may destroy data. The other way is to sync bidirectionally. This starts by importing anything from logseq that has never been imported before, and then writing ekg’s data on top. This will preserve data, but will lose the initial ordering of pages. Both of these methods, then, will significantly impact your logseq notes. It is highly advised to back up your logseq files before starting.
To export to logseq, start by requiring the ekg-logseq module and setting up
ekg-logseq-dir, which points to the base directory of your logseq data (where
there is a “pages” and “journals” directory):
(require 'ekg-logseq)
(setq ekg-logseq-dir "~/my/logseq")If you wish to maintain logseq as a read-only copy of ekg, just run
ekg-logseq-export when you wish to export data. This currently may take a few
seconds to a minute, depending on how much data you have. We attempt to not
write any files that are unchanged. To have a bidirectional synchronization,
run ekg-logseq-sync, which will first import data from logseq, then export
data.
When exporting, it’s important to understand the differences between ekg and logseq. Logseq is organized by pages, where one page is one file. Within the page there are many sections, which can be individually referenced. The ekg integration treats logseq pages like ekg tags, and logseq sections like ekg notes. In logseq, the user mostly sees one page at a time. In ekg, notes are shown in a variety of contexts, mostly tag related, but not always. In logseq, a note lives in one page and is referenced from other pages. In ekg, each note has its own identity and is tied to other notes solely via the tags it shares with them. To compensate for this difference, we export notes based on their first non-date tag as the page where the text will apear, and reference other tags, where they will appear as backlinks. In addition, in org-mode, notes in a page appear as top-level outlines, which are supposed to have text for the outline node. If there is an ekg note with a title, the title will appear as the text, otherwise the outline node will just read “Untitled note”. Because this initial headline is where various properties are stored, and is followed immediately by tags, it makes sense that this is a title instead of just part of the content.
For example, take the following note:
Tags: date/2023-04-05, ekg, logseq ekg can export into logseq!
This will be exported into “pages/ekg.org”:
#+title: ekg * Untitled note :PROPERTIES: :ID: 33134561605 :EKG_HASH: 89471eadbd7cc56b088f5513c11f68cb1d11d045 :END: #[[2023-04-05]] #[[logseq]] ekg can export into logseq
Each node points to its ID which is from ekg (but, if it was originally imported, the ekg ID might originally be from logseq). We also encode the hash of the exported data. This is to keep track of what was exported, so we do not re-import it unless it has changed. For now, even if the data is changed, it is not re-imported. Files for “pages/logseq.org” and “journals/2023-04-05” will also be created, although they won’t have any content from this note.
When exporting, inline commands (see inlines section), are evaluated before exporting to logseq, with the exception of note transclusions, which turn into logseq embeds to the same ID. So, other kinds of transclusions or any other commands will evaluate to whatever text they normally evaluate to when viewing the note before exporting to logseq. For example, if the note has a file tranclusion inline command, the file contents will be exported to logseq. Logseq embeds are roughly equivalent to note transclusions, but only roughly, since a key difference is that logseq embeds occupy their own lines and appear visually distinct, and ekg transclusions don’t. Because of this, some formatting strangeness between the two may happen.
Imports from logseq will return all top-level items as separate notes. So, for example, assuming we’re reading from the logseq file “pages/logseq.org”:
* This is my first time trying logseq #testing * The org compatibility here is especially nice #org It really helps me feel comfortable in using the various formatting options I had gotten used to.
This will turn into two notes, one that has text “* This is my first time trying
logseq #testing”, and with tags logseq, and testing, and the other with the
rest of the text, with the tags logseq and org.
There are a few things to be aware of. In logseq, any level of the hierarchy can have an id and be referenced separately. In ekg, we don’t support notes inside of other notes, so these will be imported in the context of the parent note, and won’t be available to reference as its own separate note. Also, logseq has other functionality not supported by ekg, such as queries and potentially anything provided by plugins. These will be imported as-is to ekg, but without any corresponding functionality.
Logseq embeds are imported as note transclusions.
The ekg-llm module provides a second way to use large language models (LLMs) with ekg, separately from the ekg-embeddings integration. While the ekg-embeddings module lets you find notes based on their meanings, the ekg-llm module lets you prompt an LLM with the contents of a note, and then capture the LLM’s response in the note.
As with ekg-embeddings, this is based on the llm package, which allows the
user to define their preferred llm backends, which will be stored in
ekg-llm-provider. Please see the LLM module project page for a complete
description of how to do this, but an example would be the following:
(use-package ekg
:init
(require 'llm-openai) ;; the specific LLM provider must be required
(let ((my-provider (make-llm-openai :key "my-openai-api-key")))
(setq ekg-llm-provider my-provider
ekg-embedding-provider my-provider)))The embedding and LLM providers can be different. The LLM provider can change at will, while the embedding provider must be the same for all embeddings in the database. It is necessary to create both of these providers, because some LLM functionality depends on having embeddings.
To send a note to an LLM and capture its response, call
ekg-llm-send-and-append-note, which is by default bound to [C-c .]. A
prefix argument ([C-u C-c .]) will let you edit the prompt before it is sent.
The output from the LLM is appended at the end of the note, in a special
section.
In addition to the contents of the note, ekg will construct a larger prompt for
the LLM. The prompt consists of context about previous notes that contain the
tags of your note, and similar notes, which is what will generate high-quality
content that is appropriate in the context of your notes. It also contains
instructions to the LLM to how to generate the note text to be added or
replaced. The default instructions are a fixed string you can configure in
ekg-llm-default-instructions. But alternatively, you can create alternative
prompts for different ekg tags in the same way that templates work, by creating
a note tagged with “prompt” and any other tag (the special “prompt” tag can be
changed by customizing ekg-llm-prompt-tag). The alternate prompt is created by
appending all “prompt”-tagged notes. Note that, as with templates, hierarchical
tags can have prompts attached at any or all levels of the hierarchy.
To take an example, imagine that you have a note tagged with prompt and recipe,
containing instructions saying the LLM should imagine itself an authority on
cooking and provide you helpful tips to improve your recipes. You then create a
note with a child tag of recipe, let’s say recipe/monkfish, with some details of
your attempt to cook monkfish, and then hit [C-c .]. Because recipe is a parent
of recipe/monkfish, ekg-llm will use these instructions instead of the default
one, and will also append your note, and place the LLM’s response in a special
section at the end of your note. For example:
Making monkfish again. It is thick but tends to be wet and hard to get a good sear on. Maybe I should sous vide it and then blast it with the searzall torch? Monkfish can indeed be challenging to sear properly due to its high water content. Sous-vide cooking followed by searing with a torch can be a great technique to achieve the desired result. The sous-vide process will help to cook the fish evenly and retain moisture, while the searzall torch can give it a beautiful caramelized crust. Just be sure to pat the fish dry before searing for better browning and use high heat to quickly sear the exterior without overcooking the inside. Happy cooking!
Instead of appending, the note can be replaced with the output of the LLM by
using ekg-llm-send-and-replace-note which is bound to [C-c ,]. As with the
append command, using a prefix argument will let you edit the instructions
before sending it.
All prompts sent from a note in org or markdown modes have a prelude that notes the format of the input and expected output. However, LLMs typically will produce markdown regardless of what you ask it to do, so if you want to use LLMs, you may want to use markdown as a default note format.
ekg notes are especially well suited for LLM prompting, both because of the ability to create prompts for different tags, and the ability to transclude one note’s contents within another note. While each “prompt”-tagged note should work as a standalone LLM prompt, it may be helpful to build up a set of partial prompts that you can share among many full prompts using transclusion.
For example, imagine a prompt that is designed to give an Aristotelian response to a note. A note with tags “aristotle” and “prompt” could have the basics: “You are Aristotle. Give a response to the note using Aristotle’s writing style and ideas, referencing existing works when possible.” But perhaps you also want some standard behaviors found in other prompts, such as a prompt to encourage the LLM to ask you questions when appropriate. There may be many prompts in which that sub-prompt may be applicable. You can use transclusion inlines for this, adding the transclusion to the appropriate part of the prompt. You can then iterate on each sub-part, trying to get the best behavior.
Additionally, transclusion or other inline commands could help in other ways in forming the prompt, by sharing your schedule, or your current org agenda items as context to the LLM when it is necessary. These advanced uses will require inline commands that are not part of the base ekg package, but once written, they can be seamlessly used in prompts.
If you also use embeddings, you can use the interactive function
ekg-llm-query-with-notes to find your notes that best match a query, and send
the LLM a prompt consisting of those notes. This essentially will let your
notes act as a natural language queryable knowledge base. It will work for
queries in which you have the relevant information. The answer to the query
will appear in a new buffer.
The initial part of the prompt instructing the LLM for this case is defined in
ekg-llm-query-prompt-intro. This can be changed to tune how the LLM responds
to the query.
Note that anything in your database could potentially be retrieved and sent to the LLM, so if you have notes that you consider too private to send for processing, you may not want to use this.
Theekg-auto-save module is useful for users who enter longer notes, so that the notes are protected against accidentally killing the buffer, or emacs crashing, or any similar problem. It is designed to work similarly to the built-in auto-save functionality, and has it’s own variables that default to the auto-save equivalent. So, for example, there is ekg-auto-save-timeout, which defaults to the value of auto-save-timeout.
To start using this, you need to require the module and turn on ekg-auto-save-mode in the ekg-edit-mode` and =ekg-capture-mode. For example:
(require 'ekg-auto-save)
(add-hook 'ekg-capture-mode-hook 'ekg-auto-save-mode)
(add-hook 'ekg-edit-mode-hook 'ekg-auto-save-mode)In the capture mode, this. is equivalent to saving periodically (to drafts). In edit mode, it will save the latest version while editing.
ekg can export notes as denotes. Denote is a note taking and file naming tool. Primary reason for export is taking backup of your notes in a git backed repository. Import is in road map (PR is welcome). To export to denote,ekg-denote module is required.
(use-package ekg-denote
:init
(setq ekg-denote-export-add-front-matter nil)
(setq ekg-denote-export-backup-on-conflict t)
(setq ekg-denote-export-title-max-len 50)
(setq ekg-denote-export-combined-keywords-len 150))To export, call ekg-denote-export which will export any modified
note after the last export as a denote in the denote-directory
defined by denote package. If it is your first time exporting, all
the notes will be exported to the denote-directory.
ekg-denote keeps record of the last export time in the ekg db and
use that to find the notes changed since last export. This way the
exports are much faster. It is suggested to export your notes every
day.
User can optionally enable adding of denote front-matter to exported
denotes by setting ekg-denote-export-add-front-matter to
t. Denote front matter is added using denote-add-front-matter
function defined by denote package which open note in an emacs
buffer and requires manual execution of save-buffer by the user.
Ekg and denote have differences, due to which following customization are made available:
ekg-denote-export-title-max-lento trim the title during export. Default is 50 characters.ekg-denote-export-combined-keywords-lento trim the combined length of keywords. Default is 150 characters.ekg-denote-export-backup-on-conflictto backup the denote if both ekg and denote are found to be updated after last export. Default ist.
It is user responsibility to backup the denotes before and after export to protect against accidental deletes. This can be easily done by keeping denotes in a git repository and making sure to check-in any changes before and after export.
If ekg and denote are both found to be updated after the last export
which should ideally not happen, denote is updated with ekg. A backup
is taken based on ekg-denote-export-backup-on-conflict setting.
The ekg package uses the triples package to interface with a sqlite database.
The reason a database is useful, even for text, is because databases are
extremely fast, very flexible, and extremely easy to change. In general, the
less your data looks like just files with text in them, the more a database
makes sense. In ekg, we can separate the notion of tags from the text, which
makes writing functions such as ekg-global-rename-tag trivial, and the
execution extremely fast.
The decision to use the triples package, though, is related to a different
design choice. In a triple-based system, there’s only one database table with
four columns, a subject, predicate, object, and properties. One way to
think of this schema is that it defines links of different types from a subject
to an object. This is combined with a schema, itself defined in triples. The
triples define that subjects can have types, and those types can have
properties. Those properties are expressed in this triple format. In ekg, the
subjects correspond to the IDs of the notes, or tags. Subjects can have
multiple types, and data is factored into types that belong together, with a
specific meaning. To give an example, listing out the data for a note might
look something like:
33204698034|base/type|tagged|() 33204698034|tagged/tag|"date/2022-11-06"|(:index 0) 33204698034|tagged/tag|"lentil stew"|(:index 1) 33204698034|base/type|text|() 33204698034|text/text|"Made a great lentil stew with dried porcini mushrooms and delicata squash." 33204698034|text/mode|org-mode|() 33204698034|base/type|time-tracked|() 33204698034|time-tracked/creation-time|1667787928|() 33204698034|time-tracked/modified-time|1667787986|()
In this example, 33204698034 is the ID for this note. It has a type
(base/type), of tagged, which means this is something that has tags. The
tags are a list, so the properties contain their index in the list. Because
each one is stored individually, we can easily find all entities with each tag,
by querying on all subjects with a particular object value. This is how reverse
links work in the triples package. In this case, there are two tags,
“date/2022-11-06”, and “lentil stew”. The note comes from another type, text.
And yet another important property, the modification time, is on yet another
type, time-tracked. These are all independent. It is possible to have
subjects that have tags but not text, although this doesn’t happen currently in
ekg. It’s also possible to have any object have a creation and modified time.
Using a triples scheme has the advantage that it is very easy to integrate with. All data is very “flat”, without having to worry about tables and their schemas. The uniformity means that it lends itself well to integrations, which typically would provide a new type and new data. The disadvantage is that it is typically less efficient to query, at least for more complicated queries. On databases that typically will be used with ekg, this should be not noticeable.
IDs, stored as subjects, can be resources. This is useful when we want to store
data about some unique thing, such as an URL. Because triples define a graph,
every object can be a subject. For an example, if some data in the graph has a
value of “http://emacs.org”, then we can attach more data to that value, such as
tags, notes or anything else. This is how we store notes about web pages
(ekg-capture-url). Having IDs that are meaningful is also useful to enforce
unique data, and force that data isn’t duplicated. For example, with this
design, you couldn’t have both a “tag” entity and a “page” entity that are
separate: if they are the same object value, they will be the same subject, with
the same ID. This leads, in our opinion, to a better design. Also it’s useful
to note that IDs can be anything, even different types of objects. Integers,
strings, symbols. This is useful, because objects can be anything. Because of
the design of the triple database, all data can be expanded on with their own
data, and that data itself expanded on. This seems like a useful property to
have for a personal knowledge system.
Because the user may want to modify or create both the text and other database
properties at the same time, we use a single buffer that lets the user do both.
Because of this design choice, we have to divide the buffer up into two
sections: a metadata section and the text section. The metadata section is on
top, and has a specific format. Because of this, some org-mode functions may
not work correctly, because they assume the whole buffer is an org-mode file.
Without this design, however, it isn’t clear how the user could easily see and
modify everything they need. Theoretically, having another window might work,
but this would add other complications: the user might not want several windows,
the user might select or bury one of them, and more. There isn’t an obvious
ideal solution. It’s possible that the design of the capture/edit buffer may
change in the future to fix some of the issues we see with the current
implementation.