Import / Export Commands

[chrisgrieser]

Some plugins do allow you to import or export settings. This would be incredibly useful for this plugin, too, so the more shell-adept member of the community can write small scripts other's can use.

One straightforward example for that could be using shell commands as a replacement for the Pandoc plugin, many writers are dearly missing.

[eleanorkonik]

Is the idea behind this to let this plugin serve a similar role as templater, where more advanced users could create code that others could easily use, and build out a sort of automation ecosystem?

She asks, knowing remarkably little about shell scripting but being willing to learn...

[chrisgrieser]

Basically, yes. with some additions to this plugin (the PATH issue, import/export, and maybe the possibility to have multi-line shell commands), this plugin is remarkably well suited to do what you think, and even much more.

As opposed to Templater, it gives you access to shell commands, which means "everything you can run in the Terminal, you can use in Obsidian now". Markdownlint? use it in Obsidian. Pandoc? use it in Obsidian. You remember pdfgrep (the command line tool I mentioned that can can search in PDF)? it can basically put it's output straight into a note. I am honestly surprised this plugin got so little attention for what it can (and I only stumbled upon it recently).

[FelipeRearden]

And Apple Scripts as well :)

[Taitava]

I wrote more on the actual subject in below, but I'll answer here just a couple of thins.

---

@eleanorkonik

Is the idea behind this to let this plugin serve a similar role as templater, where more advanced users could create code that others could easily use, and build out a sort of automation ecosystem?

I agree with @chrisgrieser, this would allow newcomers to more easily adapt shell command examples from advanced users. Shell commands are not an easy thing to learn, especially if the user does not have a programming background. But still, learning shell commands can be useful when working with files and directories. Knowing all about shell commands should not be a requirement to use this plugin. This plugin should not be aimed for advanced users only, but I'm afraid it's currently quite technical to use for users who are not familiar with all the terminology related to shell commands.

@chrisgrieser

maybe the possibility to have multi-line shell commands

Would you like to create a new discussion about this, as it's a good idea too, and worth planning? I'd like to hear more about your thoughts on this. 🙂 (Edit: You actually did it 🙂 )

@chrisgrieser

I am honestly surprised this plugin got so little attention for what it can (and I only stumbled upon it recently).

Again, glad to hear this kind of feedback! 🙂 Well, this plugin is recently new. The first version came out in August 2021, so it's just half a year. This is my first plugin for Obsidian (and so far the only one) and I'm not very known in the community (and not active in Discord at all), so I guess it explains why this plugin is not so visible in the community.

I'm also lacking something: Ideas on how to use this plugin in real life. I've implemented a few things with this myself, but as they are mostly related to Git, they are not something I can use to wake up the interest of other Obsidian users. Coming up with examples is hard for me.

[Taitava]

Thank you for the suggestion! 🙂

It's nice to see people thinking about features that have crossed my mind. For me, this has been a maybe sometime in the future thing, which I did not want to plan too much ahead before seeing a clear need for it. Now that you suggested it, I can see that maybe it is beneficial.

In addition to sharing shell commands with other people, I can see myself using it to copy them between my different vaults. Different vaults use partly different shell commands, so I don't want to just copy (or sync) the whole settings file between my vaults. But certain shell commands (mostly Git related) are something I use in all my vaults.

I'll divide my so-far thoughts about this feature under the following headings:

Concerns

Safety

Shell commands are dangerous. That's a fact that is obvious to all of us. Nobody should copy and paste these from the internet without knowing every detail of the shell command that they just copied from somewhere. The recent release of 0.10.0 brought events (#123) to SC, which is a nice addition, but also increases the dangerousness of this plugin. Imagine a user importing a shell command that is executed e.g. every time Obsidian starts, or every one second. The command would do some write operations in the filesystem, e.g. delete files or overwrite them. Or do other nasty things.

I like to think that people in the Obsidian community are truthful and kind to each other (which they usually are!), but there's always the risk that someone publishes badly behaving shell commands, either intentionally or by accident.

The plugin cannot safeguard users from importing dangerous shell commands. No program can determine with 100% accuracy if a shell command is safe or not. But what the plugin can do, is to try to show as clearly as possible, what features a shell command will use, if it is imported. E.g. if the shell command enables some events, display a warning for each enabled event. Warn also if the shell command's output channel is set to write into a file. These warnings would be shown immediately after a user pastes a shell command into an importing text field, before the shell command is saved and taken into use.

Still the shell command itself can be dangerous. For this, there needs to be a checkbox like I confirm that I have thoroughly inspected this shell command, and I know exactly what it does.

How global settings affect individual shell commands

Another concern is the fact that global settings that affect the execution of shell commands, should be dealt with somehow. E.g. Working directory setting and Default shell settings. When you export a single shell command, you need to take into account that it relies on these global settings, too. I have not come up with a solution for this yet.

Practical design decisions

The exported shell commands will be presented in a long textual format, that will contain a lot of features. It may happen that a user does not notice some dangerous details in the shell command. If you open up a data.json file stored in the plugin's folder (this repository does not contain the file), you'll see that a single shell command is quite complex in the file. An example shell command (SC version 0.10.0):

"51": {
  "platform_specific_commands": {
    "default": "echo \"SOMETEXT\""
  },
  "shells": {},
  "alias": "Event: After Obsidian starts",
  "confirm_execution": false,
  "ignore_error_codes": [],
  "output_channels": {
    "stdout": "current-file-caret",
    "stderr": "notification"
  },
  "output_channel_order": "stdout-first",
  "events": {
    "on-layout-ready": {
      "enabled": true
    },
    "every-n-seconds": {
      "enabled": true,
      "seconds": 60
    }
  },
  "command_palette_availability": "enabled"
}

This is just a single shell command stored in the settings file. This example is actually quite dummy, it inserts the text SOMETEXT into current file (caret position) every 60 seconds and every time a users switches between panes. These things might not be clear to a user who is not familiar with this format. And I guess not many users of this plugin actually open up the settings file to see how their shell commands look like there.

Export will use YAML instead of JSON

I can't make the above example 100% easy to read, but I can at least improve the readability a bit. The settings file is stored in JSON format, which is good enough for data written and read by computers, but not good to be read by humans. So I've decided to use YAML for exporting. Here's the same example converted to YAML:

51:
  platform_specific_commands:
    default": "echo \"SOMETEXT\""
  shells: []
  alias: "Event: After Obsidian starts"
  confirm_execution: false
  ignore_error_codes: []
  output_channels:
    stdout: current-file-caret
    stderr: notification
  output_channel_order: stdout-first
  events:
    on-layout-ready:
      enabled: true
    every-n-seconds:
      enabled: true
      seconds: 60
  command_palette_availability: enabled

ID should be random instead of sequential

Currently, shell command ids are just sequential numbers starting from 1 0 (corrected on 2022-08-19). I want to change them to alphanumeric random strings, something like 5 characters long (or longer), to avoid the chance to overlap with existing shell command ids when importing. Existing shell commands that now have a numeric id, will keep the same id without changes, as possible hotkeys are dependent on the id.

An exported shell command needs to have an id present, because I want to be able to notice a situation where a user tries to import a shell command they already have in their vault. In this case, SC can ask Do you really want to import this shell command again? The user could select to import a duplicate shell command, or to replace the old shell command. The latter makes sense if the user has edited the command in one vault, and is now bringing the updated command to another vault.

Support commenting each line (added 2022-08-18)

YAML supports adding comments by using the # character. These comments won't do any practical changes in the configuration, but they could provide human-readable explanations on why certain settings are defined as they are. An example:

60:
  platform_specific_commands:
    default": "/bin/bash my-script.sh {{file_name}}"
  ...
  events:
    on-layout-ready:
      enabled: true # This command needs to execute every time the user switches the active pane because it needs to reflect the changed current file name in a file that it writes to.
    every-n-seconds: # This command also needs to execute every minute because blah blah blah...
      enabled: true
      seconds: 60 # I decided to use 60 seconds as the value because it's well balanced between not running the heavy process too often, and not making the user to wait for too long between cycles.
  ...

These comments could be shown to the user at the time of importing, but they would disappear after importing - they would not be saved in the actual data.json settings file, because JSON does not support comments.

Future-proof

SC version should be present in the export

The shell command format changes every now and then. New fields get added, and old fields get modified or sometimes even removed. The exports need to have version numbers available, so that the import process knows if it supports the shell command or not. An old version of SC cannot read shell commands exported from a newer SC. A new version of SC can be able to migrate shell commands exported in an old format - to a certain extent.

All these migrations require some migration code to be in place in the plugin. In the future, there might be a situation where some migration code is really ancient, and only rarely needed in practise, if ever. There might come a time when it's better for the sake of code cleanness to remove some age-old migration code. This can happen when a major version of SC changes, e.g. from 0.x.y to 1.0.0, or from 1.x.y to 2.0.0.

Implement this feature when the plugin is mature

To avoid a large amount of migration code needed to be preserved, I think it's best to first develop this plugin into maturity, i.e. to the version 1.0.0. Then it should be in a state where changes to the format will not be made as frequently as now.

Migration code is in place already now, as old settings files do exist. But it has a difference: When users upgrade their plugins, their settings file gets migrated to a new format. But if there are exported shell commands out in the wild world, nothing will upgrade their format in the place where they reside. That's why I'd like the first exported shell commands to be born in a form that is mature enough to be able to live for years to come.

---

Any feedback on these thoughs? 🙂

[chrisgrieser]

regarding the safety concerns, I would maybe suggest a warning not only for specific events, but also for specific contents, e.g. an important containing the rm command will always trigger a message like "this shell command contains a command removing something"

[Taitava]

Yea, and > / >> things, too. Finding a list of dangerous commands can be a big task.

I just hope it doesn't turn out into something people interpret in the wrong way: Hey, the plugin did not give any warnings, so I assume the shell command I'm importing is totally 100% safe!

But I do agree that this is a good idea.

[Taitava]

Actually, it might be hard to implement. SC would need to interpret the shell command somehow to know if e.g. rm appears as a command or as a literal string argument.

[Taitava]

A place for sharing shell commands?

With the ability to export shell commands in text format, it's obvious that shell commands could be shared wherever users collaborate with each other: Obsidian forum, Discord, GitHub, Obsidian Publish sites, blogs etc. Those are all good platforms, but I became to think that would there be any benefit from some kind of centralized library for shell commands that would serve in addition to those?

I'm thinking of some kind of a website where users could login (maybe with GitHub login) and submit their exported shell commands under different categories.

It's just a far-future idea, and nothing certain. 🙂

[chrisgrieser]

I think a dedicated website may be a bit overkill (for now at least), how about a folder in the GitHub repo of this plugin, and people can simply make PRs?

[Taitava]

Yea, that might be a better idea. Edit 2022-02-19: It would probably be a separate repo.

[catfist]

I can see myself using it to copy them between my different vaults.

Yes, me too.
So, I made PowerShell & bash command to exports all SC-commands to Obsidian note automatically.
jq format json to copy & paste easily.

Windows:

$listfile = './Obsidian Shell commands list win.md'
Write-Output '```json' > $listfile
Get-Content "{{vault_path}}\.obsidian\plugins\obsidian-shellcommands\data.json" | jq >> $listfile
Write-Output '```' >> $listfile

Mac:

listfile='./Obsidian Shell commands list mac.md'
echo '```json' > "$listfile"
cat "{{!vault_path}}/.obsidian/plugins/obsidian-shellcommands/data.json" | jq >> "$listfile"
echo '```' >> "$listfile"

[Taitava]

Nice! 🌞

[Taitava]

Examples of other plugins with import/export functionality

@chrisgrieser I'm coming back to this old comment of yours:

Some plugins do allow you to import or export settings.

Can you provide me some examples of what plugins have this feature, please? 🙂 I haven't used too many plugins myself. I'd like to have a look on what kind of data format(s) they use (e.g. YAML, JSON, or something else).

[chrisgrieser]

style settings and dynamic highlighter do that. they both export to JSON, iirc

[Taitava]

Thanks! 🙂 Wow, that was fast! 🐎