Pass variables to stdin (was: Option to pass editor selection to command’s stdin)

[dathanb]

I’d really like to be able to pipe the editor’s current selection to a command on its stdin fd. I think that option should be a toggle in the per-command settings modal.

[dathanb]

I’m happy to make a PR for that - I have it written already, actually, just working through some edge case handling - but wanted to get feedback on the best user experience for it before finishing up and submitting it.

[Taitava]

Thank you for your idea! I converted your issue to a discussion, as I'd like to first plan this kind of thing a bit more.

I have had stdin in my mind, but so far I haven't implemented it in SC because I'm not sure if it would bring actually any new functionality. If you have a command command that uses stdin, you can currently do echo "{{selection}}" | command, which will pipe the selected text to your command's stdin. Please let me know if there would be more benefits for having explicit support for stdin that I maybe haven't noticed.

Also, if stdin support would be added, I think a user should be able to pick whatever {{variable}} they want. Maybe it could even be a multirow text field, where a user could write whatever text they want, and include as many {{variables}} as they need.

[dathanb]

Great idea about populating the stdin from any variable! Or even a template string with the potential to expand multiple variables.

The two main benefits I see to stdin vs the existing variable-expansion support are avoiding escaping (potentially simplifies logic, plus safer because it avoids any bugs that might exist in the escaping logic) and command line length limits. The latter isn't relevant most of the time, but on my Mac the shell command line maxes out at 1MB, and historically Windows has had a much shorter command line limit (caveat: I don't often use a Windows machine these days, so I'm not sure if that's still the case).

[Taitava]

Solid reasons! 🙂 Escaping is a pain at the moment (well, there is no escaping at the moment 🙁 but it's coming). I didn't know about a length limit for shell commands.

I'll come up with a plan for this - I'll probably create a screenshot/mockup that shows the UI for stdin. Currently #76 and #78 are under development and those are changing the parts of code that will affect this stdin feature too, so it's better to wait until those are done before implementing this. I think this could be done in version 0.8.0.

[Taitava]

Damn, when I converted issue #88 to this discussion, I thought that I would be able to later continue using that issue to add a comment there which would contain a summary/plan for the feature, but there's no way to reopen the commenting part for it now that I did this conversion. I need to create a new issue, but I'll do it when I have the plan ready.

My idea for discussions and issues is that discussions can be used for things that are still uncertain, and an issue will have a condensed summary of the discussion, containing things that are decided to be implemented. Issues are also used in Roadmap, because discussions cannot be included in it.

[Taitava]

Just came to my mind, that while stdin will give us a nice way to pass variable values safely into a command/script, it does have a limitation that it's just one channel, so if a user wants to submit multiple variable values to their command/script, they will be concatenated together in stdin. I know this is obvious, and not in any way a thing against stdin.

What I'm thinking further, is the ability to define extra input channels in addition to stdin. Maybe not at the same time with the basic stdin support, but later. I have never used any other channels than stdin/stdout/stderr, but if I have understood correctly, it should be possible to define extra channels in shells. So there could be even extra output channels, in case stdout and stderr are not enough.

But basic stdin support will be the first target.

[dathanb]

I'm not familiar with making new file descriptors, but you can make named or anonymous pipes easily enough and use those as channels. Maybe name them with some convention like input_1, input_2, output_1, output_2, etc, and then provide the shell with an environment variable for the number of additional input and output pipes? The three stdio streams are the only ones that a script can use implicity, AFAIK, so defining extra channels requires some sort of shared protocol that both obsidian-shellcommands and the invoked script understand, and the sequentially-numbered-with-env-var scheme is the simplest one I can think of.

However, I've played with your suggestion of echo "{{selection}}" | my-script, and it's actually working great for me other than escaping issues. So it feels like the right order of work is to handle escaping first, and then re-evaluate what use cases aren't handled by that, and then target those specific use cases when implementing stdin (and maybe additional pipes) support later?

[Taitava]

0.7.0 has been out for a few days now. While there's still not so much experience on how well the escaping system works, I think I'll already start to look into how this feature can be implemented.

[Taitava]

I had a look on Node's different child_process execution methods. Currently, .exec() is used for executing shell commands, but it doesn't support defining stdin (which I knew already), so the method needs to be switched to something else.

A few things I'm concerned about when switching the method:

• The main and child processes should stay detached, i.e. Obisdian don't have to wait for the executed commands to quit before it's able to quit, and the executed commands should not quit when closing Obsidian.
• Should output be buffered or streamed? Currently, it's buffered, which is convenient so that SC can handle the output at one time after the child process has exited. But if I'll ever implement Real time stdout handling #64, it would require streamed output. Converting SC to use streamed output is doable, but involves more work, and it's not related to this stdin feature. But if .exec() needs to be changed for something else, I'd like to find a method that has an option for both styles: buffered and streamed.

I’m happy to make a PR for that - I have it written already, actually, just working through some edge case handling - but wanted to get feedback on the best user experience for it before finishing up and submitting it.

@dathanb As I'm not very experienced with these different execution methods, I'd like to ask your opinion, what method can be used?

• child_process.execSync() seems to be synchronous, so I guess it cannot be used for streaming output? (This question is more related to Real time stdout handling #64, like mentioned above). But it would support stdin.
• child_process.spawn() looks like it takes a command and a set of arguments separatedly, but can it be used so that the command string would contain also arguments, and the actual args array would be left empty? Of course, shell option needs to be defined, so that .spawn() would execute the command via a shell, and not directly.

As you mentioned that you have already done some code, what method did you use, and would you like to post a sample code? (Does not need to be a PR, just a draft preview of what you have done, and regardless of which SC version it is for).

I can see two possible plans for this:

• I can do some trial and error with different methods, and eventually do the whole thing myself at some point.
• We can do co'operation so that you can implement changing the method to something that supports stdin and also has the ability to later change buffering to streaming. You can use whatever way to submit text to the stdin - be it a hardcoded string, or hardcoded {{selection}} variable or something else. I will later build the UI, including a multirow text field with a possibility to include multiple variables along the text.

How does this sound to you? The timetable is totally up to you, can be started whenever or postponed to the far future. I don't have any kind of timetable for this.

[dathanb]

I'm not a Node expert by any means, so I don't have any insight into the various child process options available. Synchronous would be easier to work with, but like you said, that likely means no support for streaming handling of output.

I pushed a PR with an example of how I approached the implementation previously. No support for streaming - I didn't think about that use case. I used spawn, though I didn't have a strong rationale for that - just picked one of the options and ran with it.

The biggest issue I've encountered is that if stdin ends up being empty, spawn will invoke the script with no stdin file handle, and I never figured out how to get around that. stdin not existing isn't too hard to handle in a script, but it'd be unfortunate if scripts had to handle that case in order to be usable from obsidian-shellcommands.

[Taitava]

Thanks a lot! 🙂

I glanced throught the PR quickly, but I'll check it better in the future, and do some testing with spawn. I'm not able to use time for it this month, but I'll keep this thing on my radar as it's definitely something I want to implement at some point. When I have time, I'll try to seek for a solution that can also make the streaming (#64) work, too.

[Taitava]

About no stdin file handle

Hi @dathanb! 🙂 I'm returning to this one year old comment of yours.

The biggest issue I've encountered is that if stdin ends up being empty, spawn will invoke the script with no stdin file handle, and I never figured out how to get around that. stdin not existing isn't too hard to handle in a script, but it'd be unfortunate if scripts had to handle that case in order to be usable from obsidian-shellcommands.

Can you elaborate more on this problem that you faced during your own experiments? I think I did not quite understand it. What do you mean by if scripts had to handle that case in order to be usable from obsidian-shellcommands? How scripts should handle it? 🙂 Are you using Linux or macOS, not Windows?

What I found out when experimenting on Linux, is that at least $(</dev/stdin) should not be used to read stdin in Bash. That will result in an error message /bin/bash: /dev/stdin: No such device or address. $(cat) should be used instead, and it works well. I don't know if you faced this problem or something different. This does not relate to stdin being empty.

In my solution, nothing will be passed to stdin, if the new stdin settings field (per shell command) is empty. Not even an empty string is passed in that case.

[Taitava]

I'm planning to implement this in 0.18.0

Just to give a heads up here, there will two development steps that relate to each other:

1. Version 0.17.0 will contain support for realtime output handling (Real time stdout handling #64 / Real time output handling #275). @dathanb For this, I have looked at the example PR you gave earlier to change the execution part to use Node's spawn() instead of exec(). My development can currently be seen in branch #275-realtime-output-handling.
2. Version 0.18.0 will (hopefully) contain this stdin support. I haven't done any trials on stdin yet, but at least with step 1 almost complete now, the plugin will be a bit closer to be able to support stdin. I will need to consider the problem you mentioned:

   The biggest issue I've encountered is that if stdin ends up being empty, spawn will invoke the script with no stdin file handle, and I never figured out how to get around that. stdin not existing isn't too hard to handle in a script, but it'd be unfortunate if scripts had to handle that case in order to be usable from obsidian-shellcommands.

---

stdin will be a great feature for example for the quite recently added {{note_content}} and {{file_content}} variables, as they often return huge content that exceeds shell command length limits when used as command arguments.

[Taitava]

0.18.0 and stdin support is now released

@dathanb if you happen to have time to test this, any feedback is welcome. 🙂👍
Documentation: https://publish.obsidian.md/shellcommands/Variables/Pass+variables+to+stdin

[dathanb]

Sorry for the belated reply! I updated my script which I'd been invoking like

~/bin/obsidian-to-taskwarrior <<< {{caret_paragraph}}

to just

~/bin/obsidian-to-taskwarrior

with stdin configured to use {{caret_paragraph}} and it's working as expected, so 👍 from me!

[Taitava]

No problem! 🙂 Developing features is a long process (sometimes) and so I understand that getting feedback can also take some time. 👍

Nice to hear that it's working well for you. 😎

[SrGeneroso]

Hi! I'm quite new to the plugin and most likely I'm streching the capabilities of
it. The thing, just for giving context, is that I want remove the git plugin and use shell commands instead. I use WSL and I like to use SSH for the repos within.

I'm wondering, if it would be possible to propmt in the moment when the passphrase is needed.

I thought first to use a prompt to get the password and pass it, which could be theoretically possible using something like sshpass, but I think it would be cooler to don't have such a potentially insecure tool installed as quoting from an article

SSH uses direct TTY access to ensure that the password is indeed issued by an interactive keyboard user. sshpass runs SSH in a dedicated TTY, fooling SSH into thinking it is getting the password from an interactive user.

I know what I'm doing is not very secure as well, at least could be, but anyways, this is how I learn things and maybe there is something in Shell-Commands I don't know yet.

The command I try to make work is this one
wsl -d Ubuntu-22.04 --cd "/mnt/e/00 MD Vault/Obsidian/" --exec git add . && git commit -m "backup at {{date:"HH:mm"}}" && git push origin main
It launches wsl in the cli by default, in this case cmd, navigates to the path and executes the git commands.
From

I guess stdin might be the solution, but following the thread might not be possible? I tried echo "{{_pass}}" | git push origin main and didn't work either. Might be related to the TTY access mentioned before.
So..., stdout handling wouldn't work either...
I don't know, maybe this kind off topic, I was about to open a discussion instead but this might be nearer to the real subject. Let me know if I should repost it there.

In any case, the ideal situation would be that the cli is showing and I can type the passphrase there.

but also it would be really cool to use the prompt. What do you think?

[Taitava]

Hi! Thank you for your message! 🙂

Yeah, I think it might be better to repost this in a new discussion. This discussion #89 is a bit long already 🙂. I do agree that having a possibility to open up an interactive CLI window would help a lot here, so you could put that in the new discussion's name 👍 .

Meanwhile, you could also try to:

1. Invoke WSL using a custom shell. This way you don't need to include the wsl -d Ubuntu-22.04 --cd "/mnt/e/00 MD Vault/Obsidian/" --exec part in your actual shell command. Just git add . && git commit -m "backup at "{{date:HH:mm}} && git push origin main is enough.
2. Maybe the stdin might work better now with the custom shell being used? At least it would be simpler, because then there would be no Windows CMD in the middle, possibly interfering. I'm not at all certain that this would still work. If it doesn't, can you then try the sshpass, even if you will not end up using it permanently?

Let's continue talking in the new discussion when you create it. 🙂

[SrGeneroso]

Reposted it here