Problem with executing bash commands ( bash cat $(which marcel) )

[oOosys]

Hmmm ... why is it a problem to pass shell commands to bash executing bash -c "any command, also multi-line" and to mirror bash output to the Terminal ??? General flaw in the approach of parsing the input line ?

~ $ cat $(which marcel)
#!/bin/bash

function suppress_interruptions()
{
  return 0
}

trap suppress_interruptions SIGINT
trap suppress_interruptions SIGTSTP

python3 -m marcel.main $*
~ $ 

~ $ dash
$ bash -c "cat $(which marcel)"
#!/bin/bash

function suppress_interruptions()
{
  return 0
}

trap suppress_interruptions SIGINT
trap suppress_interruptions SIGTSTP

python3 -m marcel.main $*
$ 

/home/o $ bash cat $(which marcel)
Invalid function syntax: f'''${which marcel}'''
/home/o $ bash "cat $(which marcel)"
Error: ['/bin/bash: line 1: cat /home/o/.local/bin/marcel: No such file or directory']
/home/o $ bash -c "cat $(which marcel)"
Error: ['/bin/bash: - : invalid option']
Error: ['Usage:\t/bin/bash [GNU long option] [option] ...']
Error: ['\t/bin/bash [GNU long option] [option] script-file ...']
Error: ['GNU long options:']
Error: ['\t--debug']
Error: ['\t--debugger']
Error: ['\t--dump-po-strings']
Error: ['\t--dump-strings']
Error: ['\t--help']
Error: ['\t--init-file']
Error: ['\t--login']
Error: ['\t--noediting']
Error: ['\t--noprofile']
Error: ['\t--norc']
Error: ['\t--posix']
Error: ['\t--pretty-print']
Error: ['\t--rcfile']
Error: ['\t--restricted']
Error: ['\t--verbose']
Error: ['\t--version']
Error: ['Shell options:']
Error: ['\t-ilrsD or -c command or -O shopt_option\t\t(invocation only)']
Error: ['\t-abefhkmnptuvxBCHP or -o option']
/home/o $ 

[oOosys]

I expect from a shell to be able to run any executables with any to them passed arguments according to the POSIX standard. More or other features are welcome, but the basic functionality must be there ... or is the marcel shell concept not in harmony with this expectation all other shells I have tried meet without any problems?

If I can't run any executable with any set of arguments like parameter, flags and options from command line, the application running in the Terminal can't be considered to be a shell ... it's then a Terminal shell-like application, but not a shell, right?

[geophile]

I've attempted to support the "escape" to bash without requiring the entire command to be quoted. That means that what gets passed to bash goes through marcel parsing. To get higher degrees of compatibility with bash, I may need to resort to quoting, e.g. bash "...". Or perhaps allow that, but still support quote-less usage in simpler cases.

Supporting more than the most basic bash commands has not been a high priority. My reasoning is that if you want to do more complex things in bash, then maybe just use bash.

By the way, if you are in marcel, then this should work: bash cat "$(which marcel)". I guess rather than figuring out quoting strategies like this, it would be better to just quote the whole thing.

[oOosys]

cat "$(which marcel)" works as well because:

/home/o $ which cat
/usr/bin/cat

cat is an executable.

I suppose that fixing the issue with quoting shouldn't be very hard and sure worth the effort. If I would need to switch to another shell to execute commands like it is specified by POSIX ... I can also directly run a Python module in IDLE instead of pretending to have a shell in a Terminal. There a Python modules out there targeting better interface to shell while working with shell commands.

By the way:

/home/o $ cat<<.>>shef.txt
Parsing error at position 3 of "cat<<.>>shef.txt...": Premature end of input
/home/o $ 

does not work (my most basic text editor I am using for input of short texts) ... what allows the conclusion that marcel as it currently is, is not a shell with extended capabilities compared to dash like the bash shell is, but just a python toy to play around providing most basic shell operations you need to learn the syntax of in order to use them in first place.

In other words ... I suggest that in first step you just pass all the marcel input interactively to bash keeping a bash session running and mirror its output. This way you are sure you have all the shell features a shell user expects from a shell.

In the second step you need to find a way to distinguish user input targeted towards the standard shell from user input targeting the extended capabilities of marcel in order to provide the extensions and to support additional features keeping the bash session up to date with the user input targeted at the extensions marcel comes with.

Don't take this above too serious ... I am probably not yet deep enough into the marcel shell to see what it is really good for. Is learning to cope with one more special shell language of the marcel shell worth the effort? I don't know for sure. At the current stage I am somehow not ready to learn one more syntax to obey to and to remember how it differs from the already known one switching between the two depending on what I want to achieve.

[oOosys]

And yes ... right quotation is a pain while using a shell ... to know when and what needs to be escaped too. I am somehow not ready to adjust myself to the needs of the machine ... I want the machine to adjust to my needs ... (this is what oOo is about https://github.com/oOosys/oOo )

[oOosys]

Are you aware if IPython?
What is the advantage of marcel shell over IPython shell integration?
See an example of IPython session for smooth shell integration in Python REPL:

~ $ alias ip
alias ip='ipython --no-banner'
~ $ ip

In [1]: ls=!ls -l

In [2]: lls=[(line.split()[-1], line.split()[4]) for line in ls if "ipy" in line];lls
Out[2]: []

In [3]: cd /home/o/Downloads
/home/o/Downloads

In [4]: lls=[(line.split()[-1], line.split()[4]) for line in ls if "ipy" in line];lls
Out[4]: []

In [5]: ls=!ls -l

In [6]: lls=[(line.split()[-1], line.split()[4]) for line in ls if "ipy" in line];lls
Out[6]: 
[('ipython-8.17.2-py3-none-any.whl', '808414'),
 ('ipython-8.17.2.tar.gz', '5486488')]

IPython keeps the history of the sesions, so the commands used in a previous session can be reused in the current one and the basic shell commands don't even need the ! prefix. The shell command output goes directly to a Python variable by simple assignment. Isn't it what the marcel shell is all about?

[oOosys]

Another approach similar to marcel shell is https://github.com/zqqqqz2000/shshsh . Isn't it?

[geophile]

You've raised a few issues:

• Marcel passing commands to bash: As I've mentioned elsewhere in this
  conversation, I tried an approach in which marcel args work as bash
  args. You are pointing out a number of command lines that break this
  model, so I will now consider quoting, i.e., bash "...". That should
  improve fidelity with bash, although the very presence of the quote
  marks themselves may complicate commands that are already being
  careful about quotation. Another possible approach is to invent some
  more obscure quoting syntax, e.g. bash '''...'''. But of course, that
  gets into the usual rathole -- what happens if ''' is part of the
  command being quoted?

• What's the point of marcel? Marcel may never be as bash friendly as
  you would like. I am not someone who exploits every feature of
  bash. It is just too disastrously bad a language for me to bother
  with. Like many people, once I have to do a loop or an if in bash, I
  don't bother with it, and switch to Python. Marcel has allowed me to
  stay in the command line for more complex activities. I find it
  incredibly useful for quick analysis of CSV files, or other one-off
  ETL sorts of processing. Also, marcel integrates shell and SQL
  processing, as well as distributed processing. (Marcel's forerunner,
  Object Shell -- another of my repositories -- was developed to help me
  work on a distributed system that had a database on each node.)

• I am aware of IPython but haven't used it extensively. No, marcel is
  not "all about" getting shell output into a variable. Marcel is about
  taking the idea of piping on the command line seriously, but streaming
  structured data (python values) instead of strings. That makes the
  piping idea much more powerful.

• I am not aware of shshsh, there does seem to be some overlap. Marcel
  has two interfaces. You have been playing with the command-line
  interface. The other interface is an API, presented as a marcel
  module, marcel.api. That is the part of marcel that overlaps with
  shshsh.

  Suppose that you want to find files recursively, under your current
  directory, and find the sum of file sizes by extension. From the
  command line:

  ls -fr | map (f: (f.suffix, f.size)) | red . + | sort
  

  Using marcel.api:

  from marcel.api import *
  
  for extension, size in (ls(file=True, recursive=True) |
                             map (lambda f: (f.suffix, f.size)) |
                             red (r_group, r_plus) |
                             sort()):
      print(f'{extension}: {size}')
  

[oOosys]

OK ... I see now that the focus and requirements of oOo differ from the core idea behind marcel shell to an extent where it makes maybe sense to drop the idea of adapting the shell to the needs of oOo.

How would oOo approach the task you have provided the code for? The core idea behind
oOo is to create an executable file named "list of sums of file sizes by extension for all files below a directory" which does the job using a programming language the user creating the executable (if such executable is not already available) is most proficient at. This executable would not take any parameter assuming that there is a file named "directory" in the oOo default parameter directory ( "/oOo/ooo/args/") which content is the absolute path of a directory or multiple lines with different absolute directory paths.
This "directory" file has to be updated with the right directory name before running the executable which will then output the result to stdout, append it to a general .log file as a timestamped entry with the input data and copy the result to clipboard and write it to a .log file with a name of the executable (remembered by the shell for possible usage in next command as input). If you have forgotten the lengthy name of the executable, the main tool of oOo will be a search engine you can give keywords to see which executable files are available to choose from. The oOo "shell" should be therefore able to autocomplete an entry listing some keywords or a description of a task to perform with appropriate executable name giving hints which files need to be updated in order to provide the input data to the executable if not yet happened.

In other words, in an oOo system you will have a huge amount of applications (with multiple different names for the same one), but with a shell capable of picking out the best fit to your description passed on the input line. And if there isn't one yet ... you program it and name in a way which makes it very probably that you get it with autocomplete on describing the task on the command line.

Does it generally make sense to you? Would such shell make sense? You wouldn't need to learn any new language and obey to its rules or remember names of executable files. You don't need to remember where to put braces or commas and where not ...

With a precise description of the task to perform you would get the right executable. If the directory name was the result of a previous command you would then run after the previous command "put last output to 'directory'" before running the executable establishing this way a pipe-chain if necessary.

Having a shell capable of spitting out the right executable from description of the task ... taking it as command and making any chains possible ... A part of that concept is already implemented in the marcel shell - maybe JSON would be a better way as pickling Python objects because programming language independent?

OK ... for YOUR needs marcel shell is the perfect tool ... you were already using Python, so why not using it as reference for passing data structures ... but ... what if you want use the output as input for another not Python based application or a script in another programming language?

In other words ... I am ready to accept closing this issue ... being now aware that marcel shell is not designed to be a shell-replacement. It is a shell mechanisms based Python REPL useful in the by you provided context, but not as a general purpose system shell, right?

[oOosys]

Maybe it would be a good idea to make it clear in the description of this project that marcel shell is not designed to be a replacement for a shell, but a shell-like way of using a Python REPL capable to performing system tasks utilizing the concept of piping data between commands?

[geophile]

Does it generally make sense to you? Would such shell make sense?

This is not something that fits the way I work. I think that natural
language is rarely useful for the sort of tasks I execute from the
shell. Occasionally I will try to remember where I put the script that
I wrote 2 years ago which did xyz, but I don't feel the need for
system support to help me find that script, starting from natural
language. Changing a "directory" variable or file in order to use a
command seems quite clunky to me.

Other issues you raised:

• JSON instead of pickling: I don't think that JSON and pickling are
  good alternatives for one another. Pickling is good for preserving
  structures perfectly, and even including some Python functions (using
  the dill package). Note that JSON is hierarchical, and so cannot
  represent arbitrary graph structures, which pickling can do. For what
  it's worth, marcel already supports JSON, via the json_parse and
  json_format functions.

• What if you want use the output as input for another not Python
  based application or a script in another programming language? That
  already works. The lowest common denominator is text, so that's always
  an option. For example, marcel can generate a stream of tuples
  containing integers (x, x square, x cubed), and then pipe the result
  into grep to look for '2' followed by '7' with any number of
  intervening characters:

  gen 20 | (x: (x, x**2, x**3)) | grep 2.*7

Dumb example, but it shows that marcel can pass data to standard Linux
commands.

It works the other way too. Here is a command using marcel and Linux
executables, at the beginning and end of the command pipeline (cat and
xargs). It locates users whose preferred shell is /bin/bash:

cat /etc/passwd \
| map (line: line.split(':')) \
| select (*line: line[-1] == '/bin/bash') \
| map (username, *_: username) \
| xargs echo

Marcel also supports other common interchange formats, including JSON
and CSV.

• "marcel shell is not designed to be a shell-replacement. It is a shell
  mechanisms based Python REPL useful in the by you provided context,
  but not as a general purpose system shell, right?" I think you have a
  very specific definition of "shell replacement" in mind, which marcel
  doesn't currently meet. I think it will be closer, but still not
  there, once I modify how the bash operator works.

I'll close this issue once bash passes a quoted string instead of
pasting together a string from a bunch of arguments.

[geophile]

This is fixed. You can now run bash commands as before, by providing a set of strings and expressions. As oOosys noted, that is error-prone. To address these concerns, partially, you can now pass a string to the bash operator which gets passed through as is.

This should improve the bash escape, but I have no doubt that further improvements can be made. I'll let improvements be driven by usage and bug reports.