TODO

If you want to help, there are various things you might do:

* One thing that would be helpful regardless of your level of ability
  is documentation help.  I wrote most of the docs in a hurry and they
  aren't very good.  You could read them and let me know what doesn't
  make sense, or edit them and send pull requests.  All the examples I
  used were sort of stupid things I pulled off the top of my head that
  aren't really compelling.  So if you think of some good examples
  (eg. useful pipelining, especially mixed pipelines of external
  programs and racket functions) that would be great.

* Testing!  I have written very few tests.  Tests that use external
  programs are somewhat difficult because they rely on the
  environment, they can't be run in the sandbox provided by the
  automated test servers (though this basically just means there
  should be a directory of tests that the auto server won't run and
  that I (or other contributors) can run locally), etc.  But I've been
  planning on making a directory of tests that use common unix tools
  that should be generally available, and having them do operations in
  some sandbox directory and check that they are correct.

* You can look through the TODO files in the repositories ( IE
  https://github.com/willghatch/racket-rash and
  https://github.com/willghatch/racket-shell-pipeline ) or grep for
  "TODO" in the code base and see if there are things you want to try
  working on.

* You can report any bugs you find.

* You can raise issues for things that you dislike or find weird, or
  features that you want that are missing.  One reason I have said
  "it's not stable yet" aside from some missing features is that I
  want a space to collect feedback for general improvements before a
  stable release.

* You can try writing some interesting pipeline operators (requires
  some macro know-how, so it may not be the best project if you're new
  to Racket).  You could also just think up pipeline operators you
  would find convenient or useful and tell me about them.


General Todos (written more for my memory than for something readable by others)
--------------------------------------------------------------------------------

* fix lexical scoping for redirection options in with-pipeline-config, and add the rest of the options to be configurable lexically
* add a #:context optional argument to #%linea-line so it can be used by macros as well as being an implicit thing that is read.

* unix pipeline member argument generalization of env -- run code in thread to change dynamic environment in any way (this makes env easier -- have &env *only* extend/override, and if you want to reset the environment you can use the generalization)

* make &bg work in run-pipeline/logic macro
** also the other &arguments

* have lexically scoped defaults for all pipeline options, similar to how the implicit starter operator works
** the forms for setting these (eg. the with-rash-config macro) should be able to set a new default for each or fall back to the current lexical default.  I guess there should just be a syntax parameter for each.  But should they be publicly available as syntax parameters or wrapped up in some api like the rash macro setting it or something?

* optional kw-args for rash macros that affect read tables, macros to define rash macros and rash-module-begin with different defaults for all options including read table options

* maybe make it a syntax error to have a normal racket expression [IE starting a line with an open paren] have a pipeline later on the same line.  This will clear up any issues with having the first segment of a pipeline open with parens.

* review names -- pick good ones before doing any stable release

* look through TODO comments

* better drracket integration
** write a lexer for coloring, maybe
** how can I set the color lexer for the interactions window?
** maybe I can use some print handler hook in drracket to give a shell-style prompt when in the drracket interactions window.

* make #lang rash/bash-env or something that uses bash syntax for environment variables and aliases
** it will be nice just as a configuration bridge so you can share a fair amount of config with bash

* rash macro -- don't have the same macro for embedding code strings and just setting configuration.  There should be a separate configuration macro that the rash macro uses, and the rash macro should only be used for embedding code in strings.

* Linea could use a line separator like “;” in bash.  But I don't want to use the semicolon character because it's also the comment character.  I don't want to make the hash character the comment character because it also does so many things in the reader.  In some of these ways I want to keep it closer to Racket than Bash and friends.

* readline-lib -- I can reliably make rash-repl go to 100% cpu by launching a terminal with rash, running rash, running zsh, running rash, running rash, then xkill-ing the window.  But only when I have readline enabled.

misc
----

* option-app -- needs better name (kw-check-app?), and should it be part of another package?  I think I want to use it in more packages...


About interactive shells
------------------------

* discoverability
** the default prompt should help with discoverability -- maybe say the command to get help, look up docs, etc.  Something like «type HELP START» in LoE.
** maybe the default prompt could on its first run print a random fortune-like tip message, like in so many video games?
** how can interactive features be more discoverable than just being listed in the docs/tutorial?
** in the short term this probably doesn't matter much, but long term it would be nice.
** help line-macro
*** there should be one -- «help foo» should probably do a variety of things to get «foo» docs -- look for racket docs, man pages, info pages, or have custom help.
*** «help» and «help help» should give info about the help command itself.

* things that still bug me in my interactive shell
** <() is still missing
** running with modified environment
*** a good workaround is `env FOO=bar cmd ...` -- so this isn't the highest priority...
** issues with a lack of job control
*** vim has some issue starting -- I run it and it seems like it exits immediately.  Then I hit C-d to exit the shell and it pulls up vim.  Then I exit vim and things seem to be back to normal!

** issues with SHELL not being posix compatible
*** Is there a better variable to set than SHELL to say "this is a rash repl"?  Several programs use the current SHELL variable to run posix shell code.  That seems dumb to me -- if it's always assumed to be the posix shell, why not always run /bin/sh?  But they do it.

* prompt
** I should get info about the current directory path like I do in zsh, but I should use more styles for more info (or at least make that optional).  eg. ownership/group/other, read/write/none, mount points, git repo roots, size would be cool but too expensive.
** I need to make all the computation about extra info in the prompt be done with a timeout...
** I should also have options to compress the path in different ways
*** compress $HOME to ~, maybe also allow configurable lists of paths and shorthands
*** allow a max length, and compress path to one letter per dir, eg ~/r/c/c/the-last-dir-spell-out-long-maybe


Documentation
-------------

* Maybe at some point there should be a section for Racketeers unfamiliar with Rash or shells in general, and a section for people familiar with shells to but unfamiliar with Racket.
* define-pipeline-operator #:operator option
* =pipeline-segment= and macros to create first-class pipeline segments
* pipeline/logic macro && ||
* #:object-to-out
* I probably need to comment my macros better
* I need an example table, eg. comparing “how to do X” in rash vs other shells.  Maybe also how to do them in shell/pipeline.
* Feature comparison table (vs other shells)?

Line Editing
------------

For now I've got racket's readline FFI library working for OK line editing.

In the long term, what I really want is an emacs in racket that can be used as the line editor.  Specifically, I would want an emacs-style extensible editor where the buffer objects and editing commands are not tied to any sort of display, such that they could be connected to different displays (even more flexibly than GNU emacs with its ability to be used with curses/terminal version and a gui version).  Then one such display could be a line-editor style display that just lives in the bottom lines of a terminal, but still give full emacs buffer style editing to the code in the line (or lines -- obviously it should have real support for multi-lines) that you are editing.  The editor would be connected to your repl evaluator and be able to talk with it to determine information about the code in the buffer, do various different types of completion on it, know whether it is a valid command/whether it parses, etc.  Of course then you could also make a curses-style interface, and a gui interface, and maybe (hopefully!) even embed it to be the editor inside DrRacket (and in other GUI racket programs that have text editing fields).  Then you could, among other things, have custom editing commands, keymaps, editing modes (a la vim), etc all set up in one place and used in your shell, in your stand-alone text editor, editing fields in other programs, etc.  Several people have talked about wanting an emacs in Racket.  It would be a big project.  If nobody else makes one I can definitely see myself doing it (a basic version, at least), but it's not really at the top of my priorities right now -- I need to do things like, say, work on my PhD.  So I don't see this happening terribly soon.

However distant it may be, you really want this sort of thing in your shell.  The only major difference between bash and zsh, for instance, is that zsh has an emacs as its editor.  And that makes a world of difference.  Zsh has many more and better programmed completions available.  It has many really useful plugins (eg. zaw -- it's like helm in emacs).  It's why so many people love zsh so much when as a language it is essentially the same as bash.  But while it has an emacs, you have to do all the extending in (*gulp*) zsh itself...  so plugins for zsh are rather limited by the implementation language.  Wouldn't it be great if the extension language for your editor were... any language in the Racket ecosystem?  Yes, it would.


Bugs
----

* `(rash «ls»)` at the top level of the repl acts like `ls` at the top level of the repl, which is wrong.  The in/out/err options are set by splicing-syntax-parameterize.  Is there a bug in how I set that up?




shell-pipeline
--------------

* pipeline operators
*** what operators really should be the base set?
** =unix-pipe=
*** The basic one for completeness, and the cool default one
** =object-pipe=
*** =object-pipe/expression=
**** This one really needs a shorthand, because it's particularly useful as a starter like cat -- eg {=object-pipe/expression= x |> do-something-to-x |> etc}
**** maybe |e> or =e=.  |e> could also have |e>> that is like |>> to |>.  But =e= might be more like general named operators, and therefore better.
*** |> and friends that use the expand-pipeline-argument function are sort of dumb for not accepting macros.  Is there a good way to resolve the tension between allowing macros and auto-detecting whether the argument is used?
** =pipeline-segment=
*** IE accept a first-class pipeline definition segment (IE compound-member-spec [or unix or object member spec])
** sequence pipes -- =filter= =map= =foldX=
*** what is the right way to make a fold pipe?  =foldl= init-val body-f body-arg1 body-arg2 ... ---- the current-pipeline-arg can be the iterative argument, but what name for the accumulator?
** infix math pipe
** operator that just accepts first-class pipeline specs (IE maybe sugar over =composite-pipe= to give it a better interface, or maybe it's fine as is)
*** I need a form that makes it easy to define first-class pipeline specs
**** But this has issues with the start vs joint difference.  But I'm less confident that the difference is really necessary, since you can use an optional argument with a flag for a default value instead.

* operator names
** I think they should be more to-the-point and better.  `basic-` was a mistake also.
** probably I'll use these:
*** =unix= (|), =procedure= (|>), =form= (=basic-object-pipe/expression=), =procedure/port->string= (this one is a little unwieldy...)

* option for shared stderr string capture or individual ones.  Per unix-y section, at least.

* redirection arguments for shell/pipeline and mixed-pipeline
** I don't want to accept a list with a modifier symbol eg '("foo.txt" 'append)
** I think probably I should use some redirection struct that has a field for the modifier.
** It would be good for the redirection object to be able to have a list of redirection specifications, in which case it would copy the port to all of them.  But this begs the question of what to do if copying or opening some but not all of the redirections fails.  Eg. what if you have one redirection set to truncate a file and one set to write but not truncate, and both files exist.

* globs in redirection
** bash and zsh both allow globs in redirection, but they treat them differently (at least by default, or in my configuration if I've changed it somehow).
** bash errors with an "ambiguous redirect" error if a glob matches more than one file, and does the classic "use the glob literally if there is no match" semantics.  It does this for redirections even if you set nullglob!
** zsh redirects to all files if a glob matches multiple.
** It would potentially be useful to allow multiple arguments for redirection and use port-copy on them (for output) or port-append (for input)

* improve dollar-expansion
** maybe support $() as an escape to lisp
** add ${foo/from/to} or something for quick string replacement
*** the most basic and important thing here is to have a way to delimit the variable name.
** my biggest question is whether to make it be more like posix shells and have ${} be posix-style or whether to make it match reading and make $() and ${} be escapes to racket...
*** If I make it configurable, I could eg have ◸◹ etc in the dollar escape reader, so probably the best thing would be just escapes to reading racket that support various implicit macros.
** should some aspect of dollar expansion be in object pipes?
** Rather than formatting a list to a string, it would be better for a list result in a dollar expansion to cause a list of strings to be made, one for each result.  Multiple dollar escapes in the same string that return lists should return a cartesian product of strings.

* splicing -- instead of just splicing lists, unix pipelines should splice any sequence
** generally, I need good convenient ways to access the command line, even easier than racket/cmdline for basic usage.

* pipeline send signal?  eg. to send sighup when rash is killed so that some programs (eg nohup) can decide whether or not they should die.

* Better API for redirection settings in basic and mixed pipelines
** specifically, how to tell it truncate/append/error
** how to tell it you want stderr as a string-port or shared-string-port

* good API for setting defaults
** all default values should have syntax parameters that are either exposed as such or have a nice API for setting their values, and they should probably all live in one file in private/
*** specifically these include the defaults for in/out/err, the default pipeline starter, the default truncate/append/error behavior for out/err, default strict/permissive/lazy correctness, lazy-timeout,

* option for code to run in manager thread before running members in order to modify parameters - IE generalization of modifying env for all members.
** environment variable / generic thread environment mutation per-member as well

* process fd/temp-file redirects a la <() >() from bash
** Probably I want a flag struct for each.  The struct will contain a thunk that will evaluate to a file name (for <()) or will take a file name (for >(), the pipeline running mechanism will have to give it one).
** Unless it's actually backed by FD, the output process will have to finish before starting the input process, and pipeline waiting will have to wait for the sub-pipeline stuff as well...

* inspecting pipeline segments with correct compound member handling

* add a set of symbols that the output-transformer can translate into common-case functions -- 'string, 'trim, 'lines, etc.

* a version of pipeline-macro that doesn't have all the overhead (maybe one that doesn't allow unix-segments)
** maybe it's not terribly important, because it would be doing mostly the same thing as the threading macro, but it would be more flexible -- eg. =filter=, =for=, etc are nice, especially if they have short names.
** =object-pipe= should have a keyword argument that sets whether it auto-converts ports, and the default is a syntax parameter that the efficient vs convenient rash macros can set accordingly.  Also, this allows the one-off occasions when you do want to do the opposite, without requiring a different =basic-object-pipe=
** obj-pipeline-spec, unix-pipeline-spec, and composite-pipeline-spec should be syntax parameters that are also set by the outside macro, basically determining whether they produce runtime pipeline objects or if they desugar to the fast path that has no process or thread overhead.

* I want to have a macro to define a first class pipeline segment (IE a compound-pipeline-member, but one that is always a joint and not a starter).  Maybe a compound-pipeline-member can be called as a function, which takes one argument and shoves it into the pipeline?
** how will this interact with the trimmed-down version of the macro from above?


* Currently each object segment runs in its own thread, but maybe they should always run in the manager thread instead.  That way they can eg. mutate the environment, and that can be an easy way to implement whole-pipeline environment changes.  Also it will lower threading overhead.  However, it is the way it is currently so that the manager thread can (in some future day) catch a signal to suspend for job control.

* shell/pipeline same-thread-func -- this should be removed, it was a hack to allow cd to be a pipeline function and it's obsolete.

* #:in-to-object 
** -- basically so &< FILE will still do something when a pipeline starts with an object member.  
** But it would need to have the port tracked to make sure it's closed.  It may be a bad idea to use at all unless I can get it to be automatically closed since it's a file-system port (but will executors will probably do this)

* parsing/formats
** want parsers for all major unix file types (eg. : separated, etc -- there should be a generic one, and then specialized ones for eg. /etc/passwd that knows what the fields are)
** want a parsing framework like caml-shcaml shtreams that includes a structured format AND the original line -- eg. for filtering and being able to pass the original back unedited
** want a standard object format for pipelining from eg. sys admin commands
*** how should it print in the prompt printer?
*** maybe this should be based on row types?

* bg issues
** what does it mean to run with stdin in the background unless I have job control?
** maybe the default input port should be allowed to be some function that determines what input it gets based on things like whether it's in the background or not?

* a custom exn type for unix pipeline failures?

* ulimit and other resource limiting things for subprocesses and pipeline threads -- how can I go about doing that?

* have easy forms for doing dollar, glob, etc expansion, and document them
* add a regex glob form

Job Control
-----------

* Implementation plan:
** (note that this will only work on Unix, so I will have to make job control unix-only... which is probably fine)
** launch subprocess in the same new group (with Racket's new-ish ability to do that via `subprocess`).
** Use `subprocess-pid` on the first subprocess.
** Use (with FFI) `getpgid` on the subprocess pid (and probably record this in the pipeline object).
** use (with the FFI) `tcsetpgrp` on the stdin of the terminal *** Note that this (and therefore job control in general with process groups) only makes sense if I have a file descriptor for a terminal!
** because the subprocess command doesn't let me use `tcsetpgrp` before the subprocess starts potentially reading/writing to the terminal, I might have to send SIGCONT to the process group after using tcsetpgrp because the processes could already have received SIGTTIN or SIGTTOU

* what about job control with Racket function segments?
** This is a problem because "jobs" of subprocesses are all in the same process group, which will be (and needs to be) different from the process group of the shell.  So if the shell tries to read/write to the terminal while it is controlled by another group, the shell will get SIGTTIN/SIGTTOU.
** Note that even `run-subprocess-pipeline` accepts functions to run in a similar manner to subprocess, so this problem exists even at the lowest level of shell-pipeline...
** I can have job control work only for pure subprocess pipelines.
*** But I would like a way to suspend/resume long Racket computations.
** normal shells solve the problem by forking subshells, but forking is not advised in Racket, and unless I have tight control of everything in the process forking might cause problems...
** maybe I can send appropriate terminal input/output through a proxy port for Racket functions during job control, and launch an extra subprocess in the group that does the actual reading/writing.  But then I could eg. write bytes to the intermediate process (and therefore the writer thinks it's succeeded in writing) but have the intermediate process hang or die before actually writing.


* Pipeline suspend/resume -- Threads have pause/resume, Unix processes have sigstop/sigcont, but I'm not sure how to pause/suspend windows processes.  Maybe a Unix-only feature?  But this is a pretty standard shell feature for job control.

* job control -- for an interactive shell, how should the shell capture references to the pipelines to do job control?
** importantly, currently (IE without job control) the repl is still the foreground process of the terminal when you press control c or control z, so it gets SIGINT or SIGTSTP even when you have launched eg. vim
** When Racket gets SIGHUP the main thread generates exn:break:hang-up, and for SIGQUIT it generates exn:break:terminate.  So a handler around the main repl loop can propogate SIGHUP to any jobs (except disowned ones) and quit gracefully.

** when C-c is hit in the terminal, only the main thread gets it.  This is a problem for eg. background computation or object pipelines.  The main thread needs a way to communicate these things.
** for the repl, I need to have a handler specifically for user-break that will send signals to the current pipeline if one is running.

* bg pipeline disowning?
** This is really only reasonable for external-proccess-only pipelines, and I'm not sure how reasonable it is on Windows.


make replacement
----------------

* There is a little `make` replacement demo language in the repo.  I want to leave it as it is (simple) as a quick demo of how you can make an interesting language with line macros.  But I would also like to make a full-scale version.
** it should support #:phony targets (eg. clean).  This may need to fuss with temporary files to humor the underlying `make` library that doesn't support it.
** It should support regexps as targets.  The dependency list expressions and make body should have the match from the target.  This is a generalization of the % wildcard in normal makefiles.  Any supplied target that doesn't match a literal pattern should be tested against regexp patterns.
** it should set the working directory using `define-runtime-path` so the build script can be run from any directory but always affect the proper directory.  If it parameterizes this anew for each build, users can use `cd` in recipes without worrying.
** it would be good to have command-line arguments for specifying eg. printing verbosity