Support for interactive shell using picocli

[anshunjain]

Just started using picocli for a simple command line app, and find it amazingly easy to use and cool. I have also been experimenting with having interactive shell in my program which once launched will also allow execution of several subcommands. I am using cliche for this. Though it seems like picocli should be already able to handle this. Am I missing some obvious way of putting CommandLine in a loop to continuously parse for subcommands.

Happy to contribute if there is an agreeable change which could make it work that way.

[anshunjain]

https://code.google.com/archive/p/cliche/
this is the cliche i was referring to

[remkop]

Thank you for the positive feedback! Great to hear!

I think the only thing missing to accomplish a full interactive shell is a tokenizer that understands quotes. Basically, in your for loop you want to read in a single string from the console, and then parse it into separate arguments (like the shell does for you usually).

So, input like

<command> a b c "1 2 3" '4 5 6'

Is parsed into 6 tokens:

<command>
a
b
c
1 2 3
4 5 6

This can be a separate class from picocli.CommandLine.

Do you think you'll be able to submit a pull request for this?

[BeeeWall]

I think that could be accomplished by splitting the input, looping and looking for quotes, and combining stuff. I might give this a try at some point.

[remkop]

Looking forward to any contributions!

FYI, I found the java.io.StreamTokenizer class extremely helpful when implementing @-file expansion. The following configuration is probably generally useful for parsing command line arguments:

StreamTokenizer tok = new StreamTokenizer(reader);
tok.resetSyntax();
tok.wordChars(' ', 255);
tok.whitespaceChars(0, ' ');
tok.commentChar('#');
tok.quoteChar('"');
tok.quoteChar('\'');
while (tok.nextToken() != StreamTokenizer.TT_EOF) {
    handleToken(tok.sval,  ...);
}

[BeeeWall]

On second thought, I'm not sure. I do all my work on a tablet using AIDE IDE, which unfortunately doesn't support Gradle for java projects. So I'll try, but not sure I can do proper testing. I'll probably have to do it in another project, test it, and copy it over hoping nothing breaks.

[BeeeWall]

I have some code tested for quotes and escaping spaces, I just need to work on putting it into picocli. Before I do so, any code style rules I need to be aware of? And I could also probably make some sort of runInteractive method, should I do that as a separate method, a flag for normal run, an annotation, or what?

[remkop]

Generally, try to stick to the style of the existing code. I may reformat so don't worry too much.
Please include unit tests though. :-)

Also, can you show some examples of how this facility can/should be used?

[BeeeWall]

It takes a String, splits it on any unescaped/unquoted spaces, and returns a String array.
Or do you mean the interactive command line?

[remkop]

I meant usage by users of the library.

[BeeeWall]

The main purpose is for an interactive shell, like this issue is about. You would just use it if a user of a project wanted to, say, put a path to a file with a space in it. This would split the rest up, but if the path was quoted, keep the spaces. Like in other shells, you can either quote it, or backslash escape.

[BeeeWall]

Like file.jar --open "/path/to/a file"

[remkop]

Ok, but isn't that how it currently works when a picocli-based command line utility is invoked from the command line or in a shell script?

(Picocli already supports quoted command line arguments with embedded spaces.)

I'm interested in seeing the code that application authors would need to write to support an interactive shell in their application. I imagine this would involve reading user input in a loop from standard input and processing the input as commands. I'm interested in seeing what your contribution adds in that scenario.

[BeeeWall]

Earlier on in the issue, you said that quoting wasn't supported. I made some code for that.
For an interactive terminal, here's an example:

import java.util.Scanner;
import picocli.CommandLine;
import picocli.CommandLine.Option;
import picocli.CommandLine.Parameters;

public class Main implements Runnable {

	@Parameters(index="0..*") String[] params;

	@Option(names = {"q", "quit"}) boolean optQuit = false;

	@Override
	public void run() {
		if (optQuit) System.exit(0);
		for (int i = 0; i < params.length; i++) System.out.println(params[i]);			// Print parameters
	}

	public static void main(String[] args) {
		Scanner input = new Scanner(System.in).useDelimiter("\n");						// So that prompt shows once each time
		String shPrompt = "cli> ";
		System.out.print(shPrompt);
		while (true) {
			if (input.hasNext()) {
				String cmd = input.next();
				String[] cmdSplit = new String[]{""};
				boolean inQuote = false;												// Stores if currently quoted
				char quoteChar = 0;														// Stores which char ends quotes
				for (int i = 0; i < cmd.length(); i++) {
					char curChar = cmd.charAt(i);
					char preChar = (i == 0) ? 0 : cmd.charAt(i - 1);
					String curStr = cmdSplit[cmdSplit.length - 1];
					if (curChar == '"' || curChar == '\'') {							// Check if has quotation mark
						if (i == 0 || cmd.charAt(i - 1) != '\\' && 
							(curChar == quoteChar || !inQuote)) {						// Make sure quote is unescaped and the right quote
							inQuote = !inQuote;											// Toggle quoted text
							if (inQuote) quoteChar = curChar;							// If entering a quote, set quote character to current char
						}
						else {
							if (preChar == '\\') {										// If previous char is backslash
								curStr = curStr.substring(0, curStr.length() - 1);		// Cut off last backslash
							}
							curStr += curChar;											// Add quote to string
						}
					}
					else if (!inQuote && curChar == ' ') {								// If char is space and not in quote
						if (preChar == '\\') {											// If previous char is backslash
							curStr = curStr.substring(0, curStr.length() - 1);			// Cut off last backslash
							curStr += curChar;											// Add thebspace in
						}
						else {
							String[] cmdTmp = new String[cmdSplit.length + 1];			// Make array one bigger
							System.arraycopy(cmdSplit, 0, cmdTmp, 0, cmdSplit.length);	// Copy to bigger array
							cmdSplit = cmdTmp;											// Replace old array
							curStr = "";												// Empty current string
						}
					}
					else {
						curStr += curChar;												// Add char to string
					}
					cmdSplit[cmdSplit.length - 1] = curStr;								// Copy back to array
				}
				CommandLine.run(new Main(), System.out, cmdSplit);
				System.out.print(shPrompt);
			}
		}
	}
}

(If picocli already has quote support, most of that is unnecessary.)
It will take an input String, split it at spaces (unquoted/unescaped only), and output it. For example,

cli> a b c "1 2 3" '4 5 6' 7\ 8\ 9
a
b
c
1 2 3
4 5 6
7 8 9
cli> 

[BeeeWall]

Here's a simpler version, without all the quote code:

import java.util.Scanner;
import picocli.CommandLine;
import picocli.CommandLine.Option;
import picocli.CommandLine.Parameters;

public class Main implements Runnable {

	@Parameters(index="0..*") String[] params;

	@Option(names = {"q", "quit"}) boolean optQuit = false;

	@Override
	public void run() {
		if (optQuit) System.exit(0);
		for (int i = 0; i < params.length; i++) System.out.println(params[i]);			// Print parameters
	}

	public static void main(String[] args) {
		Scanner input = new Scanner(System.in).useDelimiter("\n");						// So that prompt shows once each time
		String shPrompt = "cli> ";
		System.out.print(shPrompt);
		while (true) {
			if (input.hasNext()) {
				String cmd = input.next();
				CommandLine.run(new Main(), System.out, cmd);
				System.out.print(shPrompt);
			}
		}
	}
}	

[BeeeWall]

Basically, you just use a Scanner, split at new line, and print out a prompt (that's what the text to the left of the "$" in the terminal is called, right?).

[remkop]

Thanks for the clarification.

The quote support I mentioned is mostly coming from the operating system shell, where quoted arguments with embedded spaces are delivered to the application in a single argument. In a custom interactive shell, we need to do this work ourselves, which is what you are showing in the first example. Good stuff!

Would it be possible to use StreamTokenizer instead of custom parsing logic? About code formatting, please use 4 space indentation and redundant braces. :-)

[BeeeWall]

I will be looking into StreamTokenizer, I just haven't yet.
Got it, 4 spaces and redundant braces. Grr, rhymes.

[BeeeWall]

The thing with my parser is that it supports escaping spaces and quotes, while StreamTokenizer does not. I don't think I could even use replace and Unicode escapes, because if I remember right, the compiler replaces those, so it would be completely useless to do that. I could possibly quote them all with the other type, but that's all I can think of, and it would still have issues. You couldn't quote a quote inside a quote ("\"" -> "'"'") because the quote would be in a quote, and not quote the quote ("'"'" == '"[unmatched ']). (I'm sorry for the tongue twister).

[remkop]

I see. Escaping spaces may not be a common use case (users can quote to embed spaces instead), but I image that parameter values with embedded quotes are more common. Have you looked at doing this with a regular expression? I'd like to start with the simplest/shortest code possible.

[BeeeWall]

I haven't yet, because this felt simpler (if a lot more massive), but I can look into it.

[BeeeWall]

Good news: I got a regex. Bad news: I got a regex.

import java.util.Scanner;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import picocli.CommandLine;
import picocli.CommandLine.Option;
import picocli.CommandLine.Parameters;

public class Main implements Runnable {

	@Parameters(index="0..*") String[] params;

	@Option(names = {"q", "quit"}) boolean optQuit = false;

	@Override
	public void run() {
		if (optQuit) System.exit(0);
		for (int i = 0; i < params.length; i++) {
			System.out.println(params[i]);												// Print parameters
		}
	}

	public static void main(String[] args) {
		Scanner input = new Scanner(System.in).useDelimiter("\n");						// So that prompt shows once each time
		String shPrompt = "cli> ";
		System.out.print(shPrompt);
		while (true) {
			if (input.hasNext()) {
				String cmd = input.next();
				String[] cmdSplit = new String[0];
				Matcher match = Pattern.compile("(?:\\S*\\\\\\s)+\\S*" +
												"|[^\"'\\\\ ]" +
												"|(?<!\\\\)\".+?[^\\\\](?:\"|$)" +
												"|(?<!\\\\)'.+?[^\\\\](?:'|$)")
												.matcher(cmd);
				// Just... plug the regex into regexr or something
				while (match.find()) {
					String[] tmpArr = new String[cmdSplit.length + 1];
					System.arraycopy(cmdSplit, 0, tmpArr, 0, cmdSplit.length);
					cmdSplit = tmpArr;
					cmdSplit[cmdSplit.length - 1] = match.group(0).replace("\\ ", " ");	// Unescape spaces
					int index = cmdSplit.length - 1;
					if (cmdSplit[index].startsWith("'") ||
						cmdSplit[index].startsWith("\"")) {
						cmdSplit[index] = cmdSplit[index].substring(1);					// Remove starting quote
					}
					if (cmdSplit[index].endsWith("'") ||
						cmdSplit[index].endsWith("\"")) {
						cmdSplit[index] = cmdSplit[index]
							.substring(0, cmdSplit[index].length() - 1);				// Remove ending quote
					}
				}
				CommandLine.run(new Main(), System.out, cmdSplit);
				System.out.print(shPrompt);
			}
		}
	}
}

Now if you'll excuse me... screams into and murders innocent pillow

[remkop]

Ouch! Your original proposal looks a lot easier to read and maintain. Let's go with that.
Sorry to hear about your pillow.

I propose that we put this code in a separate class, that can be used something like this:

class Example {
    public static void main(String... args) {
        String shPrompt = "cli> ";
        System.out.print(shPrompt);
        try (LineNumberReader reader = new LineNumberReader(new InputStreamReader(System.in))) {
            String line = null;
            while (line = reader.readLine() != null) {
                String[] arguments = ArgumentSplitter.split(line);
                CommandLine.run(new Main(), System.out, cmdSplit);
                System.out.print(shPrompt);
            }
        }
    }
}

Tentatively named it ArgumentSplitter, I'm open for suggestions.

[BeeeWall]

Yeah, that makes since. Any particular reason you went with LineNumberReader, or is it just in case anybody uses it?

[BeeeWall]

Like this?

	public static class ArgumentSplitter {
		
		public static String[] split(InputStream is) {
			return split(is, "");
		}
		
		public static String[] split(InputStream is, String previous) {
			Scanner input = new Scanner(is).useDelimiter("\n");								// So that prompt shows once each time
			while (true) {
				if (input.hasNext()) {
					String cmd = previous + input.next();
					if (cmd.endsWith("\\")) {												// If new line is escaped
						System.out.print("> ");
						return split(is, cmd.substring(0, cmd.length() - 1));				// Run again
					}
					String[] cmdSplit = new String[]{""};
					boolean inQuote = false;												// Stores if currently quoted
					char quoteChar = 0;														// Stores which char ends quotes
					for (int i = 0; i < cmd.length(); i++) {
						char curChar = cmd.charAt(i);
						char preChar = (i == 0) ? 0 : cmd.charAt(i - 1);
						String curStr = cmdSplit[cmdSplit.length - 1];
						if (curChar == '"' || curChar == '\'') {							// Check if has quotation mark
							if (i == 0 || cmd.charAt(i - 1) != '\\' && 
								(curChar == quoteChar || !inQuote)) {						// Make sure quote is unescaped and the right quote
								inQuote = !inQuote;											// Toggle quoted text
								if (inQuote) quoteChar = curChar;							// If entering a quote, set quote character to current char
							}
							else {
								if (preChar == '\\') {										// If previous char is backslash
									curStr = curStr.substring(0, curStr.length() - 1);		// Cut off last backslash
								}
								curStr += curChar;											// Add quote to string
							}
						}
						else if (!inQuote && curChar == ' ') {								// If char is space and not in quote
							if (preChar == '\\') {											// If previous char is backslash
								curStr = curStr.substring(0, curStr.length() - 1);			// Cut off last backslash
								curStr += curChar;											// Add the space in
							}
							else {
								String[] cmdTmp = new String[cmdSplit.length + 1];			// Make array one bigger
								System.arraycopy(cmdSplit, 0, cmdTmp, 0, cmdSplit.length);	// Copy to bigger array
								cmdSplit = cmdTmp;											// Replace old array
								curStr = "";												// Empty current string
							}
						}
						else {
							curStr += curChar;												// Add char to string
						}
						cmdSplit[cmdSplit.length - 1] = curStr;								// Copy back to array
					}
					return cmdSplit;
				}
			}
		}
	}

Also, this version has support for escaping new lines, in case people script with their programs or something.
Example implementation:

	@Parameters(index="0..*") String[] params;

	@Option(names = {"q", "quit"}) boolean optQuit = false;

	@Override
	public void run() {
		if (optQuit) System.exit(0);
		for (int i = 0; i < params.length; i++) {
			System.out.println(params[i]);													// Print parameters
		}
	}

	public static void main(String[] args) {
		String shPrompt = "cli> ";
		System.out.print(shPrompt);
		while (true) {
			String[] params = ArgumentSplitter.split(System.in);
			CommandLine.run(new Main(), System.out, params);
			System.out.print(shPrompt);
		}
	}

Example input:

a b c "d e f\"" 'g h "i"' 1\ 2\ 3 \
4 5 \"6\"\
789

Output:

cli> a b c "d e f\"" 'g h "i"' 1\ 2\ 3 \
> 4 5 \"6\"\
> 789
a
b
c
d e f"
g h "i"
1 2 3
4
5
"6"789
cli> 

(Note that currently, if you copy and paste the example input, it looks a bit messed up because it prints ">", but data and output are the same.)
If you copy and paste:

cli> a b c "d e f\"" 'g h "i"' 1\ 2\ 3 \
4>  5 \"6\"\> 
789
a
b
c
d e f"
g h "i"
1 2 3
4
5
"6"789
cli> 

[BeeeWall]

Little issue I just realized: options inside quotes aren't escaped, so entering "quit" into the previous example shell still quits, and does not print quit.

[remkop]

Good stuff.

I would like to have a "core" method that just takes a String and splits it into substrings representing command line arguments. We could optionally support one or more convenience methods that check for multi-line input, but users may want to implement this themselves. The core method or methods would provide the building blocks for this.

I imagine the signature of the core method to looks something like this:

String[] split(String);

Ideally, I would like to leave the choice of using a BufferedReader, LineNumberReader, or a Scanner to the user. Quick question: is there any reason you're not using the Scanner::nextLine method, and do the while loop with Scanner::hasNext instead of while (true)?

[BeeeWall]

In that case, I'll probably move the other code to a function that takes a String, and then keep the newline escapes in the InputStream version (possibly also a Reader version, to allow choice).
Not using nextLine because I only learned about it earlier today while researching Scanner more :) (when you create a Java project, AIDE automatically has a Scanner.next() loop).
If you don't do while (true), the loop will be skipped, because the user has to input stuff for hasNext to be true, but the loop would end the moment the code runs.
In the convenience method, I will probably split on (unescaped/unquoted) semicolons as well, to further imitate full shells. Is that fine?

[remkop]

It sounds like the convenience method should not be a single method but a builder-like construct that users can configure before using it. Similar to StreamTokenizer.

[BeeeWall]

Got it. So should I just basically have options for everything (quotes, maybe quotechar, escape newlines, escape spaces, etc)?

[remkop]

Yes, that makes sense.

[BeeeWall]

I've made a new version of the class, using the StreamTokenizer syntax. It currently isn't well commented, I will work on that once the final version is ready.

It has two constructors, accepting either a String or an InputStream. The InputStream version has support for escaping newlines, while the String one does not (yet).

I've added toggles and settings for a lot. Most of them come from StringTokenizer, so look at those docs if you need more detail.

They are initialized with some useful defaults, such as quotes as quoteChars and backslash for escapeChars, but they can be removed.

I also plan to add a method to check if something is in a category.

Also note that while I have an option for comment characters (like StreamTokenizer), I haven't added support yet, because I'm not sure I'll keep it (I guess it could be useful with scripts?). If you want, I'll implement it, it shouldn't be hard.

public static class CommandTokenizer {

		private String[] tokens;
		private int position = 0;
		// Characters that start comments
		private List<Character> commentChars = new ArrayList<Character>();
		// Characters that escape other characters
		private List<Character> escapeChars = new ArrayList<Character>(Arrays.asList('\\'));
		// Characters to be escaped automatically
		private List<Character> escapedChars = new ArrayList<Character>();
		// Characters that begin/end quotes
		private List<Character> quoteChars = new ArrayList<Character>(Arrays.asList('"', '\''));
		// Characters to be treated as whitespace
		private List<Character> whitespaceChars = new ArrayList<Character>(Arrays.asList(' ', '\n'));
		// Characters to use to represent a new line
		private List<String> eolPatterns = new ArrayList<String>(Arrays.asList("\r\n", "\r", "\n"));
		// Remove blanks at start and end of a string
		private boolean trimBlanks = true;

		public CommandTokenizer(String cmd) {
			this.tokens = parse(cmd);
		}

		public CommandTokenizer(InputStream in) {
			this.tokens = parse(in);
		}

		private String[] parse(String cmd) {
			String[] cmdSplit = new String[]{""};
			boolean inQuote = false;												// Stores if currently quoted
			char quoteChar = 0;														// Stores which char ends quotes
			for (int i = 0; i < cmd.length(); i++) {
				char curChar = cmd.charAt(i);
				char preChar = (i == 0) ? 0 : cmd.charAt(i - 1);
				String curStr = cmdSplit[cmdSplit.length - 1];
				if (quoteChars.contains(curChar)) {									// Check if has quotation mark
					if (i == 0 || !escapeChars.contains(preChar) && 
						(curChar == quoteChar || !inQuote)) {						// Make sure quote is unescaped and the right quote
						inQuote = !inQuote;											// Toggle quoted text
						if (inQuote) quoteChar = curChar;							// If entering a quote, set quote character to current char
					}
					else {
						if (escapeChars.contains(preChar)) {										// If previous char is backslash
							curStr = curStr.substring(0, curStr.length() - 1);		// Cut off last backslash
						}
						curStr += curChar;											// Add quote to string
					}
				}
				else if (!inQuote && curChar == ' ') {								// If char is space and not in quote
					if (escapeChars.contains(preChar)) {											// If previous char is backslash
						curStr = curStr.substring(0, curStr.length() - 1);			// Cut off last backslash
						curStr += curChar;											// Add the space in
					}
					else {
						String[] cmdTmp = new String[cmdSplit.length + 1];			// Make array one bigger
						System.arraycopy(cmdSplit, 0, cmdTmp, 0, cmdSplit.length);	// Copy to bigger array
						cmdSplit = cmdTmp;											// Replace old array
						curStr = "";												// Empty current string
					}
				}
				else {
					curStr += curChar;												// Add char to string
				}
				cmdSplit[cmdSplit.length - 1] = curStr;								// Copy back to array
			}
			return cmdSplit;
		}

		public String[] parse(InputStream in) {
			String delimiters = "";
			for (String pattern : eolPatterns) {
				if (!delimiters.equals("")) {
					delimiters += "|";
				}
				delimiters += pattern;
			}
			Scanner input = new Scanner(in).useDelimiter(delimiters);
			String[] cmdSplit = new String[0];
			while (true) {
				if (input.hasNext()) {
					String cmd = input.next();
					if (escapeChars.contains(cmd.charAt(cmd.length() - 1))) {										// If new line is escaped
						System.out.print("> ");
						cmd = cmd.substring(0, cmd.length() - 1);
						if (cmd.length() > 0 &&
							whitespaceChars.contains(cmd.charAt(cmd.length() - 1)) &&
							!escapeChars.contains(cmd.charAt(cmd.length() - 2))) {
							cmd = cmd.substring(0, cmd.length() - 1);
						}
						cmdSplit = parse(cmd);
						String[] moreCmd = parse(in);
						String[] newCmd = new String[cmdSplit.length + moreCmd.length];
						System.arraycopy(cmdSplit, 0, newCmd, 0, cmdSplit.length);
						System.arraycopy(moreCmd, 0, newCmd, cmdSplit.length, moreCmd.length);
						cmdSplit = newCmd;
						break;
					}
					else {
						cmdSplit = parse(cmd);
						break;
					}
				}
			}
			while (trimBlanks) {
				if (cmdSplit.length > 0 &&
					(cmdSplit[0].equals("")) ||
					cmdSplit[0] == null) {
					String[] newCmd = new String[cmdSplit.length - 1];
					System.arraycopy(cmdSplit, 1, newCmd, 0, newCmd.length);
					cmdSplit = newCmd;
				}
				if (cmdSplit.length > 0 &&
					(cmdSplit[cmdSplit.length - 1].equals("") ||
					cmdSplit[cmdSplit.length - 1] == null)) {
					String[] newCmd = new String[cmdSplit.length - 1];
					System.arraycopy(cmdSplit, 0, newCmd, 0, newCmd.length);
					cmdSplit = newCmd;
				}
				else {
					break;
				}
			}
			for (int i = 0; i< cmdSplit.length; i++) {
				for (int j = 0; j < this.escapedChars.size(); j++) {
					String escaped = this.escapeChars.get(j).toString();
					cmdSplit[i].replace(escaped, "\\" + escaped);
				}
			}
			return cmdSplit;
		}

		public String[] tokens() {
			return this.tokens;
		}

		public String nextToken() {
			String str = this.tokens[this.position];
			this.position++;
			return str;
		}

		public CommandTokenizer commentChar(int ch) {
			if (ch == -1) {
				this.commentChars.clear();
			}
			else {
				this.ordinaryChar(ch);
				this.commentChars.add((char) ch);
			}
			return this;
		}

		public CommandTokenizer commentChars(int low, int hi) {
			for (int i = low; i <= hi; i++) {
				this.ordinaryChar(i);
				this.commentChar(i);
			}
			return this;
		}
		
		public CommandTokenizer commentChars(boolean fl) {
			if (!fl) {
				this.commentChar(-1);
			}
			return this;
		}

		public CommandTokenizer escapeChar(int ch) {
			if (ch == -1) {
				this.quoteChars.clear();
			}
			else {
				this.ordinaryChar(ch);
				this.quoteChars.add((char) ch);
			}
			return this;
		}

		public CommandTokenizer escapeChars(int low, int hi) {
			for (int i = low; i <= hi; i++) {
				this.ordinaryChar(i);
				this.escapeChar(i);
			}
			return this;
		}
		
		public CommandTokenizer escapeChars(boolean fl) {
			if (!fl) {
				this.escapeChar(-1);
			}
			return this;
		}

		public CommandTokenizer escapedChar(int ch) {
			if (ch == -1) {
				this.escapedChars.clear();
			}
			else {
				this.ordinaryChar(ch);
				this.escapedChars.add((char) ch);
			}
			return this;
		}

		public CommandTokenizer escapedChars(int low, int hi) {
			for (int i = low; i <= hi; i++) {
				this.ordinaryChar(i);
				this.escapedChar(i);
			}
			return this;
		}
		
		public CommandTokenizer escapedChars(boolean fl) {
			if (!fl) {
				this.escapedChar(-1);
			}
			return this;
		}

		// Removes a character from all lists
		private CommandTokenizer ordinaryChar(int ch) {
			char chChar = (char) ch;
			if (this.commentChars.contains(chChar)) {
				this.quoteChars.remove(chChar);
			}
			if (this.escapeChars.contains(chChar)) {
				this.escapeChars.remove(chChar);
			}
			if (this.escapedChars.contains(chChar)) {
				this.escapedChars.remove(chChar);
			}
			if (this.quoteChars.contains(chChar)) {
				this.quoteChars.remove(chChar);
			}
			if (this.whitespaceChars.contains(chChar)) {
				this.whitespaceChars.remove(chChar);
			}
			return this;
		}

		public CommandTokenizer ordinaryChars(int low, int hi) {
			for (int i = low; i <= hi; i++) {
				this.ordinaryChar(i);
			}
			return this;
		}

		public CommandTokenizer quoteChar(int ch) {
			if (ch == -1) {
				this.quoteChars.clear();
			}
			else {
				this.ordinaryChar(ch);
				this.quoteChars.add((char) ch);
			}
			return this;
		}

		public CommandTokenizer quoteChars(int low, int hi) {
			for (int i = low; i <= hi; i++) {
				this.ordinaryChar(i);
				this.quoteChar(i);
			}
			return this;
		}
		
		public CommandTokenizer quoteChars(boolean fl) {
			if (!fl) {
				this.quoteChar(-1);
			}
			return this;
		}

		public CommandTokenizer whitespaceChar(int ch) {
			if (ch == -1) {
				this.whitespaceChars.clear();
			}
			else {
				this.ordinaryChar(ch);
				this.whitespaceChars.add((char) ch);
			}
			return this;
		}

		public CommandTokenizer whitespaceChars(int low, int hi) {
			for (int i = low; i <= hi; i++) {
				this.ordinaryChar(i);
				this.whitespaceChar(i);
			}
			return this;
		}
		
		public CommandTokenizer whitespaceChars(boolean fl) {
			if (!fl) {
				this.whitespaceChar(-1);
			}
			return this;
		}
		
		public CommandTokenizer eolPattern(String pattern) {
			if (pattern == null) {
				this.eolPatterns.clear();
			}
			else {
				this.eolPatterns.add(pattern);
			}
			return this;
		}
		
		public CommandTokenizer eolPatterns(boolean fl) {
			if (!fl) {
				this.eolPatterns.clear();
			}
			return this;
		}

		// Restores defaults
		public CommandTokenizer resetSyntax() {
			this.commentChars = new ArrayList<Character>();
			this.escapeChars = new ArrayList<Character>(Arrays.asList('\\'));
			this.escapedChars = new ArrayList<Character>();
			this.quoteChars = new ArrayList<Character>(Arrays.asList('"', '\''));
			this.whitespaceChars = new ArrayList<Character>(Arrays.asList(' ', '\n'));
			this.trimBlanks = true;
			return this;
		}

		// Based off the "typical" syntax in the StreamTokenizer docs
		@Override
		public String toString() {
			String str = "Token[\"" + this.tokens[this.position] + "\"], position " + this.position;
			return str;
		}

	}

Example of an interactive shell using the InputStream variant:

	public static void main(String... args) {
		String shPrompt = "cli> ";
		System.out.print(shPrompt);
		while (true) {
			String[] params = new CommandTokenizer(System.in).tokens();
			CommandLine.run(new Main(), System.out, params);
			System.out.print(shPrompt);
		}
	}

[remkop]

Cool. I’ll need some time to take a look.

About the comment methods, we should either make them work or remove them. I’ll leave it up to you. Please provide unit tests that cover all functionality, especially for any edge cases.

I noticed that the parse(InputStream) method recursively calls itself. Why is that? Also, will this work correctly when the Scanner has read more input from the InputStream than we have consumed from the Scanner? It may be better to pass the Scanner instance.

[remkop]

Can you share future versions in a pull request? GitHub has nice support for reviewing pull requests.

[BeeeWall]

It calls itself if new line escapes are enabled and the new line is escaped, in order to get whatever the user inputs next. It's inside an if block, so it only runs when needed.

What do you mean, "read more input from the InputStream than we have consumed"? I can make it accept Scanner or Reader instead/in addition to if you want.

Unfortunately, as I stated above, I can't properly work and test via pull request, because CommandLine.java crashes one IDE and throws a gazillion errors in the other. So the only way I can work is using a normal text editor and ECJ via the command line. I may try that later, but don't have time at the moment to figure out a good editor, etc.

[remkop]

Thanks for the clarification.

About the "read more input ... consumed": my point was that once we pass the InputStream to the Scanner, the Scanner now "owns" this stream and we should no longer read from the stream directly. All access to this InputStream should go via the Scanner object. The Scanner could for example "read ahead" some characters from the InputStream into an internal buffer. We would miss the characters that are buffered in the first Scanner if we wrap the InputStream with another Scanner.

Understood about the IDE. By the way, to do a pull request you don't need to compile, you can do everything on the GitHub web site: fork picocli, in your forked project navigate to src/main/java/picocli, click "Create new file" and copy/paste the source code. You can edit on the GitHub site as many times as needed. When you are ready, click "New pull request".

[BeeeWall]

I know that. However, I personally feel it makes the git history crazy ugly that way, but I can do that if you'd prefer.

[remkop]

Yes please. Git lets you squash multiple commits into a single one to clean up the commit history.

[RobertZenz]

I know I'm a little bit late to the party, but there is jline which provides a GNU readline-like functionality in Java.

[remkop]

Picocli itself should not have any external dependencies.

If it is worth building things on top of jline, it may make sense to have a picocli-shell subproject that contains the interactive stuff.

[Kshitiz-Sharma]

I personally am not a big fan of interactive shells. I'd second the idea to keep it as a separate project. Picocli core codebase is a clean and pristine work of art at the moment, lets keep it that way.

An interactive shell might sound like a good idea initially but as your application grows in complexity you'd realize it's not.

As an example from a project I've been working on, we'd developed a set of functions to aid in debugging of our microservices. So there's a function to read test data from a file and post it onto the JMS broker. Another function to send a message for dumping internal state etc.

Since creating JMS connection pool and sessions is heavy it made sense to use something like Spring Shell, to do the initialization once on startup, and then run multiple commands:

debuggin-shell.sh
spring-shell> put-requests /home/kshar/testdata.csv
spring-shell> dump-queue mainRequestQueue /tmp/dump.log

The problem is that this goes against the Unix philosophy. Instead of having a set of small programs that can be connected to each other via pipes you now have a monolithic program that does everything and is no longer scriptable. So you can no longer do:

cat /home/kshar/testdata.csv | sed 'pattern' | grep -v 'pattern' | put-requests
or
dump-queue mainRequestQueue | grep sessionId | less

A few seconds I saved in startup initialization wasn't worth sacrificing the script-ability of my shell commands, given that often the slowest part of running a shell command is my typing speed.

Furthermore, implementing an interactive shell into picocli won't be trivial:

1. Do we support tab based autocompletion?
2. What about command history? For standalone commands that history is maintained by your shell in .bash_history. For interactive shell you'd need to record this yourself, and then add hooks to arrow keys for cycling through them

I'd advise against trying to reinvent the unix shell inside a JVM. If you're on windows, you have my sympathy. Give a shot to Windows Subsystem for Linux or Cygwin.

[remkop]

@Kshitiz-Sharma has a point about leveraging the unix shell. On the other hand, for some projects an interactive CLI may be the right choice. Picocli should work well in both use cases.

I have looked at jline2 and jline3. This is a fairly large library with a lot of functionality. I haven't looked in depth yet but there's a lot there.

Rather than re-implementing this functionality I will probably focus on making sure that picocli integrates really well with jline. For example, auto-generate jline completion functions for a picocli command. I'm also planning to play with jline some more and write some articles about combining jline with picocli to easily build powerful interactive CLIs.

JLine2 does not have a tokenizer, as far as I can see, so the tokenizer in this ticket is certainly useful when integrating with jline2. JLine3 does have a tokenizer (DefaultParser), but I still need to look in detail. At first glance the tokenizer proposed here may have more features.

[billoneil]

There is also https://github.com/facebook/nailgun which seems like it wouldn't be very difficult to use with picocli.

[remkop]

With #497 there is now a new module picocli-shell-jline2 where I plan to add functionality and documentation for building interactive shells with JLine 2 and picocli. As a starting point there will be a completer that dynamically completes the command in a JLine shell based on the picocli CommandSpec. Future interactive shell functionality should go into this module.

(Update: fixed link for renamed module)

[remkop]

I'm going to close this ticket as I believe the original request has been met with the picocli-shell-jline2 module. Feel free to open new tickets for additional requests/enhancements.

[Holmistr]

Hi @remkop ! Is the functionality actually still provided somewhere? I see that the https://github.com/remkop/picocli/tree/master/picocli-jline2-shell no longer exists :(

[Holmistr]

Ignore me :) Found it: https://github.com/remkop/picocli/tree/master/picocli-shell-jline3

[remkop]

@Holmistr Glad you found it. At some early point I renamed the module from picocli-jline2-shell to picocli-shell-jline2 (and similar for jline3). One of the comments above still had a broken link to the old module name. I fixed that now.
Enjoy picocli!