transcript generation/update

[teto]

Lately I've been working a lot on my program, tinkering with the output and all.
I miss having more tests to check for regression.
I had started creating some transcripts but they got outdated and I am not sure what's the best way to update them (from scratch ?).
Right now I believe the main approach is using history and because my program accepts commands from the command line, I can run
mptcpanalyzer -d DEBUG -l ~/pcaps/tmpatm6409s/client_2_.pcapng "summary_extended ~/pcaps/tmpatm6409s/client_2_.pcapng 1 ~/pcaps/tmpatm6409s/server_2_.pcapng 1" "history 1:1 -t transcript.txt" "quit"
but I would love to be able to update transcripts in place. Like for instance I have the following transcript.txt

server_2_filtered.pcapng> load_pcap examples/server_2_filtered.pcapng
Loading examples\/server_2_filtered.pcapng
server_2_filtered.pcapng> summary 0 Client
mptcpstream 0 transferred 308 bytes.
tcpstream 0 transferred 308 bytes out of 308, accounting for 100.00%
tcpstream 2 transferred 0 bytes out of 308, accounting for 0.00%
tcpstream 6 transferred 0 bytes out of 308, accounting for 0.00%
server_2_filtered.pcapng> summary 0 Server
mptcpstream 0 transferred 461 bytes.
tcpstream 0 transferred 456 bytes out of 461, accounting for 98.92%
tcpstream 2 transferred 5 bytes out of 461, accounting for 1.08%
tcpstream 6 transferred 0 bytes out of 461, accounting for 0.00%

And I could run a command for instance transcript update transcript.txt that reruns the commands and updates the outputs accordingly.

[tleonhardt]

If you are looking for a more automated way to updated your transcripts, you could write a script to do it. Do the following:

• Write a script which contains all of the command you want to run in a given transcript
• At the end of the script, use history -t to re-generate your transcript file, then quit
• Run this script from the command line

If you find that useful and think other would as well, then we could update our documentation to mention it.

We could potentially automate the process by adding built-in command support for it. @kotfu I think you are most familiar with the transcript code, what do you think?

[teto]

yep a script might be better than integrating support in cmd2. It could be added to the faq/wiki or in contrib/.

NB: I think cmd2 can load commands from a file so no need for scripts I guess.

[teto]

so I must confess the "history" command confuses me, many commands and unclear parameters:

• history (without args) should dump the whole history, yet it prints nothing but if I do Ctrl-R, it finds things in my history (I do save and load history). It's not consistent.
• the -s flag output commands in script format while the -o, --output-file out
• a..b, a:b, a:, ..b items by indices (inclusive) : what are the differences ? does it start from 0 or 1 ? can I use negatives indices like in python ?

I created this file tests/script_mptcp.txt to generate a transcript

load_pcap examples/client_2.pcap
list_mptcp_connections
list_subflows 0
list_subflows 1
history -t tests/transcript_mptcp.txt

then I ran
mptcpanalyzer "load tests/script_mptcp.txt" "quit"
but it seems like it went into a loop. I guess the problem might come from my program but I am not sure as history.

As a comment for 1.0; I dont like that the load function is taken by cmd2 to load scripts. It is a pretty useful command that I used to load pcap files and had to rename to load_pcap because of cmd2. I would rather see renamed to load_script or script load / generate_transcript etc
In my previous tests/script_mptcp.txt, how can I refer to the last 4 commands ? I used history -t tests/transcript_mptcp.txt but that feels wrong.

[kotfu]

I agree that the history command is a bit confusing. We have already committed some changes to master which modify the help message for the history command to separate the actions from the formatting options. History actions include editing history, output command history to a file, output command history and command output to a file, and clearing all history. Formatting options include choices about whether you want command numbers included in the output. If you have any comments or improvements on the history command documentation available by typing history -h we would be pleased to review them.

To your specific questions, history without args should dump the whole history. If it's not doing that, it's a bug of some sort and we would like to track it down.

The new history help clarifies the difference between history -s and history -o. history -o FILENAME tells the history command to place the list of commands in a filename. It also clarifies that history -o automatically sets the -s option. The -s option controls whether the command numbers are included in the output. The following example comes from code running on the master branch, which has not yet been released to PyPi.

$ python examples/example.py
(Cmd) say hello
hello
(Cmd) mumble hi there
like hi there right?
(Cmd) history
    1  say hello
    2  mumble hi there
(Cmd) history -s
say hello
mumble hi there
(Cmd)

As you can see, the -s option tells the history command to display the commands in a format that you could load as a script file. If you do history -o myscriptfile.txt, the contents of that file would be:

say hello
mumble hi there

The optional history argument that uses the a..b, a:b, a:, ..b syntax you described uses python list comprehension syntax. The difference is that python uses the zero-th element as the first element of the array, but most users (as opposed to developers) think 1 is the first element, not zero. So if you have 4 commands in the history, history 2: includes command 2 and all other commands to the end of history (where in a python array history[2:] would include the third element of the array to the end.

In looking at the history code, we don't interpret negative numbers as selectors, ie history -1. The regular expression which parses the input ignores non digits except for ':'. If you feel like this is a bug, (I kinda think it is), would you just open a new issue and I'll have a look at fixing it.

As for renaming the load command, you can do this yourself without touching the cmd2 code. Just add the following lines to your cmd2 subclass (this example assumes your subclass is called CmdLineApp:

    def do_load_script(self, cmdline):
        super(CmdLineApp, self).do_load(cmdline)

    def do_load(self, cmdline):
        self.poutput('my custom load command')

[tleonhardt]

@teto You make a valid point about the fact that the history documentation could use an extensive update/re-write to better explain:

1. That there are two logically different types of history within cmd2 - readline history and the history command history
2. How and why the two differ
3. Much more info on the history command and how to use it

I will create a new issue to encompass this work. The history command very intentionally filters certain things out from what is shown and it is also intended to not be persistent between runs, whereas the readline history accessed by <Ctrl>+r and the arrow keys is intended to give you access to exactly what you have typed and intended to be persistent across runs.

For your specific problem, when you do history -t tests/transcript_mptcp.txt it uses the entire history which includes the load tests/script_mptcp.txt command at the beginning - so it makes sense you ended up in an infinite loop. You can modify the command called in your script to bypass the first command like so: history 2: -t tests/transcript_mptcp.txt. We don't support negative indices, but we should - that is a good thing to make into an issue.

Renaming the load command would be a pretty major breaking change in terms of backwards compatibility. But if other cmd2 maintainers are in favor of it, then it is something we could do.

[tleonhardt]

I created #640 to improve the Sphinx documentation as it relates to history within cmd2.

I created #641 to add support for negative indices to the history command.

[teto]

I suspected cmd2 made a difference between readline and session history but still confusing. Maybe there should be a flag --all even though i feel like there are already too many flags xD

I would like a way to generate the tranzcript without having to adjust the number in history everytime i add a command; somzthing like 'load commands.txt' -t mytranscript.txt maybe ?

I suggested Renaming load because of thr 1.0 switch. Such a valuable keyword shouldnt be preempted imo but as mentioned, workarounds exist.

[teto]

what about my load commands.txt --record-as-transcript mytranscript.txt suggestion ? would love to use that to generate my transcripts.

[tleonhardt]

@teto Your suggestion sounds like a reasonable one - I think it would be useful and easy to implement. So yeah, we will experiment with it and if it seems to work well we will add it.